limit linux log sizes

[catfact]

not the first time a user has run out of disk space from huge log files

Output from ls /var/log from https://llllllll.co/t/norns-help/14016/972?u=zebra

-rw-r----- 1 root adm 783M Feb 12 06:25 daemon.log.1
-rw-r----- 1 root adm 783M Feb 12 06:25 syslog.1
-rw-r----- 1 root adm 207M Feb 12 22:42 syslog
-rw-r----- 1 root adm 207M Feb 12 22:42 daemon.log
-rw-r----- 1 root adm 948K Feb 12 02:26 kern.log.1
-rw-r----- 1 root adm 707K Feb 12 06:25 messages.1
-rw-r----- 1 root adm 432K Oct 2 06:25 syslog.7.gz

i'm no expert but i thought logrotate was the typical way to deal with this.

(maybe this should be on norns-image)

[simonvanderveldt]

Odd I believe we only use journald which by default doesn't persist. Seems odd that a syslog file would even be created at all.

[ngwese]

Looking at my device I see about 100MB in /var/log. The log files themselves are getting rotated but historic versions are also being retained - in some cases I have upwards of 7 compressed historic logs (across all sub systems)

Looking at /etc/systemd/journald.conf it appears we may be running a default configuration:

/etc/systemd $ more journald.conf 
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See journald.conf(5) for details.

[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitIntervalSec=30s
#RateLimitBurst=1000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#SystemMaxFiles=100
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#RuntimeMaxFiles=100
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg

[tehn]

fixed. enabled:

SystemMaxUse=100
SystemMaxFiles=1

monome/norns-image@a13bfec

[artfwo]

@tehn is that file really stored in /dev/systemd/?

[tehn]

yep!

192.168.1.27 ~/norns $ ls /etc/systemd/
journald.conf  network        system       timesyncd.conf  user.conf
logind.conf    resolved.conf  system.conf  user

[artfwo]

but that's /etc. in monome/norns-image@a13bfec you add command to put it in /dev.

[tehn]

thank you for spotting this :)

[simonvanderveldt]

I'm still unsure where the files in /var/log are coming from, default journald config is to not persist anything and don't forward to syslog. It seems like a syslog daemon is running, but that would mean the Storage option of journald needs to be set somewhere 😕
The options you've changed @tehn don't relate to syslog so probably won't make a difference for the log files show in the issue's description.

[tehn]

@simonvanderveldt any further suggestions what to change?

[simonvanderveldt]

@tehn I'll have a look this weekend

[simonvanderveldt]

journald

It seems like there's only 1 journald config file, the other locations listed in the docs https://www.freedesktop.org/software/systemd/man/journald.conf.html are empty:

cat /etc/systemd/journald.conf 
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See journald.conf(5) for details.

[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitIntervalSec=30s
#RateLimitBurst=1000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#SystemMaxFiles=100
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#RuntimeMaxFiles=100
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg

All settings are commented out, unfortunately it seems like there's no way to let journald/journalctl print which settings it's currently using :(

Assuming the docs are correct the journal itself isn't being persisted since Storage=auto and /var/log/journal doesn't exist

ls -ahl /var/log/
total 20M
drwxr-xr-x  4 root root 4.0K May 24 18:24 .
drwxr-xr-x 11 root root 4.0K May 14 23:27 ..
-rw-r--r--  1 root root 4.1K May 23 22:50 alternatives.log
drwxr-xr-x  2 root root 4.0K May 24 18:13 apt
-rw-r-----  1 root adm  821K Sep  6 21:35 auth.log
-rw-r--r--  1 root root 3.8K May 14 23:22 boot.log
-rw-r--r--  1 root root    0 Apr 18 08:08 bootstrap.log
-rw-------  1 root utmp 3.8K Sep  6 21:35 btmp
-rw-r-----  1 root adm  7.1M Sep  6 21:36 daemon.log
-rw-r-----  1 root adm   98K Sep  6 21:34 debug
-rw-r--r--  1 root root 151K Sep  6 21:22 dpkg.log
-rw-r--r--  1 root root 2.6K May 15 00:17 faillog
-rw-r-----  1 root adm  3.4M Sep  6 21:48 kern.log
-rw-rw-r--  1 root utmp 286K Sep  6 21:35 lastlog
-rw-r-----  1 root adm  2.4M Sep  6 21:34 messages
drwxr-x---  2 root adm  4.0K Mar  5  2018 samba
-rw-r-----  1 root adm  5.5M Sep  6 21:48 syslog
-rw-r-----  1 root adm   72K Sep  6 21:34 user.log
-rw-rw-r--  1 root utmp 455K Sep  6 21:36 wtmp

The changes made by @tehn are still a smart thing to do in case someone were to have norns running for a long period of time the journal might become rather large, so putting a cap on that is a good move :)

syslog

But the initial issue was with log files created by syslog. We now know there's no integration from journald to syslog but for some reason syslog is also being ran independently of journald, no clue why raspbian went with this:

systemctl list-units | grep -i syslog
rsyslog.service                                                                                 loaded active running   System Logging Service                                                                   
syslog.socket                                                                                   loaded active running   Syslog Socket                                                                            

Maybe we can just disable syslog? Anyone know why it's still there despite journald also being used?

[ngwese]

Investigation of /etc/rsyslogd.conf might be warranted.

https://linux.die.net/man/5/rsyslog.conf

The section on rules and output channels seem like they might offer the control we’re after - that is assuming the rsyslogd is being used on norns (I’m away from a device at the moment)

[tehn]

trying a few things.

for /etc/logrotate.conf

daily
rotate 1
size 1M

---

for /etc/logrotate.d/rsyslog

/var/log/syslog
{
	rotate 1
	size 1M
...

---

for /etc/rsyslog.conf under RULES

$outchannel mysyslog,/var/log/syslog,1048576
*.*;auth,authpriv.none  :omfile:$mysyslog
#*.*;auth,authpriv.none		-/var/log/syslog

---

hopefully this caps everything at 1M

[tehn]

attempted fix monome/norns-image#61

[tehn]

closing this. reopen if we see problems.