Changes to logrotate package are causing issues

[borpin]

References

• make renamecopy and copytruncate override each other logrotate/logrotate#386
• https://github.com/logrotate/logrotate/blob/master/ChangeLog.md#3181---2021-05-21)
• https://community.openenergymonitor.org/t/no-disk-space-although-enough-room/22042

1. Key issue is that 'copytruncate' and 'renamecopy' are now mutually exclusive.
2. There is also an issue caused by removing the emonhub user/group
3. There seems to be an issue with the renamecopy line in the 00_olddir config file. deleting the line and restoring it solved the issue.

This should be tested with sudo /usr/sbin/logrotate -d /etc/logrotate.conf - pay attention to errors in first few lines as config files are read.

Suggested changes:

pi@emonpi:/opt/openenergymonitor/EmonScripts $ git diff
diff --git a/defaults/etc/logrotate.d/00_defaults b/defaults/etc/logrotate.d/00_defaults
index 9399f84..7e9a726 100644
--- a/defaults/etc/logrotate.d/00_defaults
+++ b/defaults/etc/logrotate.d/00_defaults
@@ -4,8 +4,6 @@ maxsize 250k
         rotate 7
         daily
         compress
-        copytruncate
-        size 100k
         nocreate
         missingok
         notifempty
diff --git a/defaults/etc/logrotate.d/emoncms b/defaults/etc/logrotate.d/emoncms
index 2935f79..737c47e 100644
--- a/defaults/etc/logrotate.d/emoncms
+++ b/defaults/etc/logrotate.d/emoncms
@@ -3,4 +3,5 @@
     compress
     olddir /var/log.old/emoncms
     createolddir 775 root root
+    renamecopy
 }
diff --git a/defaults/etc/logrotate.d/emonhub b/defaults/etc/logrotate.d/emonhub
index 48033d1..ed5d7b9 100644
--- a/defaults/etc/logrotate.d/emonhub
+++ b/defaults/etc/logrotate.d/emonhub
@@ -1,11 +1,7 @@
 /var/log/emonhub/emonhub.log {
     maxsize 3M
-
-    norenamecopy
-    copytruncate
-    su root root
     compress
-
     olddir /var/log.old/emonhub
-    createolddir 775 root emonhub
+    createolddir 775 root root
+    renamecopy
 }

[borpin]

Another issue, logrotate logs do not go to the olddir.

This is because the 00_defaults inserted by the EmonScripts, is read beofore the 00_olddir command file.

pi@emonpi:/opt/openenergymonitor/EmonScripts/install $ cat /opt/openenergymonitor/EmonScripts/defaults/etc/logrotate.d/00_defaults
maxsize 250k

/var/log/logrotate/*.log {
        rotate 7
        daily
        compress
        nocreate
        missingok
        notifempty
        delaycompress
}
pi@emonpi:/opt/openenergymonitor/EmonScripts/install $ cat /etc/logrotate.d/00_olddir
# These settings will save the rotated logfiles to a seperate location.
# This file needs to be located in /etc/logrotate.d.
# The "00" prefix in the file name is required to ensure this file is
# loaded before any program specific logrotate settings are loaded as
# contents of /etc/logrotate.d are read in alphabetical order.

# The path & name for rotated logfiles folder.
# "/var/log.old" will `ls` next to "/var/log" and "/var/log.bak"
# This must mirror the OLD_LOG variable in the log2ram script.
olddir /var/log.old

# Create the path if it doesn't exist
createolddir 755 root root

# To allow the files to be "moved" (ie copied and original deleted)
# to another device/partition (eg out of RAM)
nocopytruncate
renamecopy

I suggest a logrotate directive file is created in the OEM folder and symlinked to, and the directives for logrotate are moved to that file.

[borpin]

Final thing, the EmonSD image has a few random log files left that I think will not rotate and could do with being removed before the image is created.

[alexandrecuer]

Do you have this in /var/log/messages ?

Jan  6 08:27:01 emonpi rsyslogd: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="457" x-info="https://www.rsyslog.com"] rsyslogd was HUPed
Jan  6 09:27:02 emonpi rsyslogd: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="457" x-info="https://www.rsyslog.com"] rsyslogd was HUPed
Jan  6 10:27:01 emonpi rsyslogd: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="457" x-info="https://www.rsyslog.com"] rsyslogd was HUPed
Jan  6 11:27:01 emonpi rsyslogd: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="457" x-info="https://www.rsyslog.com"] rsyslogd was HUPed
Jan  6 12:27:02 emonpi rsyslogd: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="457" x-info="https://www.rsyslog.com"] rsyslogd was HUPed

I dont know if this is normal

[borpin]

Do you have this in /var/log/messages ?

Do you have this in /var/log/messages ?

Seems this is normal and just says that it was restarted as a result of the rotation

https://talk.plesk.com/threads/messages-log-is-failing.342098/

[alexandrecuer]

OK thanks
I have logrotate 3.18.0. so I wonder if if should test the 2 first changes you suggest in 00_defaults and emoncms ? it is suitable for logrotate 3.18.1 if I understand correctly
I cannot test the third change because I dont have emonhub installed on the pi....

[borpin]

This was done/tested on 3.18.0. I think 3.18.1 forced the 2 to be mutually exclusive rather than just a warning.

Run the debug command sudo /usr/sbin/logrotate -d /etc/logrotate.conf and see what is thrown up by way of errors (first few lines as it reads the conf files).

I didn't save the original messages and have not reflashed the card again.

[alexandrecuer]

I've run this command, but it returns no error, just a warning :

warning: 'size' overrides previously specified 'daily'

I had 2 night crashes on a pi which is commanding pumps and three way valves for heating a building. I thought it was the logrotate problem. But maybe it is somethings else as I am commanding the pumps with the PI gpios, maybe it is not suitable and I should go for dedicated modules. But I dont think a kernel panic can be related to a logrotate problem...it is more related to sd card or hardware problem I think ?

[borpin]

I was wrong, I do have the original error messages. Many of these are specific to the EmonSD.

pi@emonpi:~ $ /usr/sbin/logrotate -d /etc/logrotate.conf
WARNING: logrotate in debug mode does nothing except printing debug messages!  Consider using verbose mode (-v) instead if this is not what you want.

reading config file /etc/logrotate.conf
including /etc/logrotate.d
reading config file 00_defaults
warning: 'size' overrides previously specified 'daily'
reading config file 00_olddir
olddir is now /var/log.old
error: 00_olddir:17 keyword 'renamecopy' not properly separated, found 0
error: found error in file 00_olddir, skipping
reading config file alternatives
reading config file apache2
reading config file apt
reading config file btmp
reading config file dpkg
reading config file emoncms
olddir is now /var/log.old/emoncms
error: emoncms:6 olddir /var/log.old/emoncms and log file /var/log/emoncms/apache2-error.log are on different devices
removing last 1 log configs
error: found error in file emoncms, skipping
reading config file emonhub
olddir is now /var/log.old/emonhub
error: emonhub:10 unknown group 'emonhub'
error: found error in /var/log/emonhub/emonhub.log , skipping
removing last 1 log configs
error: found error in file emonhub, skipping
reading config file log2ram
reading config file mariadb
reading config file mosquitto
warning: 'size' overrides previously specified 'daily'
reading config file ppp
reading config file redis-server
reading config file rsyslog
reading config file ufw
reading config file wtmp
Reading state from file: /var/lib/logrotate/status
error: error opening state file /var/lib/logrotate/status: Permission denied
Allocating hash table for state file, size 64 entries

[alexandrecuer]

I also have the logrotate logs not going to var/log.old

ls -al /var/log/logrotate
total 208
drwxr-xr-x  2 root adm     220 Jan  9 16:17 .
drwxr-xr-x 13 root root    580 Jan  9 16:17 ..
-rw-r--r--  1 root root  65123 Jan  9 16:17 logrotate.log
-rw-r--r--  1 root root 107859 Jan  9 12:17 logrotate.log.1
-rw-r--r--  1 root root   4258 Jan  9 04:17 logrotate.log.2.gz
-rw-r--r--  1 root root   4071 Jan  8 20:17 logrotate.log.3.gz
-rw-r--r--  1 root root   3752 Jan  8 12:17 logrotate.log.4.gz
-rw-r--r--  1 root root   4567 Jan  8 04:17 logrotate.log.5.gz
-rw-r--r--  1 root root   3992 Jan  7 20:17 logrotate.log.6.gz
-rw-r--r--  1 root root   3970 Jan  7 12:17 logrotate.log.7.gz
-rw-r--r--  1 root root   2064 Jan  9 16:17 logrotate.status

did you manage to find how to correct things in order to make them go into /var/log.old ?
I didn't know if you had found a solution after reading your messages

[borpin]

Do you have got this kind of things at boot in your apache2.error.log ?

Let's not conflate more than one issue.

did you manage to find how to correct things in order to make them go into /var/log.old ?

Yes

I suggest a logrotate directive file is created in the OEM folder and symlinked to, and the directives for logrotate are moved to that file.

With the contents of

/var/log/logrotate/*.log {
rotate 7
daily
compress
nocreate
missingok
notifempty
delaycompress
}

So order config files are read in is

• 00_defaults (set maxsize)
• 00_olddir (set olddir for all logs unless overridden for specific logs) - this is added by installing log2ram
• logrotate (number of rotations etc)

I didn't propose an actual solution as the update process might be a little tricky :)

[TrystanLea]

Is this closable now?

[borpin]

But I dont think a kernel panic can be related to a logrotate problem

Totally different issue.

I also have the logrotate logs not going to var/log.old

Try running an update. It should add a logrotate logrotate specific rotation config file and link it so it is picked up by logrotate :)