OpenVSwitch Logs aren't rotated

[bplessis]

It look like there is a discrepancy between expected openvswitch log location (/var/log/openvswitch/*.log) and the real log files (/var/log/ovs-*.log so no log rotation occurs on thoses files and some can grow big

-rw-r--r--  1 root root     2010 12 nov.  11:47 ovs-ctl.log
-rw-r--r--  1 root root     1510 12 nov.  11:47 ovsdb-server.log
-rw-r--r--  1 root root  5149493 21 nov.  10:35 ovs-vswitchd.log
-rw-r--r--  1 root root      594 12 nov.  11:58 ovs-xapi-sync.log

[olivierlambert]

Interesting. Let me check if I have the same behaviour in XenServer 👍

[olivierlambert]

Same issue on XenServer, can you create a XSO here? https://bugs.xenserver.org

I mean, we'll fix it, but ideally, having the "upstream" fixed is also important 😄

[olivierlambert]

@stormi I know where to make the modification, if you can create a repo (if it doesn't exists yet), I can write a fix myself.

[stormi]

Ok :)
I just need to know what you'll need to modify before I can create or point at the appropriate repo.

[olivierlambert]

Okay so I need to modify /etc/logrotate.d/openvswitch file.

This line:

/var/log/openvswitch/*.log {

Will be replaced by:

/var/log/ovs-*.log {

Do you concur @bplessis ?

[bplessis]

XSO Created https://bugs.xenserver.org/browse/XSO-919

Okay so I need to modify /etc/logrotate.d/openvswitch file.

Well there are two ways to fix this, it could be interesting to also try to change the OVS configuration so that they log in the correct directory.

But having the logrotate configuration fixed so as to rotate both location could be a plus

[stormi]

Changing the log location could be confusing because the old logs would still remain at the old location, so I favour the minimal change, that is fixing the log rotation only. Unless the wrong location causes issues (but changing the location might cause other issues if anything in XenServer relies on it).

Repository created: https://github.com/xcp-ng-rpms/openvswitch

The way to fix until it is fixed in XS is by creating a patch to xenserver/etc_logrotate.d_openvswitch

See https://xcp-ng.org/forum/topic/183/git-repository-structure-and-conventions-for-rpms ("Naming patches" and "Patch creation").

[stormi]

I have pushed an update candidate for 7.6

yum install openvswitch-2.5.3-2.2.3.1.xcp.x86_64.rpm --enablerepo='xcp-ng-updates_testing'

I don't know if changes to logrotate configuration are taken into account automatically.

[bplessis]

Logrotate is started by cron (cron.daily/logrotate) so no need for reboot/restart, however it will take two rotation cycle for the rotation to be triggered.

[stormi]

Hi! Have you been able to test the update candidate to see if it works? Note that the configuration will not be updated if you manually modified it so it would be better to test with an unaltered configuration to start with.

[bplessis]

Hi,
No sorry, i tried during two days but yum kept stating there was no such package, i'll retry now

[root@theia ~]# yum install openvswitch-2.5.3-2.2.3.1.xcp.x86_64.rpm --enablerepo='xcp-ng-updates_testing'
Loaded plugins: fastestmirror
dell-system-update_dependent                                                                                                                     | 2.3 kB  00:00:00     
dell-system-update_independent                                                                                                                   | 2.3 kB  00:00:00     
xcp-ng-base/signature                                                                                                                            |  455 B  00:00:00     
xcp-ng-base/signature                                                                                                                            | 3.0 kB  00:00:00 !!! 
xcp-ng-updates/signature                                                                                                                         |  455 B  00:00:00     
xcp-ng-updates/signature                                                                                                                         | 3.0 kB  00:00:00 !!! 
xcp-ng-updates_testing/signature                                                                                                                 |  455 B  00:00:00     
xcp-ng-updates_testing/signature                                                                                                                 | 3.0 kB  00:00:00 !!! 
xcp-ng-updates_testing/primary_db                                                                                                                | 4.9 kB  00:00:00     
Loading mirror speeds from cached hostfile
No package openvswitch-2.5.3-2.2.3.1.xcp.x86_64.rpm available.
Error: Nothing to do

[stormi]

Hi. Could you try again now? I promise this time if you get the same error I'll react quickly and find why :)

[bplessis]

Hi,

Still same error, and i can't seem to have any upgrade whatsoever, could this be related to the additionnel dell repo ?

[root@theia ~]# yum install openvswitch-2.5.3-2.2.3.1.xcp.x86_64.rpm --enablerepo='xcp-ng-updates_testing'
Loaded plugins: fastestmirror
Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
dell-system-update_dependent                                                                                                                                                           | 2.3 kB  00:00:00     
dell-system-update_independent                                                                                                                                                         | 2.3 kB  00:00:00     
xcp-ng-base/signature                                                                                                                                                                  |  455 B  00:00:00     
xcp-ng-base/signature                                                                                                                                                                  | 3.0 kB  00:00:00 !!! 
xcp-ng-updates/signature                                                                                                                                                               |  455 B  00:00:00     
xcp-ng-updates/signature                                                                                                                                                               | 3.0 kB  00:00:00 !!! 
xcp-ng-updates_testing/signature                                                                                                                                                       |  455 B  00:00:00     
xcp-ng-updates_testing/signature                                                                                                                                                       | 3.0 kB  00:00:00 !!! 
(1/4): xcp-ng-updates/primary_db                                                                                                                                                       |  26 kB  00:00:00     
(2/4): xcp-ng-updates_testing/primary_db                                                                                                                                               | 5.6 kB  00:00:00     
(3/4): dell-system-update_dependent/7/x86_64/primary_db                                                                                                                                |  32 kB  00:00:00     
(4/4): dell-system-update_independent/primary_db                                                                                                                                       | 122 kB  00:00:00     
Determining fastest mirrors
No package openvswitch-2.5.3-2.2.3.1.xcp.x86_64.rpm available.
Error: Nothing to do
[root@theia ~]# yum makecache fast
Loaded plugins: fastestmirror
dell-system-update_dependent                                                                                                                                                           | 2.3 kB  00:00:00     
dell-system-update_independent                                                                                                                                                         | 2.3 kB  00:00:00     
xcp-ng-base/signature                                                                                                                                                                  |  455 B  00:00:00     
xcp-ng-base/signature                                                                                                                                                                  | 3.0 kB  00:00:00 !!! 
xcp-ng-updates/signature                                                                                                                                                               |  455 B  00:00:00     
xcp-ng-updates/signature                                                                                                                                                               | 3.0 kB  00:00:00 !!! 
Loading mirror speeds from cached hostfile
Metadata Cache Created
[root@theia ~]# yum install openvswitch-2.5.3-2.2.3.1.xcp.x86_64.rpm --enablerepo='xcp-ng-updates_testing'
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
No package openvswitch-2.5.3-2.2.3.1.xcp.x86_64.rpm available.
Error: Nothing to do
[root@theia ~]# yum upgrade
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Resolving Dependencies
--> Running transaction check
---> Package dell-system-update.x86_64 0:1.5.3-18.03.00 will be updated
---> Package dell-system-update.x86_64 0:1.6.0-18.11.00 will be an update
---> Package dsucatalog.noarch 0:18.11.00-1KYXP will be updated
---> Package dsucatalog.noarch 0:18.12.00-244P6 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================================================
 Package                                           Arch                                  Version                                          Repository                                                     Size
==============================================================================================================================================================================================================
Updating:
 dell-system-update                                x86_64                                1.6.0-18.11.00                                   dell-system-update_independent                                1.8 M
 dsucatalog                                        noarch                                18.12.00-244P6                                   dell-system-update_independent                                523 k

Transaction Summary
==============================================================================================================================================================================================================
Upgrade  2 Packages

Total download size: 2.3 M
Is this ok [y/d/N]: y
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
(1/2): dsucatalog-18.12.00-244P6.noarch.rpm                                                                                                                                            | 523 kB  00:00:01     
(2/2): dell-system-update-1.6.0-18.11.00.x86_64.rpm                                                                                                                                    | 1.8 MB  00:00:03     
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                         616 kB/s | 2.3 MB  00:00:03     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : dell-system-update-1.6.0-18.11.00.x86_64                                                                                                                                                   1/4 
  Updating   : dsucatalog-18.12.00-244P6.noarch                                                                                                                                                           2/4 
  Cleanup    : dsucatalog-18.11.00-1KYXP.noarch                                                                                                                                                           3/4 
  Cleanup    : dell-system-update-1.5.3-18.03.00.x86_64                                                                                                                                                   4/4 
  Verifying  : dsucatalog-18.12.00-244P6.noarch                                                                                                                                                           1/4 
  Verifying  : dell-system-update-1.6.0-18.11.00.x86_64                                                                                                                                                   2/4 
  Verifying  : dsucatalog-18.11.00-1KYXP.noarch                                                                                                                                                           3/4 
  Verifying  : dell-system-update-1.5.3-18.03.00.x86_64                                                                                                                                                   4/4 

Updated:
  dell-system-update.x86_64 0:1.6.0-18.11.00                                                                dsucatalog.noarch 0:18.12.00-244P6                                                               

Complete!

[stormi]

Ok, I made a mistake when I gave you the installation command.

Here's the one that works:

yum install openvswitch-2.5.3-2.2.3.1.xcp.x86_64 --enablerepo='xcp-ng-updates_testing'

[bplessis]

There is one file left over: ovsdb-server.log

[bplessis]

Well actually given the content i should try to get rid of this, this log connection attempts to a OpenVswitch controller that is not here anymore ...

[stormi]

So about ovsdb-server.log is there a configuration change that we should do?

[bplessis]

Hi,

Sorry i missed this in the noise of my mailbox.

Well it is a file generated by the ovs daemon so given that it can get pretty big in some case (and that's not easily fixable btw) i would say it should be rotated with the rest of them

[stormi]

I have uploaded openvswitch-2.5.3-2.2.3.2.xcp that should also rotate that file, if I got the syntax correctly.

yum install openvswitch-2.5.3-2.2.3.2.xcp.x86_64 --enablerepo='xcp-ng-updates_testing'

[stormi]

I have pushed the update to all XCP-ng 7.6 users. Closing. Thanks for the report.

[bplessis]

There seem to be a little bug in the current logrotate configuration, the system is rotating rotated logs:

-rw-r--r-- 1 root root 0 24 févr. 00:33 ovsdb-server.log.10.gz
-rw-r--r-- 1 root root 77 14 févr. 00:40 ovsdb-server.log.10.gz.1.gz
-rw-r--r-- 1 root root 77 13 févr. 00:53 ovsdb-server.log.10.gz.2.gz
-rw-r--r-- 1 root root 77 12 févr. 00:03 ovsdb-server.log.10.gz.3.gz
-rw-r--r-- 1 root root 77 11 févr. 01:33 ovsdb-server.log.10.gz.4.gz
-rw-r--r-- 1 root root 77 10 févr. 01:27 ovsdb-server.log.10.gz.5.gz
-rw-r--r-- 1 root root 77 9 févr. 00:42 ovsdb-server.log.10.gz.6.gz
-rw-r--r-- 1 root root 275 8 févr. 10:24 ovsdb-server.log.10.gz.7.gz
-rw-r--r-- 1 root root 0 23 févr. 00:35 ovsdb-server.log.11.gz

The file selection line in /etc/logrotate.d/openvswitch

/var/log/ovs-*.log /var/log/ovsdb-* {

should probably be better like this:

/var/log/ovs-*.log /var/log/ovsdb-*.log {

Also maybe the "notifempty" flag could prevent loosing usefull archived data to useless empty files.

I can't really grasp however why there is only one rotated xensource.log file and 17 ovs-xapi-sync.log

[stormi]

Thanks for the notification. Can you test http://koji.xcp-ng.org/kojifiles/work/tasks/2782/2782/openvswitch-2.5.3-2.2.3.3.xcpng.x86_64.rpm ?

[stormi]

Someone reported the issue on the forum. I did not expect the wrong pattern to create 75000 files! https://xcp-ng.org/forum/topic/1031/too-many-weird-ovsdb-server-log-file

We need to test the update candidate quickly.

[bplessis]

Look like my mail reply didn't make it here:

I cleaned up every server and installed the proposed upgrade, looks ok for now
(and it still is now)

and yes, it (previous update) did create an awefull lot of file, had to use "find -delete" to remove them

[stormi]

When did you install the update?

[bplessis]

3 days ago, like 2h after your post

[stormi]

Did you restart the host or restart the openvswitch service to check that the updated binaries are still functional? Since the package went through the build system again, it's not exactly the same binary though it's built from the exact same sources, so I'd like to be sure.

[bplessis]

Hum, no did not do that. Will do and report

[stormi]

@bplessis earlier you said

I can't really grasp however why there is only one rotated xensource.log file and 17 ovs-xapi-sync.log

At first I did not understand because my test hosts have many rotated xensource.log files, not just one. Then I saw on one of our lab hosts that there was indeed only one rotated xensource.log file and wondered why the difference. Here's the state of my analysis: #149

[bplessis]

it rebooted ^^ vm running on it, looks ok

[stormi]

Thanks. All fine here also, and I got positive feedback on the forum. I think we're ready to push it as an update.

[stormi]

Update pushed.

[mas90]

@stormi Any chance that the update could also clear up the mess of multiply-rotated logs left behind in /var/log? I'd rather not have to do this manually on every host...

Perhaps as a post-install script: find /var/log -maxdepth 1 -name "ovsdb-*.log.*.*.*" -size -2b -delete

(which avoids anything larger than 512 bytes, just in case).

[stormi]

I considered doing it, but after the first mistake, in the prospect of releasing a fix fast, I preferred not to risk writing a broken post-install script. I could create an update candidate that cleans up and find time and users to have it tested if there's enough demand for it. Do you have many hosts?

[borzel]

@stormi I would also vote for a conservative way. After this "hickup" we don't need a broken post install script :-)

[De-Occultist]

Dependencies Resolved

Package Arch Version Repository Size

Updating:
openvswitch x86_64 2.5.3-2.2.3.3.xcpng xcp-ng-updates 2.2 M

Transaction Summary

Upgrade 1 Package

Total size: 2.2 M
Is this ok [y/d/N]: Y
Downloading packages:
Running transaction check
Running transaction test

Transaction check error:
installing package openvswitch-2.5.3-2.2.3.3.xcpng.x86_64 needs 1 inodes on the /var/log filesystem

For others who encounter the same problem trying to upgrade openvswitch.

Delete some 0 byte files in /var/log/*
I had about 110K 0 byte files.

[hugodeco]

Same problem of no available inodes on a 27 TB disk.

The compressed files are deleted by another script, based on total disc use.
Therefore for logrotate we specify a large number.
The log rotate wont start because it has a lot of free space, but no inodes.

Resolved !