systemd rsyslog service does not create PID file, that is used by logrotate scripts for SIGHUP

[selivan]

OS: Ubuntu 16.04 Xenial
rsyslog package: 8.34.0-0adiscon2xenial1 from ppa:adiscon/v8-stable

Logrotate scripts often use invoke-rc.d rsyslog rotate to tell rsyslog to close all open files. /etc/init.d/rsyslog script expects PID file to be /var/run/rsyslogd.pid.

Systemd script launches rsyslog with this arguments: ExecStart=/usr/sbin/rsyslogd -n -iNONE. PID file is not created. Logrotate is broken.

To fix this, systemd rsyslog script should launch it like that:

ExecStart=/usr/sbin/rsyslogd -n -i/var/run/rsyslogd.pid

[selivan]

systemd script is inside rsyslog source, here is the PR: rsyslog/rsyslog#2739

[davidelang]

This is a deliberate change that was just made at the request of the maintainers.

[selivan]

@davidelang How are logrotate scripts supposed to send SIGHUP then?

[davidelang]

<sarcasam> everything needs to go through systemd </sarcasam> I generally have used killall -HUP rsyslogd instead of the PID file, but I haven't looked at what the distros are doing instead now.

[selivan]

@davidelang Here is what stock rsyslog in Ubuntu Xenial do:

root@ubuntu-2gb-nbg1-dc3-1 ~ # dpkg -l | grep rsyslog
ii  rsyslog                          8.16.0-1ubuntu3                            amd64        reliable system and kernel logging daemon
root@ubuntu-2gb-nbg1-dc3-1 ~ # systemctl cat rsyslog.service  | grep Exec
ExecStart=/usr/sbin/rsyslogd -n
root@ubuntu-2gb-nbg1-dc3-1 ~ # cat /etc/logrotate.d/rsyslog  | grep -A2 postrotate
	postrotate
		invoke-rc.d rsyslog rotate > /dev/null
	endscript
--
	postrotate
		invoke-rc.d rsyslog rotate > /dev/null
	endscript

I suggest to have PID file for interaction with rsyslog process in upstream. It won't hurt anybody. killall is sloppy approach, for example it may affect lxc containers also running rsyslog like main system.

Now if you install rsyslog from ppa:adiscon/v8-stable on Ubuntu Xenial, logrotate is broken.

[friedl]

We have just published a new revision of the packages. The new ones are now creating the pid file again on service startup. This should also solve the logrotate issue.

[wsldankers]

Why not systemctl kill --kill-who=main --signal=HUP rsyslog?

[rgerhards]

Why not systemctl kill --kill-who=main --signal=HUP rsyslog?

There was a lot of complaints on the rsyslog mailing list (see archive) that other programs need the pid file as well.

[mbiebl]

pid files are not needed under systemd and should be avoided if possible. The official Debian package does not need a pid file and uses -iNONE and systemctl kill -s HUP rsyslog.service in the logrotate postrotate. I'm not aware of other programs needing the rsyslog pid file.

[selivan]

@mbiebl
Configurations that use imfile to forward log files with rsyslog may rely on pid file. Right now rsyslog 8.1904.0-0adiscon1bionic1 from official ppa comes with logrotate file using invoke-rc.d rsyslog rotate, which uses pid file. Syslog daemon is core system software, it should keep compatibility with old configurations as long as possible.

[mbiebl]

Configurations that use imfile to forward log files with rsyslog may rely on pid file.

How so?

[selivan]

@mbiebl

My mistake, files rotated externally do not need it. OK, then software writing to syslog, something like this:

/etc/rsyslog.d/foobar.conf:

if( $syslogtag == 'foobar:')  then {
        action(type="omfile" file="/var/log/foobar/main.log" template="OnlyMsg")
        stop
}

/etc/logrotate.d/foobar.conf:

/var/log/foobar/*.log {
        daily
        rotate 7
        sharedscripts
        postrotate
            [ -s /run/rsyslogd.pid ] && kill -HUP `cat /run/rsyslogd.pid`
        endscript
}

[mbiebl]

Using systemctl kill -s HUP rsyslog.service as I mentioned earlier is a much more reliable way to trigger a reload under systemd. And as said, the whole Debian archive ships no single 3rd party package which require rsyslogd.pid (checked via codesearch.debian.net)

[davidelang]

We had a release that disabled the pid file by default and ran into numerous problem reports as a result. As a result, I believe that we should create the pid file by default. If a distro doesn't want the pid file to exist, we provide the option for you to disable it. David Lang

[mbiebl]

@davidelang Can you point to such reports?

[mbiebl]

I'm asking because Debian is a rather conservative distro and I have the pid file disabled there for over a year now without a single bug report.

[selivan]

I am not saying to use it in new configuration, I am saying there are a lot of productions systems using this already, together with official rsyslog packages from ppa, and people won't be happy to suddenly get broken system. Or upgrade to new distro version, that got rid of pid file, and suddenly stumble over broken configuration.

Again, you say it's reliable? What if rsyslog launches additional processes, with mmexternal for example or some output plugins that require it? Are this processes going to get the signal too or just the main process?

Backward compatibility is important, system core software should keep it as long as possible.

[mbiebl]

Again, you say it's reliable? What if rsyslog launches additional processes, with mmexternal for example or some output plugins that require it? Are this processes going to get the signal too or just the main process?

If you use [ -s /run/rsyslogd.pid ] && kill -HUP $(cat /run/rsyslogd.pid) those child processes don't get the HUP signal, do they?

[mbiebl]

Really, pid files are a terrible idea. The less there are of them, the better.

[selivan]

f you use [ -s /run/rsyslogd.pid ] && kill -HUP $(cat /run/rsyslogd.pid) those child processes don't get the HUP signal, do they?

That's the point, this processes should not get the signal, they might do unexpected things. Signal is for the main process only.

Really, pid files are a terrible idea. The less there are of them, the better.

It's called backward compatibility, it should be kept as long as possible.

[mbiebl]

f you use [ -s /run/rsyslogd.pid ] && kill -HUP $(cat /run/rsyslogd.pid) those child processes don't get the HUP signal, do they?

That's the point, this processes should not get the signal, they might do unexpected things. Signal is for the main process only.

If you want to send a signal to the main process only, that is easily doable with systemctl.

Really, pid files are a terrible idea. The less there are of them, the better.

It's called backward compatibility, it should be kept as long as possible.

I don't agree. Sometimes bad practices should not be prolonged unnecessarily.

[davidelang]

On Mon, 29 Apr 2019, Michael Biebl wrote: I'm asking because Debian is a rather conservative distro and I have the pid file disabled there for over a year now without a single bug report.

note that not everyone uses the latest rsyslog build on the latest versions of the distros. I would have to go back through the mailing list to see the bug reports, but there were a bunch of them, which is why we reverted back to creating the PID file by default. not all distros use systemd (OpenWRT is a significant one that doesn't) We really don't like to break backwards compatibility, and only do it if there is a really important reason. I don't see a compelling reason here. David Lang

[mbiebl]

On Mon, 29 Apr 2019, Michael Biebl wrote: I'm asking because Debian is a rather conservative distro and I have the pid file disabled there for over a year now without a single bug report.
note that not everyone uses the latest rsyslog build on the latest versions of the distros. I would have to go back through the mailing list to see the bug reports, but there were a bunch of them, which is why we reverted back to creating the PID file by default.

I'd be honestly interested in such bug reports, because the only breakage I'm aware of is in the Ubuntu rsyslog package. They messed up the rsyslog package by not updating the logrotate script when they stopped creating a PID file.

not all distros use systemd (OpenWRT is a significant one that doesn't)

Irrelevant here as this change discussed here is specifically for distros using systemd.

[davidelang]

On Mon, 29 Apr 2019, Michael Biebl wrote: > not all distros use systemd (OpenWRT is a significant one that doesn't) Irrelevant here as this change discussed here is specifically for distros using systemd.

How is this not relevant? you are asking us to change the default for compiling rsyslog, aren't you? A change of the default will affect all distros. We have different configs for compiling the packages, but not different defaults in what the options do. I also note that if someone has a logrotate config that they have brought forward from older versions of Debian, then it will still depend on the pid file, even if Debian how offers a different way to do signal the program. Other than your personal dislike of pid files, what is the harm in continuing to create one? It doesn't break your new way of signaling the program, you have a config option to turn it off if you don't want it. David Lang

[mbiebl]

How is this not relevant? you are asking us to change the default for compiling rsyslog, aren't you? A change of the default will affect all distros.

This here is for rsyslog-pkg-ubuntu, i.e. rsyslog packages that are provided by Adiscon for Ubuntu.
I'm saying that https://github.com/rsyslog/rsyslog-pkg-ubuntu/blob/master/rsyslog/cosmic/master/debian/patches/02-rsyslog-systemd.patch is a bad idea and should be dropped.

[mbiebl]

to be specific: The rsylogd.pid part of that patch.
Then again, I'm no fan of /etc/default files either, so probably the whole patch could go.

[selivan]

How is this not relevant? you are asking us to change the default for compiling rsyslog, aren't you? A change of the default will affect all distros.

This here is for rsyslog-pkg-ubuntu, i.e. rsyslog packages that are provided by Adiscon for Ubuntu.
I'm saying that https://github.com/rsyslog/rsyslog-pkg-ubuntu/blob/master/rsyslog/cosmic/master/debian/patches/02-rsyslog-systemd.patch is a bad idea and should be dropped.

This will break at least one thing in my production configuration, I posted simplified version in upper comments. It will probably break somebody's else configuration. And it will even break the ppa rsyslog package itself, because it comes with logrotate config using invoke-rc.d rsyslog rotate.

Oh yeah, and it will make harder for people to maintain configurations working both on Linux and *BSD, which does not use systemd.

And I completely miss the whole point of making this calamity.

[mbiebl]

If you don't know why pid files are bad, I suggest you read up on that topic.

[davidelang]

On Mon, 29 Apr 2019, Michael Biebl wrote: If you don't know why pid files are bad, I suggest you read up on that topic.

Let me re-phrase the question. Why are pid files so bad that it's worth breaking anyone's systems who are still configured to use them? I'm not calling for eliminating of the ability to not use the pid file, I am just talking about the ability to use the pid file in addition to your other methods. David Lang

[mbiebl]

Why are pid files so bad that it's worth breaking anyone's systems who are still configured to use them?

I'd argue that the upstream defaults should be sane. If there are user setups which need a pid file for whatever reason, they are free to configure their system this way.

[mbiebl]

(or better, update their configuration to not rely on pid files)

[rgerhards]

I don't like to go into systemd politics. This here is the repository Adiscon maintains primarily for it's customers - those folks that pay to make it happen. So in contrast to the main project we have a very clear focus here: whoever pays decides how things should be. Our customers have clearly told us they want the pid file. And so the patch will remain.

I do not say this is the best way to do it. I say this is the way folks pay us to do it.

[rgerhards]

I should also mention that I personally tend to say that "no pid file" is a good idea. But from a pragmatical PoV I really don't care that much and avoid hassle if possible.

[atc0005]

Sorry for adding any noise, but I had to use rsyslog's "debug on demand" feature today and noticed that the docs reference use of a pid file to determine the current process id:

https://www.rsyslog.com/doc/v8-stable/troubleshooting/debug.html#getting-debug-information-from-a-running-instance

Example:

kill -USR1 $(cat /var/run/rsyslogd.pid)

Intent of this post:

Note that if the PID files are not created by default then that example would need to be updated to use a different approach of retrieving the PID.