rsyslog.d/90-google.conf directs logging to /dev/console, but rsyslog doesn't have perm to open

[zbjornson]

Hello! This has been going on for ages but I hadn't taken the time to track it down until now. Our GCE VM logs (Ubuntu 19.10, nearly out-of-box) are filled with these messages:

Apr 18 00:36:45 srv-zgff systemd[1]: Started System Logging Service.
Apr 19 00:01:47 srv-zgff rsyslogd[530]:  [origin software="rsyslogd" swVersion="8.1901.0" x-pid="530" x-info="https://www.rsyslog.com"] rsyslogd was HUPed
Apr 19 00:36:41 srv-zgff rsyslogd[530]: file '/dev/console': open error: Permission denied [v8.1901.0 try https://www.rsyslog.com/e/2433 ]
Apr 19 00:36:41 srv-zgff rsyslogd[530]: action 'action-7-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.1901.0 try h
Apr 19 00:36:41 srv-zgff rsyslogd[530]: action 'action-7-builtin:omfile' resumed (module 'builtin:omfile') [v8.1901.0 try https://www.rsyslog.com/e/2359 ]
Apr 19 00:36:41 srv-zgff rsyslogd[530]: action 'action-7-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.1901.0 try h
# these resumed/suspended lines repeat frequently

That config comes from

compute-image-packages/packages/google-compute-engine/src/etc/rsyslog.d/90-google.conf

Line 6 in 66ca87b

daemon,kern.* /dev/console

and indeed the syslog user doesn't have access

$ ls -l /dev/console
crw--w---- 1 root tty 5, 1 Apr 19 00:01 /dev/console
$ ps aux | grep rsyslog
syslog     530  0.0  0.1 224508  5144 ?        Ssl  Apr18   0:03 /usr/sbin/rsyslogd -n -iNONE
$ groups syslog
syslog : syslog adm

Should there be a udev rule in this package to change permissions on /dev/console?

Looks like this has been happening for quite some time for at least one other user (xenial, bionic): https://irclogs.ubuntu.com/2018/11/14/%23ubuntu.html#t16:07

[kkm000]

I remember there used to be an udev rule file in the old package that changed ownership of /dev/console to root:syslog. It must be somewhere down the commit history here, pre-split.

[slashdd]

At first glance, it looks related to the Privilege Drop in rsyslog that both point to syslog for the user and group.
It no longer matches the ownership/permission of /dev/console.

Looking in Bionic/18.04LTS, /dev/console used to be root:syslog, nowadays it's root:tty

# /etc/rsyslog.conf
$PrivDropToUser syslog
$PrivDropToGroup syslog

I was able to reproduce with the PrivDrop, but I can't without it since it is obsviously always using 'root'.

I'm still investigating, but wanted to give my 2 cents so far.

• Eric

[slashdd]

While I would not suggest getting rid of the privileges drop, unless if it's a debug exercise in a non-production system, like I did.
As a potential workaround, one could possibly make 'syslog' user a member of the 'tty' group (by hand) and validate if it mitigates the problem while it is still under investigation.

[slashdd]

https://launchpad.net/bugs/1890177

[slashdd]

Please see this Ubuntu bug:
https://launchpad.net/bugs/1890177

I'll soon upload a fix in Ubuntu (Focal) that the Ubuntu SRU verification team already pre approved (pre SRU).

Note that during the verification phase of the package, I'll need feedbacks from affected users.

Thanks in advance.

[zbjornson]

Awesome, thank you.

I was mistaken then. I thought this issue was because of the rsyslog conf provided by this GCP guest OS package, but I take it that configuration should work?

[slashdd]

Could someone affected by this bug test the focal-proposed package ? and report the outcome to the bug ?
https://bugs.launchpad.net/ubuntu/+source/rsyslog/+bug/1890177/comments/6

[zbjornson]

@slashdd I'll install it on a few servers today and report there.

[bob-rove]

It seems that the issue is still out there.
I'm running Ubuntu 22.04 LTS on GCE (from provided cloud images) and having the very same issue.
The only problem now is that adding syslog user to tty group doesn't help as stated by links - it is still bleeding:

imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd.  [v8.2112.0]
rsyslogd's groupid changed to 111
rsyslogd's userid changed to 104
[origin software="rsyslogd" swVersion="8.2112.0" x-pid="589" x-info="https://www.rsyslog.com"] start
file '/dev/console': open error: Permission denied [v8.2112.0 try https://www.rsyslog.com/e/2433 ]
action 'action-9-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2112.0 try https://www.rsyslog.com/e/2007 ]
action 'action-9-builtin:omfile' resumed (module 'builtin:omfile') [v8.2112.0 try https://www.rsyslog.com/e/2359 ]
action 'action-9-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2112.0 try https://www.rsyslog.com/e/2007 ]
action 'action-9-builtin:omfile' resumed (module 'builtin:omfile') [v8.2112.0 try https://www.rsyslog.com/e/2359 ]
action 'action-9-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2112.0 try https://www.rsyslog.com/e/2007 ]
...

[pib]

This is still happening on 22.04 cloud images, but adding the syslog user to the tty group does seem fix it. After running systemctl restart rsyslog it starts back up with no errors.

[pib]

There's a udev rule in /usr/lib/udev/rules.d/99-gce.rules that sets the group on /dev/console to syslog, but something is either preventing that rule from applying or resetting the group on boot, it seems.

Running udevadm test /sys/devices/virtual/tty/console triggers the rule and sets the /dev/console group to syslog. After a reboot, the group is back to tty, and restarting rsyslog brings the problem back again.

[hopkiw]

@pib FYI this is a closed issue on a deprecated repository. if there's a current bug you should file an issue on the guest-configs repo https://github.com/GoogleCloudPlatform/guest-configs

[vorakl]

Thanks, Liam ;)