Can't output systemd.show_status=true to both tty0 and ttyS0

[thomasgoirand]

Submission type

• Bug report
• Request for enhancement (RFE)

NOTE: Do not submit anything other than bug reports or RFEs via the issue tracker!

systemd version the issue has been seen with

230

Used distribution

Debian testing (aka: Stretch)

In case of bug report: Expected behaviour you didn't see

systemd logs its output to both tty0 and ttyS0

In case of bug report: Unexpected behaviour you saw

systemd only logs to ttyS0

In case of bug report: Steps to reproduce the problem

Hi Lennart and friends! These days, I maintain the official Debian OpenStack image, available at:
http://cdimage.debian.org/cdimage/openstack/

In this image, I'm setting logging to go on both ttyS0 and tty0, so that the logs can be seen in both the "nova console-log" command output (which is the same as what's shown in the openstack-dashboard web GUI tab), and on the VNC console. For this, I'm setting this command line in Grub:

console=tty0 console=ttyS0,115200 earlyprintk=ttyS0,115200 consoleblank=0 systemd.show_status=true

The kernel understands this very well, and it outputs to both tty0 and ttyS0. If running sysv-rc and bootlogd, it works well as well: in Debian Wheezy, I added a patch for bootlogd to output to multiple console (this patch was sent to the Debian BTS). However, systemd doesn't seem to understand this, and has the behavior of the older bootlogd, which only outputs to the last set console= parameter. The result is that it logs only to ttyS0. As a consequence, someone using the OpenStack VNC console waiting for the virtual machine to boot could be fooled into believing something went wrong, when in fact, everything is working as expected. What's even weirder, is that I saw some output like this on the tty0:

[
[
[

ie: some kind of a start of a [ ok ], but nothing after the first char, and some seemingly empty lines before and after. This really looks like a bug to me.

I'm not sure if this is really a bug or a RFE, but this needs to be fixed anyways, as this would be super helpful for debugging (and not only in the case of the cloud).

Last: I can provide you with some access to some OpenStack servers if you need to test stuff, or I could test fixes that you would provide (a patch against the latest systemd version in Debian testing would do, for example...). You can get in touch with me on IRC if you do need that (my nick there, on both OFTC and Freenode, is zigo, not sure if you remembered me from Debconf in Switzerland...).

Thanks in advance for caring about this issue.

[poettering]

So, we only write to /dev/console, really, which should end up on all kernel consoles, last time I looked. Are you saying that "echo foo > /dev/console" doesn't output the specified string on all consoles for you?

[neingeist]

On my system (Debian stable, though), "echo foo > /dev/console" only goes to the serial console (ttyS0). So systemd only outputs on the serial, not on the VNC display of the VM.

Kernel console config:

# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-3.16.0-4-amd64 root=/dev/mapper/vg_pxe-root ro console=tty0 console=ttyS0,115200

[neingeist]

I also checked Rawhide (kernel 4.8.0-0.rc1.git3.1.fc26.x86_64, systemd-231-3.fc26.x86_64); There, systemd output goes to both consoles! ("echo foo > /dev/console" still goes to serial only, though.)

[keszybz]

I also checked Rawhide (kernel 4.8.0-0.rc1.git3.1.fc26.x86_64, systemd-231-3.fc26.x86_64); There, systemd output goes to both consoles!

Thanks for testing. Closing as this seems to be fixed now.

[rudimeier]

Does anybody know which particular patch fixed this issue?

[stapelberg]

http://www.tldp.org/HOWTO/Remote-Serial-Console-HOWTO/configure-kernel.html states:

When multiple consoles are listed output is sent to all consoles and input is taken from the last listed console. The last console is the one Linux uses as the /dev/console device.
[…]

console=tty0 console=ttyS0,9600n8
Kernel messages will appear on both the first virtual terminal and the serial port. Messages from the init system and the system logger will appear only on the first serial port. This can be slightly confusing when looking at the attached monitor: the machine will appear to boot and then hang. Don't panic, the init system has started but is now printing messages to the serial port but is printing nothing to the screen. If a getty has been configured then a login: prompt will eventually appear on the attached monitor.

That’s also what I can observe on Debian unstable as of today (2018-01-08), i.e. the attached monitor only displays kernel messages, and systemd messages are only visible on the serial console.

I’d be curious why some systems differ in that regard. Are they configured differently? How?

[vincentbernat]

Maybe the content of /proc/consoles would help to understand? On Debian, I have:

ttyS1                -W- (EC p a)    4:65
tty0                 -WU (E  p  )    4:1

The C means CONSDEV and is documented as Last on the command line.

[stapelberg]

I started Fedora-Minimal-armhfp-27-1.6-sda.raw.xz on my Raspberry Pi 3. The /proc/consoles output is:

# cat /proc/consoles
ttyS1                -W- (EC p a)    4:65
tty0                 -WU (E  p  )    4:1

Note that writes to /dev/console only show up on the serial console, even though boot messages show up on both. I’m not sure where that difference comes from.

[stapelberg]

Figured it out with help from @hannes: Fedora uses Plymouth by default, which takes care of surfacing the messages on tty1 (HDMI on the Raspberry Pi). As soon as you dnf remove plymouth and rebuild the initrd, boot messages show up only on the serial port, while kernel messages still show up on both devices (just like in Debian).

If my reading of https://github.com/torvalds/linux/blob/master/Documentation/admin-guide/serial-console.rst is correct, this is the intended behavior: /dev/console does not write to all specified devices, just to the preferred console (the one with the C flag in /proc/consoles). The kernel itself always prints to all devices specified via console=, independent of /dev/console.

So I think the take-away is: if you want the messages to appear on multiple outputs, use plymouth (or similar).

[rudimeier]

On Saturday 10 March 2018, Michael Stapelberg wrote: So I think the take-away is: if you want the messages to appear on multiple outputs, use plymouth (or similar).

The thing is that before my favorite distro unfortunately switched to systemd everything was fine. Installing plymouth is a bit too much for my use case ... just need a minimal system which looks somehow equal on the monitor and serial console.

[stapelberg]

How did the init system you used before systemd implement this? Did it do more than writing to /dev/console?

[rudimeier]

I guess it was a distro specific thing, openSUSE, blogd or something!? But since systemd eats all the nice distro specific differences, and distro developers don't care about losing their nice features ... this thing seems to be gone. But IMO this issue is a musthave and probably easy to implement within systemd.

[jamespharvey20]

Can someone re-open this? This is not fixed, but only worked around if Plymouth is installed.

As @stapelberg explains, systemd writes only to /dev/console, which only goes to the preferred console (the one with the C flag in /proc/consoles.)

systemd messages are only sent to both consoles if Plymouth is installed (such as on most Fedora - but not on minimal) which appears to step in and make everything written to all consoles.

Without Plymouth, a kernel line such as console=ttyS0,9600n8 console=tty0 sets tty0 as the primary console. Kernel messages are written to all consoles, so show up on both, but systemd messages are only written to the primary console, so only show up on tty0. (Again, if Plymouth is installed, systemd messages seem to be written to all consoles.)

[jamespharvey20]

Perhaps in early boot, /proc/consoles could be parsed to obtain all consoles, for systemd to write to all of them rather than /dev/console directly. This would leave out consoles started other than through the kernel line, but would be much better. Could either write in a way to refresh the consoles, or maybe changes to /proc/consoles can be monitored - I have no idea on that.