Enable serial console by default for pxe

[stbenjam]

If a problem happens in IPA, it can help to use the serial console to
debug. We can also use the serial console to log the output in CI
systems.

We specify 2 console options, one for the serial console, and one for
the graphical display. According to the kernel docs, the last devices
will be used for /dev/console, but output will appear on all of them[1].
tty0 is last so if there's an emegency prompt needed it'll be put
on the graphical console by default.

[1] kernel.org/doc/html/v5.5-rc2/admin-guide/serial-console.html

[stbenjam]

/assign @derekhiggins

[derekhiggins]

/lgtm

[metal3-io-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: stbenjam
To complete the pull request process, please assign derekhiggins
You can assign the PR to them by writing /assign @derekhiggins in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[stbenjam]

/hold

Based on @juliakreger's comment above

[metal3-io-bot]

New changes are detected. LGTM label has been removed.

[stbenjam]

/test-integration

[hardys]

I tested this and while it does output some of the console content seen on the VGA console, it doesn't contain the IPA log output AFAICS - are we missing something to add that?

The kernel CLI looks like this in /var/log/libvirt/qemu/ostest_master_0-serial0.log (testing with dev-scripts):

[    0.000000] Command line: deploy_kernel selinux=0 troubleshoot=0 text nofb nomodeset vga=normal console=ttyS0 console=tty0 ipa-debug=1 ipa-api-url=http://[fd00:1101::2]:6385 BOOTIF=00:9e:6f:e6:9c:0a initrd=deploy_ramdisk

However in inspector.ipxe I see ipa-inspection-collectors=default,extra-hardware,logs systemd.journald.forward_to_console=yes - any ideas why those don't show up in the cmdline above?

[derekhiggins]

I tested this and while it does output some of the console content seen on the VGA console, it doesn't contain the IPA log output AFAICS - are we missing something to add that?

The kernel CLI looks like this in /var/log/libvirt/qemu/ostest_master_0-serial0.log (testing with dev-scripts):

[    0.000000] Command line: deploy_kernel selinux=0 troubleshoot=0 text nofb nomodeset vga=normal console=ttyS0 console=tty0 ipa-debug=1 ipa-api-url=http://[fd00:1101::2]:6385 BOOTIF=00:9e:6f:e6:9c:0a initrd=deploy_ramdisk

However in inspector.ipxe I see ipa-inspection-collectors=default,extra-hardware,logs systemd.journald.forward_to_console=yes - any ideas why those don't show up in the cmdline above?

The ironic images can be used to do inspection in two ways, the one we are using is managed by ironic, in this case ironic is
using the kernel params configured in ironic.conf (while inspector.ipxe isn't used). inspector.ipxe would be used if a node is powered on without ironic writing out a PXE config for it (kind of a fallback), we may have been using it at some stage in the past, or maybe for discovery.

I've tried a few different configurations trying to get IPA to output to both the VGA console and ttyS0 (so it could be logged), here is my understanding

Multiple console= entries on the command line result in the kernel messages being logged to multiple devices but the last one on the line is also known as /dev/console

systemd.journald.forward_to_console=yes causes journal messages to be output to /dev/console

So if we have console=ttyS0 console=tty0 systemd.journald.forward_to_console=yes then journal logs will only be logged to tty0 while kernel messages will be logged to both ttyS0 and tty0

The only way I have managed to get IPA logs to output to both tty0 and ttyS0 is to set systemd.journald.forward_to_kmsg=True in this case journal message are sent to kmsg but for some reason tty0 get all of the log entries but ttyS0 get some IPA log entries and not others ( I've yet to figure this out) ....

[metal3-io-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues will close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[metal3-io-bot]

Stale issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle stale.

/close

[metal3-io-bot]

@metal3-io-bot: Closed this PR.

In response to this:

Stale issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle stale.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.