cloud.ks: Add tty0 console output for ignition logging

[Rahuls0720]

Many platforms, like QEMU, use tty0 for console output. This PR adds tty0 console output to allow ignition logging/debugging.

Jira card: https://projects.engineering.redhat.com/browse/COREOS-202

[jlebon]

Have you confirmed this fixes the issue? My memory is a bit fuzzy on this, but I remember looking into this some time ago and found systemd/systemd#3403. Basically, the order in which the console= arguments are given matters. I wonder if we also need systemd.show_status=1 for this?

[ashcrow]

@jlebon that's a good find. I believe @Rahuls0720 is attempting to build a local version of RHCOS with these changes. We chatted a bit about it yesterday but I wasn't able to help as much as I would have liked due to meetings.

@Rahuls0720 were you able to successfully create a local version of RHCOS with your change to test?

[Rahuls0720]

I was able to run it locally using vagrant, but Ignition logs output just fine to both tty0 and tty1. So the card doesn't apply to vagrant. I will test it on qemu today hopefully

[crawford]

@Rahuls0720 were you looking at the VGA output or serial output when you tested vagrant? Which virtualization backend were you using with vagrant?

[Rahuls0720]

I used virsh to view the output. Oh, does it look at serial? :/

[crawford]

From the kernel docs: "Note that you can only define one console per device type (serial, video)."

Just replace tty1 with tty0.

[crawford]

/hold

[ashcrow]

I spoke with @crawford today and it sounds like we should go with setting to tty0 and ttyS0.

@crawford Agree?

[crawford]

@ashcrow As mentioned in the last comment, I think we should go with tty0 and ttyS0. I don't know what order they should be in (there are conflicting docs regarding the ordering).

[Rahuls0720]

Does anyone know how to log the tty0 console output for either qemu or vagrant? I'm looking for something like -nographic or -serial file:FILENAME qemu flags that allows me to view and log the tty0 output (to ensure ignition ran).

@crawford mentioned one method of ensure ignition ran/logged is using an invalid url in the ignition config, but boot up halts at an emergency shell

[ashcrow]

/cc @arithx

[crawford]

@Rahuls0720 Use an Ignition config that references something from an address that is unreachable. Something like this should be interesting:

{
  "ignition": {
    "version": "2.2.0",
    "config": {
      "append": [{
        "source": "http://localhost"
      }]
    }
  }
}

I haven't validated that or tested it.

[arithx]

In kola we use -chardev file,id=log,path=<path_to_log> to capture the serial console on QEMU (https://github.com/coreos/mantle/blob/master/platform/machine/qemu/cluster.go#L189) but I'm not sure about specifically capturing the tty0

[Rahuls0720]

@jlebon Do you know the reason tty1 is being used?

Also, the last console set in the kernel command line arg is the one systemd logs everything to (this console has the C flag in /proc/consoles). The other consoles only get the kernel logs. If tty1 is needed, console=tty1 console=ttyS0,115200n8 console=tty0 should do the trick. Otherwise console=ttyS0,115200n8 console=tty0.

Let me know your thoughts on this!

[jlebon]

@Rahuls0720 It's just inherited from the Fedora kickstart ours is based on.

So, reading systemd/systemd#3403 again, it sounds like there's no way right now to get systemd messages to go to all consoles without Plymouth? If that's the case, then I'd vote for ttyS0 rather than tty0 being the last argument. E.g. cloud providers make it easy to access those logs and locally we can use virsh console (which I've found super useful exactly wrt. debugging ignition).

[Rahuls0720]

I'm a bit confused on Plymouth. Plymouth is part of the initramfs and RHCOS doesn't include Plymouth. So why do boot up messages still appear on both ttyS0 and tty0?

Can you use virsh console for tty0? I thought it only displays serial output

[jlebon]

Right, virsh console connects to the serial console. Since it's already set as the last console= argument, systemd messages show up there. I thought @crawford wanted to make them appear on both of them. Though reading the original Jira ticket now, it looks like the goal was to see journal output such as ignition, which definitely works today with virsh console.

I also tried running QEMU directly and got ignition log messages using e.g. -chardev file,id=log,path=/tmp/log.txt -serial chardev:log like kola does. @crawford Does that not work for you?

[ashcrow]

Related coreos/bugs#2136

[Rahuls0720]

@crawford @ashcrow @jlebon I believe this should solve the problem.

Just to ensure everyone is on the same page, I replaced tty1 with tty0 and moved tty0 as the last console in the kernel args. My understanding is tty0 should look identical to ttyS0. Currently tty0 looks identical to ttyS0, except that tty0 has ignition logs . The journald.conf option 'forwardtoconsole' is included in RHCOS, but doesn't work with ignition which is why tty0 and ttyS0 differ. Please see coreos/bugs#2136 for details about this ignition bug.

[crawford]

@jlebon I was watching the VGA console (tty0).

@Rahuls0720 We need to use ForwardToKMsg so it shows up on both.

[Rahuls0720]

@jlebon @crawford Can you give this a lgtm label if no other changes are needed?

[crawford]

/lgtm

[crawford]

/hold cancel

[crawford]

/ok-to-test

[cgwalters]

https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/OI7BOWZSXDJM5W2KVDK3BBEN6S7ASLZC/