coreos-printk-quiet.service: Lower kernel log level to 4

Closes: coreos/fedora-coreos-tracker#1244

A lot going on for this simple service.  See the tracker issue above
for more info, but briefly:

Anaconda has historically injected `quiet` into the kernel command
line in many cases, and this suppresses *both* kernel and systemd
output.  On computers in general, but particularly many bare metal
servers, the serial console can be slow.  This can cause reliability
issues.

However for servers, we usually *do* want to see informational
output when they boot.  For example, today the kernel "mitigations"
information for hardware vulnerabilities is output.

This change is a compromise; we boot up at the kernel's default
verbosity level (which for Fedora and derivatives is the upstream 7),
but switch to 4 very early on in the real root.  At that point, we've
gotten most of the boot time output, and our initramfs is not
extremely performance sensitive right now.

Also, we explicitly only *lower* the output log level, and only if
there isn't explicitly `debug` on the kernel command line.

overlay.d/05core/usr/lib/systemd/system-preset/40-coreos.preset

@@ -5,6 +5,8 @@ enable console-login-helper-messages-gensnippet-os-release.service
 enable console-login-helper-messages-gensnippet-ssh-keys.service
 # CA certs (probably to add to base fedora eventually)
 enable coreos-update-ca-trust.service
+# Set kernel console log level
+enable coreos-printk-quiet.service
 # https://github.com/coreos/ignition/issues/1125
 enable coreos-ignition-firstboot-complete.service
 # Delete Ignition config from provider on platforms where it's possible

overlay.d/05core/usr/lib/systemd/system/coreos-printk-quiet.service

@@ -0,0 +1,27 @@
+[Unit]
+Description=CoreOS: Set printk To Level 4 (warn)
+Documentation=https://github.com/coreos/fedora-coreos-tracker/issues/1244
+# We can run right after `/proc` being mounted at least
+DefaultDependencies=no
+# We run as early as possible; the only dependency we have really
+# is the implicit After=systemd-journald.socket injected by the
+# default of our stdout writing to the journal.
+Conflicts=shutdown.target
+Before=sysinit.target shutdown.target
+# We want this service to read what we wrote
+Before=systemd-sysctl.service
+# Relatedly, we don't want to override an explicitly specified kernel argument
+ConditionKernelCommandLine=!debug
+ConditionKernelCommandLine=!quiet
+ConditionKernelCommandLine=!loglevel
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+# We need to make /run/sysctl.d if it doesn't exist and also
+# ensure it has a SELinux label that works for systemd-sysctl.service.
+# Then we just generate a sysctl file which is read by systemd-sysctl.service.
+ExecStart=/bin/bash -euo pipefail -c 'mkdir -p /run/sysctl.d && chcon --reference=/etc/sysctl.d /run/sysctl.d && echo "kernel.printk = 4" > /run/sysctl.d/01-coreos-printk.conf'
+
+[Install]
+WantedBy=sysinit.target

tests/kola/logging/data/commonlib.sh

@@ -0,0 +1 @@
+../../data/commonlib.sh

tests/kola/logging/printk

@@ -0,0 +1,15 @@
+#!/bin/bash
+# kola: { "exclusive": false }
+# Verify default console log level; xref https://github.com/coreos/fedora-coreos-tracker/issues/1244
+
+set -xeuo pipefail
+
+. $KOLA_EXT_DATA/commonlib.sh
+
+printk=$(cat /proc/sys/kernel/printk)
+
+if ! [[ "$printk" =~ ^4 ]]; then
+    fatal "printk: expected console log level 4, found ${printk}"
+fi
+
+ok "Found expected printk value "