audit messages are flooding the console

[lucab]

On a freshly booted instance, the console is quickly filled with audit messages from the kernel. This makes spotting the <INTERFACE>:<IPs> harder, and any interactive console usage very messy.

This seems to be caused by our console level being very low (7 is debug):

$ sysctl kernel.printk
kernel.printk = 7       4       1       7

I am not sure where this is coming from, though. For the moment, it can be work-arounded with the following snippet in /etc/sysctl.d/10-kernel-printk.conf:

kernel.printk=4

OS info

State: idle
AutomaticUpdates: disabled
Deployments:
● ostree://fedora:fedora/x86_64/coreos/testing
                   Version: 30.20190716.0 (2019-07-16T15:00:38Z)
                    Commit: b25e7016d621c9d308747ccc4a886224488aa506290c0f8246f3c15216ba2a5c
              GPGSignature: (unsigned)

[dustymabe]

are you saying we should use a different value for CONFIG_CONSOLE_LOGLEVEL_DEFAULT in the fedora kernel configs ?

[dustymabe]

FTR 7 4 1 7 seems to be the default on debian 10 (buster) as well

[arithx]

I'd bring up that notably having some debugging messages printed on the console is useful even in production use cases. Specifically when interacting with clouds one of the primary ways of fetching information about a node that you cannot SSH into is using the built-in cloud APIs to fetch the console. This is especially important in the case of Ignition failures.

I do agree that some of the audit spam is not optimal but that might be something we want to look into directly rather than lowering the entire console log level.

[dustymabe]

I do agree that some of the audit spam is not optimal but that might be something we want to look into directly rather than lowering the entire console log level.

👍

benjamin also mentioned that 7 4 1 7 was the default on CL too

[lucab]

Sorry for the confusion, I am not suggesting to raise the defaults. At least not without further investigation first. This was just to track that I had some friction in getting the IP from the serial console due to audit spam, and that tweaking the sysctl helped.

[dustymabe]

ok I'm working on setting up some sort of interactive TUI experience for our live ISO installer and having audit messages periodically print to the screen is not a good user experience. From what I can tell there are a few options to fix this issue:

• change kernel logging level defaults (as mentioned earlier in this ticket)
• set audit=0 on the kernel command line
• add auditd to the host

I'm not sure how bad it would be to disable auditing (audit=0). If there is a silver lining to adding auditd to the host it would be that we get back an easy way to find SELinux denials (via ausearch -m avc). Currently the audit rpm has one dependency that would get brought in but it's not one we need and I opened a PR to remove it altogether.

[lucab]

Followup from IRC:

• we generally like getting audit events
• we don't hardcode any audit= kargs on the kernel command-line
• systemd-journald correctly receives the audit events and they are available in the journal (with _TRANSPORT=audit)
• it looks like journald itself enables auditing
• as we are not running anything else (i.e. no auditd), there may be some missing logic in journald that results in the kernel kprint-ing event to console

As a next step here, we are going to check this behavior with systemd folks.

[lucab]

Tracked at systemd/systemd#15324.

[lucab]

According to the above systemd ticket, we pinpointed this to a limitation / missing feature in the kernel audit subsystem.
The recommendation is that right now the sanest path is to also ship auditd, possibly with the caveat that we may want to drop it when the kernel gets fixed in the future.

[dustymabe]

👍 - there is a lot of context in that ticket that provides useful historical information. Thanks for opening that issue @lucab and following up here.

I'd propose we add auditd with a comment about the kernel issue (a link where we could track the kernel issue would be real nice). I shamelessly don't mind having auditd myself because of the ausearch -m avc functionality to search for and help diagnose SELinux denials.

[bgilbert]

It appears there might be some movement in linux-audit/audit-kernel#102.

Meanwhile, if we ship auditd as a short-term workaround, we'll inevitably get objections once we try to remove it again. Would it make sense to write a small daemon that registers with the kernel using auditd's interface and then drops all messages?

[bgilbert]

I did some digging. Container Linux ships auditctl but not auditd, along with an audit-rules.service which loads the audit rules on boot. The default audit rules disable most auditing, but users can change those rules and the resulting messages will go to the journal.

On Fedora CoreOS, though, we're not shipping a way to load audit rules into the kernel, so we don't have a coherent audit solution right now. We should at least ship auditctl. It's in the same package as auditd, so maybe we should ship that too and not worry about it. Alternatively, we could filter out auditd and add something like audit-rules.service. In the latter case, if we enable some auditing by default, we might also want to ship a boilerplate auditd replacement to avoid the logging issue until the kernel is fixed. It shouldn't be too hard to write one.

[lucab]

@bgilbert thanks for the trip in CL land. We have indeed a larger story to untangle wrt loading audit rules. I previously opened linux-audit/audit-userspace#111 in order to get support for fragments. I'm both wary of just sticking to current auditd status and writing a scratch-new replacement.

(Drifting away from this ticket, I think it'd makes sense to split auditd / augenrules / auditctl binary packages anyway. If nobody opposes, I'm going to ask for that in bugzilla)

[dustymabe]

my $.02: let's just ship auditd and spend our time solving other problems. I don't feel super strongly about it though.

[jamescassell]

I feel we should ship auditd, as many security compliance regimes require such functionality, so it'd end up layered otherwise in many cases.

[jlebon]

I'd be very wary of pulling in initscripts. It's a bunch of legacy stuff which has caused lots of issues for us in the past.

[dustymabe]

I'd be very wary of pulling in initscripts. It's a bunch of legacy stuff which has caused lots of issues for us in the past.

I agree. If we ship auditd I think we should rip out initscripts.

[dustymabe]

I'd be very wary of pulling in initscripts. It's a bunch of legacy stuff which has caused lots of issues for us in the past.

I agree. If we ship auditd I think we should rip out initscripts.

coreos/fedora-coreos-config#340 should help us make sure it doesn't get pulled in.

[dustymabe]

This need to fix this is increased right now because we are trying to support a workflow where a user can interactively do an install using the Live ISO and use TUI interfaces for configuring networking. The increased user experience provided by a TUI is completely lost if we have audit messages scrolling on the console every 10 seconds or so.

I have opened a few change requests that are options we have for fixing this problem immediately. They are meant to spark discussion and hopefully get us to a point where this problem is fixed for the user trying to run a TUI and not have audit messages scrolling across their TUI canvas.

• live: disable audit on Live ISO
• manifest: include audit rpm
• overlay: lower kernel console logging level on Live ISO

Maybe we can discuss these options (or any other viable short term option) in the meeting today to see if we can make progress.

EDIT: added a 3rd option to the bulleted list.

[dustymabe]

I added a 3rd option to the bulleted list in #220 (comment)

[dustymabe]

We talked about this in the community meeting today:

13:01:58  dustymabe | #agreed We have some outstanding issues that we'd like to try to push in
                    | audit upstream before we include that package in FCOS. For now we will
                    | quiet the kernel log level printed to the console of the machine and
                    | continue to push upstream for changes to make things more ideal

This is the overlay: lower kernel console logging level on Live ISO option from #220 (comment).

I also agreed to open a ticket to discuss audit upstream features and packaging changes we'd like to see. It is here: #461

[egeturgay]

added our use case of audit messages in journald in #461 but thought I should elaborate the fact that we use the logging in journald for compliance purposes by using log delivery agents parsing journald[audit] messages.

As there is no auditd.log on Container Linux, that is the only way for us to be able to parse audit messages. Bringing auditd would solve this as suggested but we haven't found an disadvantage of getting the logs from journald vs getting it from auditd.log in terms of compliance requirements.

In addition to the above, FCOS currently does not ship auditctl and augenrules (unlike Container Linux) which prevents us using file integrity monitoring rules on FCOS.

We would appreciate any estimation or timelines in concluding the above to have enough time to look into alternatives if this is unlikely to land on to FCOS before the Container Linux EOL dates.

[dustymabe]

For those hitting this that don't care about a lowered kernel log level you can workaround with:

$ cat fcct_silence_audit.yaml
variant: fcos
version: 1.1.0
passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ssh-rsa AAAA...
storage:
  files:
    - path: /etc/sysctl.d/20-silence-audit.conf
      contents:
        inline: |
          # Raise console message logging level from DEBUG (7) to WARNING (4)
          # so that audit messages don't get interspersed on the console that
          # may frustrate a user trying to interactively log in.
          kernel.printk=4

[dustymabe]

So this issue should now be solved in multiple ways. First, we now lower the printk level after boot. Second, in F39+ we're going to include auditd.

I'll close this out now.