flatcar 2605.9.0+ with systemd 246 journal logs produced is not readable by older versions

[mindw]

Description

  If the $SYSTEMD_JOURNAL_KEYED_HASH boolean
  environment variable for systemd-journald.service is set to 0 this
  new hardening functionality may be turned off, so that generated
  journal files remain compatible with older journalctl
  implementations.

Impact

This affects all current dockerized systemd log collection methods. Specifically the reference k8s one.

Environment and steps to reproduce
We have a custom setup - but the below should work. The links below should provide a lot better reference.

1. Set-up:
   a. setup a k8s cluster using flatcar https://typhoon.psdn.io/flatcar-linux/aws/
   b. install the fluentd elastic addond https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch
2. Task: check kibana for systemd logs
3. Action(s):
   a. [ requested the start of a new pod or container ]
   b. [ container image downloaded ]
4. Error: there are no host systemd logs collected.

Expected behavior
systemd logs are collected

Additional information
bottlerocket-os/bottlerocket#1200
one reference workaround:
bottlerocket-os/bottlerocket#1206

Even if this ticket isn't handled - at least it hopefully save some people precious time.

[jcrowthe]

We ran into this issue as well. In our case, we use a fluent-bit daemonset to ship the journal from all kubernetes hosts into splunk. In moving to Flatcar 2605.10.0 we stopped receiving journal logs. We fixed this by making two changes, both of which were necessary:

# Set the mentioned environment variable
$ mkdir -p /etc/systemd/system/systemd-journald.service.d/
$ vim /etc/systemd/system/systemd-journald.service.d/test.conf
[Service]
Environment=SYSTEMD_JOURNAL_KEYED_HASH=0

# Disable journald log compression, which now defaults to zstd rather than lz4. No way to change compression algorithms today.
$ mkdir -p /etc/systemd/journald.conf.d/
$ vim /etc/systemd/journald.conf.d/test.conf
[Journal]
Compress=no

Update: FYI if you need to update an existing host, you will need to erase the journald log directory before the compression settings will take effect.

We'll be opening a ticket against fluent-bit as well here shortly and will link back here. Thanks @mindw for tipping us off to this!

[dharmab]

Fluent Bit issue: fluent/fluent-bit#2998

[ltagliamonte-dd]

@jcrowthe tried this looks not working, i tried also restarting the server still no logs collected by fluent-bit

[surajnarwade]

any update on this issue ?

[tormath1]

Hi @surajnarwade, following what it has been said into the issue it seems there are two options:

• wait for the upstream to fix this issue (see dockerfiles: Update the systemd libs used to build docker image fluent/fluent-bit#3177)
• patch systemd in the way bottlerocket did (see disable keyed hashes for the systemd journal bottlerocket-os/bottlerocket#1206)

Seems like we don't have much activity on fluent-bit PR - let us explore the other option !

[t-lo]

Hey folks,

As has been mentioned above, keyed hashes can be deactivated by setting SYSTEMD_JOURNAL_KEYED_HASH=0 as documented in @jcrowthe 's comment above.

Keyed hashes are a security / integrity feature employed by journald to mitigate hash collision attacks on the journal. Switching off this feature may expose hosts to journal integrity / manipulation issues - which can be critical if you use the journal for e.g. security auditing.

Switching off keyed hashes however does seem to be the only available stop-gap solution to keep fluentd log collection / aggregation remaining functional - at least until upstream picks up the PR and updates their log parser.

Considering Flatcar's mission statement to be a fully open source, minimal-footprint, secure by default and always up-to-date Linux distribution for running containers at scale I do not feel however that disabling keyed hashes by default (e.g. using the Bottlerocket patch) would be a good service to the wider Flatcar user base.

[pothos]

Upstream merged the change to support the newer format, I think we can close this as there is nothing to be done anymore (assuming that an upstream release comes with it goes out).