systemd input plugin cannot read zstd compressed/hash collision hardened journal files in systemd >= 246

[dharmab]

Bug Report

Describe the bug

systemd 246 changed the default format of journal files:

    * systemd-journald gained support for zstd compression of large fields
      in journal files. The hash tables in journal files have been hardened
      against hash collisions. This is an incompatible change and means
      that journal files created with new systemd versions are not readable
      with old versions. If the $SYSTEMD_JOURNAL_KEYED_HASH boolean
      environment variable for systemd-journald.service is set to 0 this
      new hardening functionality may be turned off, so that generated
      journal files remain compatible with older journalctl
      implementations.

See https://github.com/systemd/systemd/blob/v246/NEWS#L323-L331 and https://www.freedesktop.org/software/systemd/man/journald.conf.html.

The systemd input plugin is unable to read these files unless SYSTEMD_JOURNAL_KEYED_HASH=0 (to disable the hash table hardening) and Compress=false (to disable compression of large fields).

To Reproduce

• Steps to reproduce the problem:

1. Configure a systemd input plugin in fluent bit
2. Run fluent bit on a system using systemd >= 246 without SYSTEMD_JOURNAL_KEYED_HASH=0 and Compress=false
3. Observe that no records are emitted to output plugins and that metrics show no records/bytes are being ingested by the input plugin.

Expected behavior

The systemd input plugin should be able to parse any valid journal file format. Alternatively, the behavior and workaround should be documented in the input plugin's documentation.

Screenshots

Your Environment

• Version used: 1.5.4
• Configuration:
• Environment name and version (e.g. Kubernetes? What version?):
• Server type and version:
• Operating System and version: Flatcar Linux 2605.9.0+
• Filters and plugins: systemd input

Additional context

This was tricky to spot since Fluent Bit did not log any errors attempting to read the new files. The best way to detect this is to examine the input plugin metrics.

We're running an older version of Fluent Bit but the systemd input plugin hasn't changed significantly since that version.

[dharmab]

Further context:

systemd/systemd@8653185#diff-da05bd703b33dc8898bd760a37cea6eb031d04632f4930f7c5a2cdaede90d8e1

systemd/systemd#16749

[chadcatlett]

FWIW, our solution was to customize the docker file for 1.6.x, switching the builder from buster-slim to buster-backports, so that the container had systemd v247.x shared libraries inside of it instead of v241.x shared libraries.

[xcompass]

FWIW, our solution was to customize the docker file for 1.6.x, switching the builder from buster-slim to buster-backports, so that the container had systemd v247.x shared libraries inside of it instead of v241.x shared libraries.

@chadcatlett Do you mean bullseye-backports? As buster-backports still has v241-7.

▶ docker run --rm -it debian:buster-backports bash
Unable to find image 'debian:buster-backports' locally
buster-backports: Pulling from library/debian
0ecb575e629c: Pull complete
ea88ecbad7fe: Pull complete
Digest: sha256:ee514ad79258245afe9dfa823f0792de04aa979d6caf51b630f004dfeeb5b7e3
Status: Downloaded newer image for debian:buster-backports
root@731182bc6ba0:/# apt update
Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:2 http://deb.debian.org/debian buster InRelease [122 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:4 http://deb.debian.org/debian buster-backports InRelease [46.7 kB]
Get:5 http://security.debian.org/debian-security buster/updates/main amd64 Packages [268 kB]
Get:6 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB]
Get:7 http://deb.debian.org/debian buster-updates/main amd64 Packages [9504 B]
Get:8 http://deb.debian.org/debian buster-backports/main amd64 Packages [453 kB]
Fetched 8922 kB in 2s (3741 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
1 package can be upgraded. Run 'apt list --upgradable' to see it.
root@731182bc6ba0:/# apt search libsystemd-dev
Sorting... Done
Full Text Search... Done
libsystemd-dev/stable 241-7~deb10u6 amd64
  systemd utility library - development files

[chadcatlett]

@xcompass whoops, i forgot one detail. I made all calls to apt-get have -t buster-backports so that it would use the backports repo. It is also possbile to change apt's settings to make it so you don't have to pass in the target release information.

edit:

specifically apt-get install and I also added a apt-get dist-upgrade as well to make it ensure the update was installed.

[xcompass]

Got it. Thanks!

[yasn77]

FYI I've opened a PR about this #3177

[yasn77]

@edsiper Any chance #3177 can be looked at and merged?

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[george-angel]

remove stale

[checkmypi]

Bump to merge PR from yasn77

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[george-angel]

remove stale

[t-lo]

Hello folks,

reaching out to you from the Flatcar Container Linux maintainers team to get an update on this issue.
The issue impacts a number of our users; we maintain a tracking issue in our project.

We understand keyed hashes are a security / integrity feature in journald to mitigate a hash collision vulnerability of earlier versions. Currently, the only workaround available to Flatcar users to keep fluentd working is to switch off this feature, exposing their journals to potential manipulation, and making the journal less usable for e.g. auditing purposes.

I noticed there's a PR open to fix the root cause, updating the systemd libraries in fluentd's docker image to a version which is capable of parsing the new journal. Also, this issue appears to be on the roadmap for the fluent bit 1.7 release.

Is there anything we can help you with to get more traction with ingesting the fix into fluent bit?

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[yasn77]

remove stale

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[sedlund]

Remove kale

[yasn77]

I've updated PR #3177.

As pointed out by @t-lo, in its current state using docker images provided by fluentbit is problematic on systems with newer versions of systemd. This issue and PR have been open for some time. It would be really appreciated in getting some feedback.

Even if my PR isn't accepted, I'd just be happy knowing that something is in the pipeline to fix this issue.

[yasn77]

The ritual remove stale before it rears its ugly head.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.