dockerfiles: Update the systemd libs used to build docker image

[yasn77]

The systemd libraries used in the docker image can not read the journal db in newer
versions of systemd. This specifically impacts users that run systemd 246 or
above on their host server:

• systemd-journald gained support for zstd compression of large fields in
  journal files. The hash tables in journal files have been hardened against
  hash collisions. This is an incompatible change and means that journal files
  created with new systemd versions are not readable with old versions. If the
  $SYSTEMD_JOURNAL_KEYED_HASH boolean environment variable for
  systemd-journald.service is set to 0 this new hardening functionality may be
  turned off, so that generated journal files remain compatible with older
  journalctl implementations.

(Taken from https://github.com/systemd/systemd/blob/v246/NEWS)

This commit uses the backported version of the systemd libs, which are 247 for
Debian buster. Fixing it for most distributions.

It might be worth considering having a separate build stage for systemd, that
would give better flexibility to the version shipped in the image.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• [N/A] Example configuration file for the change
• [N/A] Debug log output from testing the change

• [N/A] Attached Valgrind output that shows no leaks or memory corruption was found

Documentation

• [N/A] Documentation required for this feature

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[yasn77]

@edsiper @fujimotos @koleini

Hi, is there any chance this can get looked at? I can rebase if it's something you would merge.

Thanks

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[yasn77]

Bump?

[yasn77]

Bumping again before it goes stale...

I am happy to look at the merge conflicts if I know this has a chance of being merged. But it's been hanging around for a while and still an issue with newer versions of systemd

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[george-angel]

remove stale

[Kuckkuck]

Any update on this? because new Flatcar images with zstd compression for journald logs are rolled out everywhere and fluent-bit is blind. ;-)

[ced0ps]

Taking a look at the failing check it seems that it was a sporadically failing test which was reported here and fixed here

[yasn77]

I've rebased my branch with master. It would be really nice if someone can actually look at this.

[jonkerj]

@edsiper (being commit champion at fluent), could you maybe look into this one? This PR is required to run fluent on recent container linux, which is quite common in Kubernetes-land. We are using @yasn77's branch for quite a while now, would love to see his work upstream.

[yasn77]

The commit message has been updated (+squashed) and signed.

Hope that's OK. I am still not sure why the integration step fails.

[patrick-stephens]

I think you need to update the PR title as well

[patrick-stephens]

It looks like it's assuming an image build for this PR so not sure if there has been an action change that happened which meant it has not been done, @agup006 ? It looks like other PRs have them skipped.

[agup006]

I'm +1 to this, also adding @niedbalski to take a look as he maintains CI/CD/Docker

[niedbalski]

Hello @yasn77

The patch looks good to me.

1. Could you please provide me with a configuration file that exposes this issue? I would like to add an integration test that validates the systemd journal input.

2. "It might be worth considering having a separate build stage for systemd, that
   would give better flexibility to the version shipped in the image." -- Can you elaborate a bit more on this proposal?

Thanks,

[yasn77]

Hi @niedbalski

1. Could you please provide me with a configuration file that exposes this issue? I would like to add an integration test that validates the systemd journal input.

I can't really provide a config file as it's related to the host running a newer version of systemd and not a fluentbit config issue. For example, you can try reading journalctl logs from a fluentbit image running on a flatcar host. Trying this will show that the Docker image doesn't pick up any of the logs.

2. "It might be worth considering having a separate build stage for systemd, that
   would give better flexibility to the version shipped in the image." -- Can you elaborate a bit more on this proposal?

Sure, the current Docker build relies on systemd from Debian. Quite a few hosts running in container environments (such as Kubernetes), will be running distros that most likely will be using newer versions of systemd. My concern was that it's possible to have this issue come up again.

My thought was that it might be an idea to separately build systemd and pin the version, making it more flexible to keep up to date with systemd new features.

For most users, I suspect this might not be a big issue, but for users running Kubernetes with distros like flatcar, this issue leaves them blind to host logs (when using the container).

I suppose it's not a massive issue to pin systemd to Debian backports. That should be enough, so this PR should be fine. If I get time, I might open a PR to have systemd built from source... But TBH this PR has taken quite a while to be dealt with, I feel like it might not be worth it.

[patrick-stephens]

I did also have a similar issue with my Openshift containers due to a security upgrade on the hosts but as I was building from source like you suggest it was just a case of triggering a new build.

[jonkerj]

fluent-devs, is it possible to cut out a release for this?

[niedbalski]

@jonkerj I guess you'll have to wait until the next milestone/tag release. In the meanwhile, you can use the
master branch images published here: https://hub.docker.com/layers/fluentbitdev/fluent-bit/x86_64-master/images/sha256-bb41f9517d2e49bdb36e0943f53822a59b64357265e5b69a5445bbb1bf9824fb?context=explore

[jonkerj]

@niedbalski : there have been many releases since this PR was merged, but none of them seem to carry the fix. If we use the PR tag images like you suggest, they do not contain fixes introduced after that (such as #3905, which we face too). This makes it fairly convoluted to keep track with FB releases, as we have to cherry pick/build/upload every time.

Would it be possible to include this fix in the 1.8.x train?

[patrick-stephens]

I agree this should be a simple one to backport although little concerned you then said you wanted other fixes as well. It might be easier just to submit a PR to the 1.8 branch with the changes you want so we can easily approve & merge then everything you need.

https://github.com/fluent/fluent-bit/tree/1.8

Currently master is targeting the next release (1.9) but is the default branch so we need another PR to then backport it. I might update the PR template to suggest that when people submit a PR they also consider one for the current release series (i.e. 1.8.X now).

I agree it's not always obvious that a merged change is not necessarily in the next release and I did try to suggest some improvements in this area at previous community meetings - be good to have some more suggestions or improvements to the docs/process too.

[jonkerj]

@patrick-stephens that totally makes sense, thanks for elaborating this. My assumptions about the release process of FB were based on a linear model. I will create a PR to backport @yasn77's fixes to 1.8 as well.

I think a PR template would be great, as well as some explaning in CONTRIBUTING.md (you could steal some inspiration at elastic, they have a very complex branching system explained in their doc). I would have missed both, however, as I'm not the author of this PR, just a random stakeholder 😆

[patrick-stephens]

@jonkerj great minds think alike - I've submitted a PR (#4565 ) to do exactly that. It's pretty basic for now but I'll have a look at the Elastic info if I can find the right bit or @agup006 may know already...