[Filebeat] Journald input doesn't work in container when host systemd is too recent

[mickymiek]

We couldn't make filebeat's journald input work (running in a container managed by k8s).
There was nothing significant in debug logs, the only weird thing was that when we ran lsof on the filebeat process, it wasn't opening journal files (mounted from host).
Docker image used: docker.elastic.co/beats/filebeat-oss:8.0.0-rc2
Host systemd version: 249

We managed to fix this issue by installing a more recent version of systemd library in the container (ubuntu jammy -> 249.9-0ubuntu2).

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[vinzent]

similiar/same issue here.

If running the official elastic filebeat 8.0.1 container on a Fedora Core 35 host, the journald input just reads nothing without a warning or error.

If I install the filebeat 8.0.1 rpm inside a fedora 35 container and use this image, the journald input works.

Fedora Core 35 uses systemd 249.7

[der-eismann]

Same problem here, although using a Fedora container didn't help, lsof is not showing the journal directory. It's a pity there isn't more output to debug, all I have is

"message":"Starting input (ID: 15297911848916620766)"
"message":"Loading and starting Inputs completed. Enabled inputs: 1"
"message":"Input journald starting"

Host is Flatcar Container Linux which is also using systemd 249. This is my Containerfile:

FROM fedora:35

RUN dnf install -y https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.1.2-x86_64.rpm systemd-devel

COPY deployment/docker/resources/filebeat.yml /usr/share/filebeat/filebeat.yml

RUN chmod 600 /usr/share/filebeat/filebeat.yml

ENTRYPOINT [ "/usr/share/filebeat/bin/filebeat", "-e", "--path.home", "/usr/share/filebeat", "--path.data", "/usr/share/filebeat/data" ]

with the following config

name: filebeat

filebeat.inputs:
- type: journald
  id: systemd
  seek: cursor

output.logstash:
  enabled: true
  hosts: ["${LOGSTASH_HOST}"]

logging.to_files: false
logging.level: debug

logging.metrics.enabled: false

[vinzent]

What we needed to mount to get Filebeat working in a Fedora 35 Container on a Fedora CoreOS 35 host:

  --read-only \
  --privileged \
  --user 0 \
  --mount type=bind,src=/var/log/journal,dst=/var/log/journal,readonly=true \
  --mount type=bind,src=/etc/machine-id,dst=/etc/machine-id,readonly=true \
  --mount type=bind,src=/run/systemd,dst=/run/systemd,readonly=true \
  --mount type=bind,src=/etc/hostname,dst=/etc/hostname,readonly=true \
  --mount type=volume,src=filebeat-data,dst=/var/lib/filebeat \
  --mount type=volume,src=filebeat-logs,dst=/var/log/filebeat \
  --mount type=bind,src=/etc/filebeat/filebeat.yml,dst=/etc/filebeat/filebeat.yml,readonly=true \

[zagr0]

thnx @vinzent! can confirm that this config in addition to defaults of the chart do the trick in kubernetes deployment:

extraVolumeMounts:
  - name: runsystemd
    mountPath: /run/systemd
    readOnly: true
  - name: machine-id
    mountPath: /etc/machine-id
    readOnly: true

extraVolumes:
  - name: runsystemd
    hostPath:
      path: /run/systemd
      type: ''
  - name: machine-id
    hostPath:
      path: /etc/machine-id
      type: ''

[bryce-wilkinson-443]

Hi I am also having an issue with this but no luck if anyone has any advice or can provide me with a solution that works on their end please let me know

Dockerfile

FROM fedora:35
RUN dnf install -y https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.5.2-x86_64.rpm 
COPY filebeat.yaml /usr/share/filebeat/filebeat.yml
RUN chmod 600 /usr/share/filebeat/filebeat.yml
ENTRYPOINT [ "/usr/share/filebeat/bin/filebeat", "-e", "--path.home", "/usr/share/filebeat", "--path.data", "/usr/share/filebeat/data" ]

Filebeat.yml

name: filebeat

filebeat.inputs:
- type: journald
  id: everything
  seek: cursor

logging.to_files: false
logging.level: debug

logging.metrics.enabled: false

fields:
  caas:
    env: qa
    region: xxx
    cluster_id: xxx-xxx-xxx
    source: kubernetes
  geap:
    version: 1
    client_id: xxxx
    client_secret: xxxx
    region: xxxx
    log_group: caas_journal_logs
    env: qa

output:
  logstash:
    hosts:
    - xxxxx
    - xxxxx
    - xxxxx
    compression_level: 9
    worker: 3
    loadbalance: true
    bulk_max_size: 2048

RUN on Server

docker run ${HARBOR_URL}/${REPO}/fcos-filebeat:v6 
--read-only \
--privileged \
--user 0 \
--mount type=bind,src=/var/log/journal,dst=/var/log/journal,readonly=true \
--mount type=bind,src=/etc/machine-id,dst=/etc/machine-id,readonly=true \
--mount type=bind,src=/run/systemd,dst=/run/systemd,readonly=true \
--mount type=bind,src=/etc/hostname,dst=/etc/hostname,readonly=true \
--mount type=volume,src=filebeat-data,dst=/var/lib/filebeat \
--mount type=volume,src=filebeat-logs,dst=/var/log/filebeat \
--mount type=bind,src=/etc/filebeat/filebeat.yml,dst=/etc/filebeat/filebeat.yml,readonly=true \

ERROR MSG

{"log.level":"error","@timestamp":"2022-11-30T07:02:12.450Z","log.logger":"input.journald","log.origin":{"file.name":"compat/compat.go","file.line":124},"message":"Input 'journald' failed with: input.go:130: input everything failed (id=everything)\n\tinput.go:174: failed to create reader for LOCAL_SYSTEM_JOURNAL journal (path=LOCAL_SYSTEM_JOURNAL): reader.go:99: failed to open local journal: unable to open a handle to the library","service.name":"filebeat","id":"everything","ecs.version":"1.6.0"}

[DYH-never-giveup]

Hi I am also having an issue with this but no luck if anyone has any advice or can provide me with a solution that works on their end please let me know

Dockerfile

FROM fedora:35
RUN dnf install -y https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.5.2-x86_64.rpm 
COPY filebeat.yaml /usr/share/filebeat/filebeat.yml
RUN chmod 600 /usr/share/filebeat/filebeat.yml
ENTRYPOINT [ "/usr/share/filebeat/bin/filebeat", "-e", "--path.home", "/usr/share/filebeat", "--path.data", "/usr/share/filebeat/data" ]

Filebeat.yml

name: filebeat

filebeat.inputs:
- type: journald
  id: everything
  seek: cursor

logging.to_files: false
logging.level: debug

logging.metrics.enabled: false

fields:
  caas:
    env: qa
    region: xxx
    cluster_id: xxx-xxx-xxx
    source: kubernetes
  geap:
    version: 1
    client_id: xxxx
    client_secret: xxxx
    region: xxxx
    log_group: caas_journal_logs
    env: qa

output:
  logstash:
    hosts:
    - xxxxx
    - xxxxx
    - xxxxx
    compression_level: 9
    worker: 3
    loadbalance: true
    bulk_max_size: 2048

RUN on Server

docker run ${HARBOR_URL}/${REPO}/fcos-filebeat:v6 
--read-only \
--privileged \
--user 0 \
--mount type=bind,src=/var/log/journal,dst=/var/log/journal,readonly=true \
--mount type=bind,src=/etc/machine-id,dst=/etc/machine-id,readonly=true \
--mount type=bind,src=/run/systemd,dst=/run/systemd,readonly=true \
--mount type=bind,src=/etc/hostname,dst=/etc/hostname,readonly=true \
--mount type=volume,src=filebeat-data,dst=/var/lib/filebeat \
--mount type=volume,src=filebeat-logs,dst=/var/log/filebeat \
--mount type=bind,src=/etc/filebeat/filebeat.yml,dst=/etc/filebeat/filebeat.yml,readonly=true \

ERROR MSG

{"log.level":"error","@timestamp":"2022-11-30T07:02:12.450Z","log.logger":"input.journald","log.origin":{"file.name":"compat/compat.go","file.line":124},"message":"Input 'journald' failed with: input.go:130: input everything failed (id=everything)\n\tinput.go:174: failed to create reader for LOCAL_SYSTEM_JOURNAL journal (path=LOCAL_SYSTEM_JOURNAL): reader.go:99: failed to open local journal: unable to open a handle to the library","service.name":"filebeat","id":"everything","ecs.version":"1.6.0"}

There is no journalctl command in the base linux image, such as the fedora:35 base image in the filebeat image you are using. So you can replace the linux base image in the filebeat image with the image containing the journalctl command

[DYH-never-giveup]

Hi I am also having an issue with this but no luck if anyone has any advice or can provide me with a solution that works on their end please let me know

Dockerfile

FROM fedora:35
RUN dnf install -y https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.5.2-x86_64.rpm 
COPY filebeat.yaml /usr/share/filebeat/filebeat.yml
RUN chmod 600 /usr/share/filebeat/filebeat.yml
ENTRYPOINT [ "/usr/share/filebeat/bin/filebeat", "-e", "--path.home", "/usr/share/filebeat", "--path.data", "/usr/share/filebeat/data" ]

Filebeat.yml

name: filebeat

filebeat.inputs:
- type: journald
  id: everything
  seek: cursor

logging.to_files: false
logging.level: debug

logging.metrics.enabled: false

fields:
  caas:
    env: qa
    region: xxx
    cluster_id: xxx-xxx-xxx
    source: kubernetes
  geap:
    version: 1
    client_id: xxxx
    client_secret: xxxx
    region: xxxx
    log_group: caas_journal_logs
    env: qa

output:
  logstash:
    hosts:
    - xxxxx
    - xxxxx
    - xxxxx
    compression_level: 9
    worker: 3
    loadbalance: true
    bulk_max_size: 2048

RUN on Server

docker run ${HARBOR_URL}/${REPO}/fcos-filebeat:v6 
--read-only \
--privileged \
--user 0 \
--mount type=bind,src=/var/log/journal,dst=/var/log/journal,readonly=true \
--mount type=bind,src=/etc/machine-id,dst=/etc/machine-id,readonly=true \
--mount type=bind,src=/run/systemd,dst=/run/systemd,readonly=true \
--mount type=bind,src=/etc/hostname,dst=/etc/hostname,readonly=true \
--mount type=volume,src=filebeat-data,dst=/var/lib/filebeat \
--mount type=volume,src=filebeat-logs,dst=/var/log/filebeat \
--mount type=bind,src=/etc/filebeat/filebeat.yml,dst=/etc/filebeat/filebeat.yml,readonly=true \

ERROR MSG

{"log.level":"error","@timestamp":"2022-11-30T07:02:12.450Z","log.logger":"input.journald","log.origin":{"file.name":"compat/compat.go","file.line":124},"message":"Input 'journald' failed with: input.go:130: input everything failed (id=everything)\n\tinput.go:174: failed to create reader for LOCAL_SYSTEM_JOURNAL journal (path=LOCAL_SYSTEM_JOURNAL): reader.go:99: failed to open local journal: unable to open a handle to the library","service.name":"filebeat","id":"everything","ecs.version":"1.6.0"}

You can try it, I have solved the problem this way. I mounted the journalctl command in the fully functional linux system to the base image container, then mounted the dependencies required by journalctl, and finally executed docker commit to create a linux base image that could solve the problem

[bryce-wilkinson-443]

Thank you @DYH-never-giveup can you give me an example of your config

Do you have an example image I thought Fedora Core OS has journalctl installed by default in the vanilla

so far I have the following:
--systemd=true
--mount type=bind,src=/usr/bin/journalctl,dst=/usr/bin/journalctl \

[ederst]

Yet another "we also have this issue". We switched from Ubuntu 20.04 to 22.04 on our K8s nodes and now we do not get journald entries anymore.

The only solution which worked for us was to rebuild filebeat with a newer/the same Ubuntu version so that the systemd version of the host matches the one in the container, like the initial post here suggested. Non of the other shenanigans worked.

Is it perhaps possible to update the Ubuntu version here:

beats/dev-tools/packaging/packages.yml

Lines 160 to 177 in f20ce9d

	- &docker_spec
	<<: *binary_spec
	extra_vars:
	from: 'ubuntu:20.04'
	buildFrom: 'ubuntu:20.04'
	user: '{{ .BeatName }}'
	linux_capabilities: ''
	files:
	'{{.BeatName}}.yml':
	source: '{{.BeatName}}.docker.yml'
	mode: 0600
	config: true
	- &docker_arm_spec
	<<: *docker_spec
	extra_vars:
	from: 'arm64v8/ubuntu:20.04'
	buildFrom: 'arm64v8/ubuntu:20.04'

Or at least make it possible to override with an ENV var, like PLATFORMS?

And what's interesting, I found out (through my quick and dirty debug app here) that although the the systemd library can read out things like the BootID and recognizes events, it cannot get the content of the entries. That's also a bit troubling because then the only way of finding out that it does not work is by not seeing any journald messages in e.g. Kibana.

This is of course not the fault of the filebeat journald input per se (which is also still experimental), but maybe it is possible to at least warn or quit with an error when events are received but it is unable to see any content for the Nth time or so. Maybe somewhere around here:

beats/filebeat/input/journald/pkg/journalread/reader.go

Lines 163 to 173 in f20ce9d

	// no new entry, so wait
	case c == 0:
	hasNewEntry, err := r.checkForNewEvents()
	if err != nil {
	return nil, err
	}
	if !hasNewEntry {
	// TODO: backoff support is currently not cancellable :(
	r.backoff.Wait()
	}
	continue

Anyhow, just some ideas here. Maybe its even better to see what the underlying C library or the go bindings for it does (some uncaught error maybe?).

[barzog]

We also hit that on Flatcar with k3s and filebeat 8.7.0 (using helm-chart from https://github.com/elastic/helm-charts/tree/main/filebeat). Mounted /etc/hostname, /etc/machine-id and /run/systemd as suggested with no luck (no errors in filebeat logs but also no events from journald)

kubectl -n kube-system logs daemonsets/filebeat-filebeat | grep journa
Found 3 pods, using pod/filebeat-filebeat-fqp6g
{"log.level":"warn","@timestamp":"2023-03-31T07:55:57.358Z","log.logger":"input","log.origin":{"file.name":"v2/loader.go","file.line":102},"message":"EXPERIMENTAL: The journald input is experimental","service.name":"filebeat","input":"journald","stability":"Experimental","deprecated":false,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-31T07:55:57.358Z","log.logger":"input.journald","log.origin":{"file.name":"compat/compat.go","file.line":113},"message":"Input 'journald' starting","service.name":"filebeat","id":"B5EB7CEF4CE33688","ecs.version":"1.6.0"}

[ederst]

After digging quickly through the release notes of systemd I might have found something:

CHANGES WITH 246:

# ...

        * systemd-journald gained support for zstd compression of large fields
          in journal files. The hash tables in journal files have been hardened
          against hash collisions. This is an incompatible change and means
          that journal files created with new systemd versions are not readable
          with old versions. If the $SYSTEMD_JOURNAL_KEYED_HASH boolean
          environment variable for systemd-journald.service is set to 0 this
          new hardening functionality may be turned off, so that generated
          journal files remain compatible with older journalctl
          implementations.

# ...

– Warsaw, 2020-07-30

Source: https://github.com/systemd/systemd/blob/main/NEWS#L4797-L4805

Newer OS versions might use a version >=v246 and the provided container image, which is based on Ubuntu 20.04, uses libsystemd v245. So this is likely why this issue occurs.

Disclaimer: I was not able to test it to confirm this being the "fix" - or more like workaround for using an "ancient" version of libsystemd in the image. ;)

Edit:

Mb some related issues:

• [Journalbeat] Can't read journal logs since systemd 246 #22129
• Journalbeat skips connection to logstash or elasticsearch #23220

Edit2:

I learned from fluent/fluent-bit#2998 (also related issue but different log provider), that it is also necessary to set Compress=false in the journald config.

[barzog]

I've performed some tests with different base images (ubuntu 20.04 with systemd 245 and ubuntu 22.04 with systemd 249): all tests failed, i.e. no logs from journal.
But if we add container filebeat user (uid=1000) to group adm (gid=4) - all is started work perfectly.
So working example looks as follow:
Dockerfile:

FROM ubuntu:focal
RUN
apt-get -y update &&
apt-get -y upgrade &&
apt-get -y install tini &&
apt-get -y install https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.7.0-arm64.deb
apt-get clean
ENV PATH=/usr/share/filebeat:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ELASTIC_CONTAINER=true
GODEBUG=madvdontneed=1
LIBBEAT_MONITORING_CGROUPS_HIERARCHY_OVERRIDE=/
RUN
mkdir /var/lib/filebeat /var/log/filebeat &&
groupadd --gid 1000 filebeat &&
useradd --uid 1000 --gid 1000 --groups 0,4 --home-dir /usr/share/filebeat --no-create-home filebeat &&
chown -R 1000:0 /etc/filebeat/ /var/lib/filebeat /var/log/filebeat
USER filebeat
ADD --chown=1000:0 --chmod=0755 docker-entrypoint /usr/local/bin/
WORKDIR /etc/filebeat
ENTRYPOINT ["/usr/bin/tini","--","/usr/local/bin/docker-entrypoint"]
CMD ["-environment","container"]

Build it with docker build docker build . -t "filebeat-20"
Exact string after which everything goes working
useradd --uid 1000 --gid 1000 --groups 0,4 --home-dir /usr/share/filebeat --no-create-home filebeat

docker-compose:

version: '2'
services:
filebeat1:
image: filebeat-20:latest
container_name: filebeat20
hostname: filebeat20
restart: unless-stopped
volumes:
- /root/filebeat-journald.yml:/etc/filebeat/filebeat.yml
- /run/systemd:/run/systemd
- /etc/machine-id:/etc/machine-id
- /etc/hostname:/etc/hostname
- /var/log/journal:/var/log/journal
- /run/log/journal:/run/log/journal

[ederst]

FYI: https://discuss.elastic.co/t/journald-input-cannot-read-read-zstd-compressed-journal/330379

According to the contribution guidelines it is necessary to discuss issues first there. Let's see ;)

[ebuildy]

We should create Docker images with latest Ubuntu as base?