[8.18](backport #7995) Enable Filebeat's Journald input in some docker image variants

[mergify]

What does this PR do?

Filebeat's Journald input requires journalctl, so the systemd package is added to all docker image variants.

Changes in image size

Images built from main (f108618):

docker.elastic.co/elastic-agent/elastic-agent-complete         9.1.0-SNAPSHOT         123d3a5d0a22   2 hours ago   3.57GB
docker.elastic.co/elastic-agent/elastic-otel-collector         9.1.0-SNAPSHOT         a5182ef34ae0   2 hours ago   1.2GB
docker.elastic.co/elastic-agent/elastic-agent                  9.1.0-SNAPSHOT         cf9309afdc3f   2 hours ago   1.89GB

Images built from this PR:

docker.elastic.co/elastic-agent/elastic-agent-complete         9.1.0-SNAPSHOT         18845462f221   16 minutes ago   3.58GB
docker.elastic.co/elastic-agent/elastic-otel-collector         9.1.0-SNAPSHOT         2f29d6c0f17a   18 minutes ago   1.25GB
docker.elastic.co/elastic-agent/elastic-agent                  9.1.0-SNAPSHOT         d5b99148748a   18 minutes ago   1.94GB

Increase in size

• docker.elastic.co/elastic-agent/elastic-agent-complete: 10MB
• docker.elastic.co/elastic-agent/elastic-otel-collector: 50MB
• docker.elastic.co/elastic-agent/elastic-agent: 40MB

Why is it important?

Currently it is only possible to use integrations that requires the journald input with the complete docker image variant, this PR allows it to be used with any docker image variant. The complete docker image is the largest with more than 3Gb, while the standard Elastic-Agent image is less than 2Gb.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

## Disruptive User Impact

Author's Checklist

• Add integration test for each docker image: mount the journal into the container and ensure the Journald input can read it.

How to test this PR locally

1. Package the Elastic-Agent:

   DEV=true SNAPSHOT=true EXTERNAL=true PACKAGES="docker" PLATFORMS=linux/amd64 mage -v package
   

2. Check the version of journalctl on each docker image by running the following script/commands:

   #!/bin/bash
   
   echo "############################## Testing images with journalctl ##############################"
   
   set -o xtrace
   docker run --rm -it --entrypoint journalctl docker.elastic.co/elastic-agent/elastic-agent-complete:9.1.0-SNAPSHOT  --version
   docker run --rm -it --entrypoint journalctl docker.elastic.co/elastic-agent/elastic-agent-ubi:9.1.0-SNAPSHOT  --version
   docker run --rm -it --entrypoint journalctl docker.elastic.co/elastic-agent/elastic-agent:9.1.0-SNAPSHOT  --version
   docker run --rm -it --entrypoint journalctl docker.elastic.co/elastic-agent/elastic-otel-collector:9.1.0-SNAPSHOT  --version
   
   set +o xtrace
   echo "############################## Testing images without journalctl ##############################"
   
   set -o xtrace
   docker run --rm -it --entrypoint journalctl docker.elastic.co/elastic-agent/elastic-agent-complete-wolfi:9.1.0-SNAPSHOT  --version
   docker run --rm -it --entrypoint journalctl docker.elastic.co/elastic-agent/elastic-agent-slim-wolfi:9.1.0-SNAPSHOT  --version
   docker run --rm -it --entrypoint journalctl docker.elastic.co/elastic-agent/elastic-agent-slim:9.1.0-SNAPSHOT  --version
   docker run --rm -it --entrypoint journalctl docker.elastic.co/elastic-agent/elastic-agent-wolfi:9.1.0-SNAPSHOT  --version
   docker run --rm -it --entrypoint journalctl docker.elastic.co/elastic-agent/elastic-otel-collector-wolfi:9.1.0-SNAPSHOT  --version
   

3. Alternatively, check the Dockerfile for each of the images:

   grep "systemd" build/package/elastic-*/*/docker-build/Dockerfile

Related issues

• Relates Add journalctl to Filebeat and Elastic-Agent docker images beats#44040
• Docs update: Update the docs about Journald support. docs-content#1272
• Closes Journald (input and integration) not working on some 8.x Docker images #8445

Questions to ask yourself

• How are we going to support this in production?
• How are we going to measure its adoption?
• How are we going to debug this?
• What are the metrics I should take care of?
• ...

---

This is an automatic backport of pull request #7995 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of fd33d85 has failed:

On branch mergify/bp/8.18/pr-7995
Your branch is up to date with 'origin/8.18'.

You are currently cherry-picking commit fd33d8569.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1745609163-journalctl-on-all-docker-variants.yaml
	modified:   magefile.go
	new file:   testing/integration/journald_test.go
	modified:   testing/integration/kubernetes_agent_standalone_test.go
	new file:   testing/integration/testdata/journald-input.yml
	new file:   testing/integration/testdata/journald-otel.yml

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	deleted by us:   dev-tools/packaging/settings.go
	both modified:   dev-tools/packaging/templates/docker/Dockerfile.elastic-agent.tmpl

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[mergify]

This pull request has not been merged yet. Could you please review and merge it @belimawr? 🙏

[pkoutsovasilis]

sorry @belimawr I hope that you don't mind but I am taking over this PR, I am anticipating a large k8s fix that needs to come and I need this one merged 🙂

The first reason that I see these tests failing is because of this #8452

[pkoutsovasilis]

ok the ugprade failures are known and being handled in the beats repo. but @belimawr have a look at the changes I did and merge it if you are ok 🙂

[belimawr]

sorry @belimawr I hope that you don't mind but I am taking over this PR, I am anticipating a large k8s fix that needs to come and I need this one merged 🙂

The first reason that I see these tests failing is because of this #8452

Be my guest :D. Thanks for the help!

I had seen #8452, hence I was considering this PR blocked until it was solved, I should have linked it here. Sorry for that.

[belimawr]

I removed myself from the assignee list, but feel free to ping me if you need anything.

[pkoutsovasilis]

@belimawr please tell me that I didn't remove anything critical with these two commits 4d4c251 968f123 🙂

[belimawr]

@belimawr please tell me that I didn't remove anything critical with these two commits 4d4c251 968f123 🙂

All good!

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 5eb1ddb

Failed CI Steps

• Win2022:sudo:standalone-upgrade
• Win2022:sudo:standalone-upgrade
• Win2025:sudo:standalone-upgrade
• Win2025:sudo:standalone-upgrade

History

• 💔 Build #22667 failed 968f123
• 💔 Build #22664 failed 37a4769
• 💔 Build #22503 failed 9a16de2
• 💔 Build #22384 failed 698f3ce

cc @pkoutsovasilis