Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use journalctl to read Journald logs #39820

Open
belimawr opened this issue Jun 6, 2024 · 1 comment · May be fixed by #40061
Open

Use journalctl to read Journald logs #39820

belimawr opened this issue Jun 6, 2024 · 1 comment · May be fixed by #40061
Assignees
Labels
Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team

Comments

@belimawr
Copy link
Contributor

belimawr commented Jun 6, 2024

We need to replace the current github.com/coreos/go-systemd/v22/sdjournal we use to read Journald logs by calling journalctl direct.

The problem with github.com/coreos/go-systemd/v22/sdjournal is that it relies on libsystemd and it is crashing with SIGBUS in multiple Linux distros, aside from that there are also other errors while reading the journal, the related issues:

The OpenTelemetry collector contrib calls journalctl directly and does not experience the same SIGBUS crash as Filebaeat. Example of OpenTelemetry calling journalctl: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/4322a3826e618666e679b5257d65441ed013ddf1/pkg/stanza/operator/input/journald/input.go#L67

I did some testing calling journalctl directly to filter messages and it could get all messages without any problems. OpenTelemetry was mostly successful but it had a couple of crashes (the input stopped working, not the whole binary) related to the sending queue getting full and the same "bad message" crash we see very often on Filebeat (this is easy to reproduce on Debian 11).

Because it will require Filebeat to use the fork syscall, ideally, we should only enable this syscall if the journald input is being used.

The focus of this issue is to get the core journald input working with journalctl and remove all the CGO dependency brought by `github.com/coreos/go-systemd/v22/sdjournal.

Filtering options like units, syslog_identifiers, transports, include_matches, etc can be done as part of follow up tasks.

@belimawr belimawr added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Jun 6, 2024
@elasticmachine
Copy link
Collaborator

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants