Agent cannot scrape journal for Systemd 246

[zanglang]

Agent 0.13.x is not scraping any journald logs on Ubuntu 20.10.

Unsure if it is related to this issue grafana/loki#2792 where Loki < 2.1.0 is unable to parse Journald logs due to a backwards-incompatible compression flag enabled in Systemd 246.

Seems to me that the loki component is not enabled when only journal targets are defined.

$ systemd --version
systemd 246 (246.6-1ubuntu1.2)
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid

$ grafana-agent -version
agent, version v0.13.1 (branch: HEAD, revision: 05f26e6)
  build user:       runner@fv-az216-940
  build date:       2021-04-09T14:26:08Z
  go version:       go1.15.8
  platform:         linux/amd64

$ sudo grep -A20 loki /etc/grafana-agent.yaml
loki:
  configs:
  - name: default
    positions:
      filename: /tmp/positions.yaml
    clients:
    - url: xxx
    scrape_configs:
    - job_name: journal
      journal:
        json: false
        max_age: 12h
        path: /var/log/journal
        labels:
          job: systemd-journal

No loki logs were sent to stdout:

$ journalctl -x -u grafana-agent | grep -iE 'loki|journal'
$

This can be verified by enabling static_configs, only the static provider is enabled for the loki component.

Apr 13 08:55:02 azure-node-2 grafana-agent[276312]: level=info ts=2021-04-13T08:55:02.644720906Z caller=server.go:229 http=[::]:12345 grpc=[::]:9095 msg="server listening on addresses"
Apr 13 08:55:02 azure-node-2 grafana-agent[276312]: level=debug ts=2021-04-13T08:55:02.647018263Z caller=instance.go:268 agent=prometheus instance=2ae07eb430d7bbbdc351609850906af2 msg="initializing instance" name=
2ae07eb430d7bbbdc351609850906af2
Apr 13 08:55:02 azure-node-2 grafana-agent[276312]: level=info ts=2021-04-13T08:55:02.667684171Z caller=wal.go:173 agent=prometheus instance=2ae07eb430d7bbbdc351609850906af2 msg="replaying WAL, this may take a whi
le" dir=/tmp/grafana-agent-wal/2ae07eb430d7bbbdc351609850906af2/wal
Apr 13 08:55:02 azure-node-2 grafana-agent[276312]: level=info ts=2021-04-13T08:55:02.667947377Z caller=wal.go:220 agent=prometheus instance=2ae07eb430d7bbbdc351609850906af2 msg="WAL segment loaded" segment=0 maxS
egment=3
...
Apr 13 08:57:10 azure-node-2 grafana-agent[276864]: level=info ts=2021-04-13T08:57:10.380121256Z caller=server.go:229 http=[::]:12345 grpc=[::]:9095 msg="server listening on addresses"
Apr 13 08:57:10 azure-node-2 grafana-agent[276864]: level=debug ts=2021-04-13T08:57:10.382152905Z caller=instance.go:268 agent=prometheus instance=2ae07eb430d7bbbdc351609850906af2 msg="initializing instance" name=
2ae07eb430d7bbbdc351609850906af2
Apr 13 08:57:10 azure-node-2 grafana-agent[276864]: level=debug ts=2021-04-13T08:57:10.384476261Z caller=manager.go:195 component=loki component=loki loki_config=default component=discovery msg="Starting provider"
 provider=static/0 subs=[system]
Apr 13 08:57:10 azure-node-2 grafana-agent[276864]: level=debug ts=2021-04-13T08:57:10.385610089Z caller=manager.go:213 component=loki component=loki loki_config=default component=discovery msg="Discoverer channel
 closed" provider=static/0

Config:

loki:
  configs:
  - name: default
    positions:
      filename: /tmp/positions.yaml
    clients:
    - url: xxx
    scrape_configs:
    - job_name: system
      static_configs:
        - labels:
            job: varlogs
            __path__: /var/log/fail2ban.log
    - job_name: journal
      journal:

A suitable fix may be to bump up the Loki version here? https://github.com/grafana/agent/blob/main/go.mod#L15 Or to rebuild with updated libsystemd.

libsystemd may not be the issue as the journal logs began collecting at some point.

[rfratto]

Hey, did you figure it out? You may not be the only one that runs into this so I'd like to know how to help people in the future.

[zanglang]

@rfratto oddly after some prodding the agent started picking up journald logs, so I went ahead to close the ticket first. This at least proves that it's not a libsystemd issue that I initially thought. I'll see if I can accurately reproduce it again on another environment.

[zanglang]

Was able to replicate this behavior again. I will re-open the ticket and update my findings.

[zanglang]

Figured out the reason. The default UID/GID for grafana-agent does not have access to systemd-journal group used by Ubuntu 20.10.

To solve this:

usermod -a -G adm grafana-agent
usermod -a -G systemd-journal grafana-agent

[tonyplovich]

Just a heads up, I'm seeing something kind of like this issue on RHEL8. Although it appears to be related to Promtail, because the behavior I'm seeing is the same in both.

I found that I'm unable to scrap journal logs as a non-privileged user. Even when I run the agent/Promtail as root, it only initially ships logs, then goes back to silently failing until I restart it.

grafana/loki#3779