Promtail config for journald logging, Podman considerations

[CJLove]

Hi Tim,

This has been a fantastic resource for getting grafana/loki deployed in my environment! Many thanks!

Fedora doesn't configure journald to log files under /var/log so an additional job in the promtail-config.yml may be needed to scrape the journal files:

- job_name: journal
  journal:
    max_age: 12h
    labels:
      job: systemd-journal
    path: /var/log/journal
  relabel_configs:
    - source_labels: ['__journal__systemd_unit']
      target_label: 'unit'
    - source_labels: ['__journal__hostname']
      target_label: 'hostname'

This also adds the "hostname" label so you can distinguish if you've got multiple hosts sending journald logs to loki.

There are additional volumes to pass into the promtail container as well:

/run/log/journal:/run/log/journal
/etc/machine-id:/etc/machine-id

For those trying to do this with Podman on Fedora I found that loki and grafana can be run as non-root user in a pod with promtail running separately as root, using the public loki port.