Promtail with Docker+Journal

[tfchan0]

Hi, I'm new to Loki and trying out the Promtail with JournalD support. Currently we are using journal driver for Docker. With Promtail recent commit #730 I'm able to retrieve journal entry into Loki.

However, wonder if there's a way to enrich the log with Kubernetes labels (namespace, pod name, etc)?

Thanks!

[rfratto]

Hi! Unfortunately, I'm not sure this is possible using just the journal support: when reading logs from the journal, promtail isn't aware of any extra metadata about the environment that's not already placed in the labels of each journal entry; so while the journal docker driver adds fields like CONTAINER_ID and you could pull that out, it doesn't seem to me that you can add Kubernetes-specific metadata in there. (But I could be wrong! I tried searching out for you but wasn't able to find anything.)

If you're trying to read logs from Kubernetes containers along with all the metadata, what's worked in production for us is deploying Promtail as a DaemonSet, bind mounting the host's log directory, and configuring promtail to discover and read from those files directly rather than through something like the systemd journal:

• Helm example
• Ksonnet example
• More details

[tfchan0]

Thanks @rfratto for the response. For the model of having Promtail to parse logs from /var/log (generated by Docker driver), does Promtail glean the Kubernetes info/label from the log's medadata? Or does it query kube-api for k8s information? Thanks!

[rfratto]

Promtail gets the metadata from the Kubernetes REST API. Internally it's using Prometheus' discovery code for doing that, reference on configuration and more details here: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config