- Scrape Configs
- Entry Parsing
- Deployment Methods
- Promtail API
- Config and Usage Examples
- Failure modes
Promtail is an agent which reads log files and sends streams of log data to
the centralised Loki instances along with a set of labels. For example if you are running Promtail in Kubernetes
then each container in a single pod will usually yield a single log stream with a set of labels
based on that particular pod Kubernetes labels. You can also run Promtail outside Kubernetes, but you would
then need to customise the
scrape_configs for your particular use case.
The way how Promtail finds out the log locations and extracts the set of labels is by using the
section in the Promtail yaml configuration. The syntax is the same what Prometheus uses.
scrape_configs contains one or more entries which are all executed for each container in each new pod running
in the instance. If more than one entry matches your logs you will get duplicates as the logs are sent in more than
one stream, likely with a slightly different labels. Everything is based on different labels.
The term "label" here is used in more than one different way and they can be easily confused.
- Labels starting with
__(two underscores) are internal labels. They are not stored to the loki index and are invisible after Promtail. They "magically" appear from different sources.
- Labels starting with
__meta_kubernetes_pod_label_*are "meta labels" which are generated based on your kubernetes pod labels. Example: If your kubernetes pod has a label "name" set to "foobar" then the scrape_configs section will have a label
__meta_kubernetes_pod_label_namewith value set to "foobar".
- There are other
__meta_kubernetes_*labels based on the Kubernetes metadata, such as the namespace the pod is running (
__meta_kubernetes_namespace) or the name of the container inside the pod (
- The label
__path__is a special label which Promtail will read to find out where the log files are to be read in.
- The label
filenameis added for every file found in
__path__to ensure uniqueness of the streams. It contains the absolute path of the file being tailed.
The most important part of each entry is the
relabel_configs which are a list of operations which creates,
renames, modifies or alters labels. A single
scrape_config can also reject logs by doing an
action: drop if
a label value matches a specified regex, which means that this particular
scrape_config will not forward logs
from a particular log source, but another scrape_config might.
Many of the scrape_configs read labels from
__meta_kubernetes_* meta-labels, assign them to intermediate labels
__service__ based on a few different logic, possibly drop the processing if the
__service__ was empty
and finally set visible labels (such as "job") based on the
In general, all of the default Promtail scrape_configs do the following:
- They read pod logs from under /var/log/pods/$1/*.log.
- They set "namespace" label directly from the
- They expect to see your pod name in the "name" label
- They set a "job" label which is roughly "your namespace/your job name"
Idioms and examples on different
- Drop the processing if a label is empty:
- action: drop regex: ^$ source_labels: - __service__
- Drop the processing if any of these labels contains a value:
- action: drop regex: .+ separator: '' source_labels: - __meta_kubernetes_pod_label_name - __meta_kubernetes_pod_label_app
- Rename a metadata label into another so that it will be visible in the final log stream:
- action: replace source_labels: - __meta_kubernetes_namespace target_label: namespace
- Convert all of the Kubernetes pod labels into visible labels:
- action: labelmap regex: __meta_kubernetes_pod_label_(.+)
Each job can be configured with a
pipeline_stages to parse and mutate your log entry.
This allows you to add more labels, correct the timestamp or entirely rewrite the log line sent to Loki.
Rewriting labels by parsing the log entry should be done with caution, this could increase the cardinality of streams created by Promtail.
Aside from mutating the log entry, pipeline stages can also generate metrics which could be useful in situation where you can't instrument an application.
See Processing Log Lines for a detailed pipeline description
The original design doc for labels. Post implementation we have strayed quit a bit from the config examples, though the pipeline idea was maintained.
See the pipeline label docs for more info on creating labels from log content.
Metrics can also be extracted from log line content as a set of Prometheus metrics. Metrics are exposed on the path
/metrics in promtail. By default a log size histogram (
log_entries_bytes_bucket) per stream is computed. This means you don't need to create metrics to count status code or log level, simply parse the log entry and add them to the labels. All custom metrics are prefixed with
There are three Prometheus metric types available.
Gauge record metrics for each line parsed by adding the value. While
Histograms observe sampled values by
See the pipeline metric docs for more info on creating metrics from log content.