Add hostname labels for promtail to docker_sd_configs

[patsevanton]

Hello! Thanks for promtail and loki

I try adding hostname labels for promtail to docker_sd_configs.
promtail.yml:

server:
  http_listen_port: 9080

positions:
  filename: /var/lib/promtail/positions.yml

clients:
  - url: http://xxxx:3100/loki/api/v1/push

scrape_configs:
  - docker_sd_configs:
    - host: unix:///var/run/docker.sock
      refresh_interval: 5s
    - labels:
        host: myhostname
    job_name: flog_scrape
    relabel_configs:
    - regex: /(.*)
      source_labels:
      - __meta_docker_container_name
      target_label: container

Get error

Unable to parse config: /etc/promtail/promtail.yml: yaml: unmarshal errors:
line 14: field labels not found in type moby.plain

[jeschkies]

Hi @patsevanton,

we are using the service discovery from Prometheus: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#docker_sd_config

It lists these labels

    __meta_docker_container_id: the id of the container
    __meta_docker_container_name: the name of the container
    __meta_docker_container_network_mode: the network mode of the container
    __meta_docker_container_label_<labelname>: each label of the container
    __meta_docker_network_id: the ID of the network
    __meta_docker_network_name: the name of the network
    __meta_docker_network_ingress: whether the network is ingress
    __meta_docker_network_internal: whether the network is internal
    __meta_docker_network_label_<labelname>: each label of the network
    __meta_docker_network_scope: the scope of the network
    __meta_docker_network_ip: the IP of the container in this network
    __meta_docker_port_private: the port on the container
    __meta_docker_port_public: the external port if a port-mapping exists
    __meta_docker_port_public_ip: the public IP if a port-mapping exists

Unfortunately hostname is not part of it.

[nahsi]

    - job_name: "docker"
      docker_sd_configs:
        - host: "unix:///var/run/docker.sock"
          refresh_interval: "5s"
      pipeline_stages:
        - static_labels:
            host: "{{ hostname }}"

[srstsavage]

If you can set your hostname when creating /etc/promtail/config.yml this will work as well:

clients:
  - url: http://loki.url:3100/loki/api/v1/push
    external_labels:
      host: {{ your hostname }}

[durcon]

@nahsi

      pipeline_stages:
        - static_labels:
            host: "{{ hostname }}"

There is nothing like static_labels in the documentation, see: https://grafana.com/docs/loki/latest/clients/promtail/configuration/#pipeline_stages

I tried it with:

- job_name: flog_scrape 
  docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
  relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'    
  pipeline_stages:
    - labels:
      host: "{{ hostname }}"

but it didn't work. I got failed to make Docker service discovery target manager: pipeline stage must contain only one key.

[nahsi]

@durcon because your indentation is wrong + it is static_labels not labels

      pipeline_stages:
        - static_labels:
            host: "{{ hostname }}"

https://grafana.com/docs/loki/latest/clients/promtail/stages/static_labels/

[durcon]

@nahsi

@durcon because your indentation is wrong + it is static_labels not labels

      pipeline_stages:
        - static_labels:
            host: "{{ hostname }}"

I tried your solution first, but unfortunately the copied lines didn't work. Because I didn't find any documentation, I tried again with labels (also without success).

Now I use shane-axiom's solution, which works without any changes.

https://grafana.com/docs/loki/latest/clients/promtail/stages/static_labels/

Thank you for the link. The documentation seems to be inconsistent, because the first link doesn't contain static_labels. Maybe I don't understand the difference between the two documentations.

[durcon]

@shane-axiom

If you can set your hostname when creating /etc/promtail/config.yml this will work as well:

clients:
  - url: http://loki.service.keryx.consul:3100/loki/api/v1/push
    external_labels:
      host: {{ your hostname }}

Your solution worked with version 2.5. Now with version 2.6.1 it doesn't work anymore. Do you know the reason?

Update

It is only a view problem, Promtail doesn't show the external_labels in the label colum. The logs contain the external labels.

[skvalex]

Here's my workaround in docker-compose.yml if anyone interested:

  promtail:
    image: grafana/promtail
    entrypoint: [ "/bin/sh","-c" ]
    command:
      - |
        export HOSTNAME=$$(cat /etc/hostname)
        /usr/bin/promtail -config.file=/etc/promtail-config.yml -client.external-labels=hostname=$$HOSTNAME
    configs:
      - source: promtail-yml
        target: /etc/promtail-config.yml
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /etc/hostname:/etc/hostname:ro
    deploy:
      mode: global

[dungdm93]

Try with this config. And it's works:

config:
 snippets:
    addScrapeJobLabel: true
    extraScrapeConfigs: |
      - job_name: docker-container
        pipeline_stages:
          - docker: {}
        docker_sd_configs:
          - host: unix:///var/run/docker.sock
        relabel_configs:
         // other configs
          - replacement: ${HOSTNAME}
            target_label: node_name

extraArgs:
- -config.expand-env=true

[askya]

One solution is :

• Add Environment="HOSTNAME=%H" in service file.
• Add -config.expand-env=true command lilne argument
  Service file looks like below :

[Unit]
Description=Promtail service
After=network.target

[Service]
Environment="HOSTNAME=%H"
Type=simple
User=promtail
ExecStart=/usr/bin/promtail -config.file /etc/promtail/config.yml -config.expand-env=true -print-config-stderr
# Give a reasonable amount of time for promtail to start up/shut down 
TimeoutSec = 60
Restart = on-failure
RestartSec = 2

[Install]
WantedBy=multi-user.target

In promtail's configuration file, add external_labels and hostname: ${HOSTNAME} (I used nodename) under client url.

...
clients:
  - url: https://loki.domain.tld/loki/api/v1/push
    external_labels:
      nodename: ${HOSTNAME}
...

It works under CentOS 8 and Debian 11.

[mrlioncub]

Here's my workaround in docker-compose.yml if anyone interested

I would recommend using it like this:

  promtail:
    image: grafana/promtail
    entrypoint: [ "/bin/sh","-c" ]
    command:
      - |
        export DOCKER_HOST=$$(cat /run/secrets/DOCKER_HOST)
        /usr/bin/promtail -config.file=/etc/promtail-config.yml -client.external-labels=hostname=$${DOCKER_HOST}
    configs:
      - source: promtail-yml
        target: /etc/promtail-config.yml
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    deploy:
      mode: global
    secrets:
      - DOCKER_HOST

secrets:
  DOCKER_HOST:
    file: /etc/hostname

[phyzical]

@durcon thank you for updating that comment, i wasted a bit of time expecting to see it in the columns on promtail dashboard too ><

Can confirm external_label still works