How to dynamically retrieve internal IP for docker container?

[pschiffe]

Hello,

I'm trying to gather metrics from docker containers with grafana agent. Containers are not uniform, metrics are often exposed on different paths. I'm trying to create a semi-dynamic configuration for grafana agent with discovery.docker, for example:

discovery.docker "default" {
  host = "unix:///var/run/docker.sock"
}

prometheus.scrape "default" {
  targets = [{
    __address__ = json_path(discovery.docker.default.targets, "[?(@.__meta_docker_container_name == \"/cockroachdb\")].__meta_docker_network_ip") + ":8080",
    metrics_path__ = "/_status/vars",
  }]

  forward_to = [prometheus.remote_write.default.receiver]
}

But in this case, I get an error: Error: /etc/grafana-agent-flow.river:43:32: discovery.docker.default.targets should be string, got array

What would be the best approach here? Ideally without static internal IPs for containers :/

[pschiffe]

I've solved it with discovery.relabel:

discovery.docker "default" {
  host = "unix:///var/run/docker.sock"
  port = 8080

  filter {
    name = "label"
    values = ["metrics.job"]
  }
}

discovery.relabel "docker" {
  targets = discovery.docker.default.targets

  rule {
    source_labels = ["__meta_docker_network_ip", "__meta_docker_container_label_metrics_port"]
    separator     = ":"
    target_label  = "__address__"
    action        = "replace"
  }

  rule {
    source_labels = ["__meta_docker_container_label_metrics_path"]
    target_label  = "__metrics_path__"
    action        = "replace"
  }

  rule {
    source_labels = ["__meta_docker_container_label_metrics_job"]
    target_label  = "job"
    action        = "replace"
  }
}