[FEATURE REQUEST] Use File based service discovery instead of DNS based for flexibility

[bvis]

Current Situation

Currently DFSL + DFM use the DNS based service discovery. AFAIK DFSL just checks changes in services in the docker API and sends each change to DFM.

This approach is simple and effective but comes with some limitations:

• We don't have any way to add the node information where each task is running, as Docker is not providing this info ATM.
• We cannot add arbitrary labels to each service/task for more granular info
• Each time an exporter is restarted the id, that is based in the task IP changes, then we lose continuity in the metrics.

As you may see relabelling does not offer any value useful for identifying where the request is coming from:

Proposal

Prometheus allows to use other discovery systems, while Swarm is not integrated (I've not seen any improvement in this area in a year) we could make usage of Docker API calls + File based discovery service for getting more info for improving the collected metrics.

Example

~ $ curl -s -g -H "Content-Type: application/json" --unix-socket /var/run/docker.sock "http:/v1.24/tasks?filters={\"desired-state\":[\"running\"],\"service\":[\"monitoring_cadvisor\"]}" | jq '[.[] | {NodeID: .NodeID, Labels: .Spec.Conta
inerSpec.Labels, Addresses: .NetworksAttachments[0].Addresses[0] }]'
[
  {
    "NodeID": "zixucyzgugv4fmcf0funaqele",
    "Labels": {
      "com.docker.stack.namespace": "monitoring",
      "com.df.environment": "prod",
      "com.df.xxxx.scrapeTasks": "true"
    },
    "Addresses": "10.0.3.13/24"
  },
  {
    "NodeID": "qz9u95jos3fibvewxxbj9y5i8",
    "Labels": {
      "com.docker.stack.namespace": "monitoring",
      "com.df.environment": "prod",
      "com.df.xxxx.scrapeTasks": "true"
    },
    "Addresses": "10.0.3.12/24"
  },
  {
    "NodeID": "l0n731v8859ejdnnubqvjyuwd",
    "Labels": {
      "com.docker.stack.namespace": "monitoring",
      "com.df.environment": "prod",
      "com.df.xxxx.scrapeTasks": "true"
    },
    "Addresses": "10.0.3.11/24"
  }
]
$ curl -s --unix-socket /var/run/docker.sock http:/v1.31/nodes/zixucyzgugv4fmcf0funaqele | jq '.Description.Hostname'
"node-1"

These queries finally allow to generate an structure like this for Prometheus:

[
  {
    "targets": [ "10.0.3.13:8080" ],
    "labels": {
      "env": "prod",
      "job": "cadvisor",
      "node": "node-1"
    }
  },
  {
    "targets": [ "10.0.3.12:8080" ],
    "labels": {
      "env": "prod",
      "job": "cadvisor",
      "node": "node-2"
    }
  },
  {
    "targets": [ "10.0.3.11:8080" ],
    "labels": {
      "env": "prod",
      "job": "cadvisor",
      "node": "node-3"
    }
  },
  {
    "targets": [ "10.0.0.8:9100" ],
    "labels": {
      "env": "prod",
      "job": "node-exporter",
      "node": "node-1"
    }
  },
  {
    "targets": [ "10.0.0.9:9100" ],
    "labels": {
      "env": "prod",
      "job": "node-exporter",
      "node": "node-2"
    }
  }
]

Which could be stored in a dynamic file that should change every time something in the exporters change. This file can be used with a reference in the prometheus.yml file:

- job_name: 'dummy'  # This is a default value, it is mandatory.
  file_sd_configs:
    - files:
      - /etc/prometheus/prometheus.sd.json

And doing some relabelling to use the node entry as target in Prometheus.

[vfarcic]

That can be done. I can start working on this feature next week (after DockerCon).

[thomasjpfan]

One possible way to implement this feature:

1. Add node hostname and service ip discovery to DFSL here: https://github.com/vfarcic/docker-flow-swarm-listener/blob/master/service/util.go#L36 by using the docker go SDK: (https://godoc.org/github.com/moby/moby/client#Client.TaskInspectWithRaw and https://godoc.org/github.com/moby/moby/client#Client.NodeList). This would add the IPs and the corresponding nodes to the DFSL notification.
2. Configuration option for DFM decide which labels to include in file_sd_configs.

This way, DFM does not need access to the docker socket. If this looks good, I can work on this!

[vfarcic]

@thomasjpfan Is the idea for DFSL to send IPs to DFM? If that's the case, DFSL would need to re-send that info periodically since nodes/replicas might change with time.

[thomasjpfan]

@vfarcic Looking at https://github.com/vfarcic/docker-flow-swarm-listener/blob/master/service/service.go#L131, DFSL checks for label and replica changes before sending out notifications. Maybe we can add the (IP, node-hostname) information into the cache and check if that changes as well?

[vfarcic]

That's a very good idea @thomasjpfan. I'd just make sure that using IP and hostname info to determine whether to send notifications is optional and disabled by default since most services will not benefit from that. What do you think?

[thomasjpfan]

That looks good. I will work on a PR for DFSL.

[thomasjpfan]

Consider this configuration snippet:

...
services:
  cadvisor:
    image: google/cadvisor
    networks:
      - monitor
      - flow
    deploy:
      mode: global
      labels:
        - com.df.scrapeNetwork=monitor
        - com.df.scrapePort=8080

  swarm-listener:
    image: vfarcic/docker-flow-swarm-listener
    networks:
      - monitor
    environment:
      - DF_INCLUDE_NODE_IP_INFO=true

  monitor:
    image: vfarcic/docker-flow-monitor
    networks:
      - monitor
...

I want to add a com.df.scrapeNetwork label to the exporter. This way the swarm-listener knows which network to use when querying the IP of the exporter. In the example above, com.df.scrapeNetwork=monitor, thus DFSL will use the monitor network to gather node and ip info. What do you think?

[vfarcic]

@thomasjpfan I think you need the name of the service (not the network) and retrieve the IPs with something like net.LookupHost.

[thomasjpfan]

In my example above, since cadvisor is on two networks, it will have two IPs associated with each network. If swarm-listener and monitor were connected to the flow network, the cadvisor IP associated with the flow network must be passed on to the monitor service.

Since DFSL has access to the docker socket, TaskList already returns all the IPs needed. The return swarm.Task objects has a NetworksAttachments field containing a list of IPs for each network the service is connected to. With a label com.df.scrapeNetwork=monitor on the service, DFSL would know to extract the IP associated with the monitor network.

In the end, swarm-listener would pass a list: [["node-1", "1.0.1.1"], ["node-2", "1.0.1.2"]] to monitor.

[vfarcic]

You're right @thomasjpfan . I missed the case of two networks :)