Consul discoverer doesn't respect datacenter provided by the node

[ajankovic]

Bug Report

What did you do?

Tried to replicate consul API for discovery. API is returning agent's datacenter on /v1/agent/self and each node returns it's own datacenter in /v1/catalog/service/<service> list.

What did you expect to see?

Discovered label __meta_consul_dc should contain datacenter returned by the node itself.

For example:

...
        "Node": "foobar",
	"Address": "192.168.10.10",
	"Datacenter": "dc1",
        "TaggedAddresses": {
...

Should label the target with "dc1" datacenter.

What did you see instead? Under which circumstances?

Discovered label __meta_consul_dc contains datacenter reported by the agent itself.

Environment

• Prometheus version:

Version | 2.5.0
67dc912ac8b24f94a1fc478f352d25179c94ab9b
HEAD
root@578ab108d0b9
20181106-11:40:44
go1.11.1

Question:

In summary why is discoverer for Consul assuming that agent's datacenter should be used for all discovered targets?

[simonpasquier]

To be clear, you're emulating the Consul API to implement your own SD mechanism, /v1/agent/self is returning someDc and /v1/catalog/service/<service> returns anotherDc?

In summary why is discoverer for Consul assuming that agent's datacenter should be used for all discovered targets?

I'm not the best qualified person but it was probably the easiest thing to do at the time being. Maybe @iksaif has more insights?
Another question would be whether or not this could be an issue for "real" Consul setups.

[ajankovic]

To be clear, you're emulating the Consul API to implement your own SD mechanism, /v1/agent/self is returning someDc and /v1/catalog/service/ returns anotherDc?

That is correct, but to be precise /v1/catalog/service/<service> may or may not return the same DC as the agent itself it can return multiple different DCs.

The strange thing is that response format for the node already has a field for DC which tells me it should be used.

Also Consul documentation says:

Datacenter is the data center of the Consul node on which the service is registered.

If you take a look at the home page example for "Multiple Datacenters" (scroll down) you can see that returned list can have nodes with multiple different DCs.

$ curl http://localhost:8500/v1/catalog/datacenters
[
  "dc1",
  "dc2"
]
$ curl http://localhost:8500/v1/catalog/nodes?dc=dc2
[
    {
        "ID": "7081dcdf-fdc0-0432-f2e8-a357d36084e1",
        "Node": "10-0-1-109",
        "Address": "10.0.1.109",
        "Datacenter": "dc2",
        "TaggedAddresses": {
            "lan": "10.0.1.109",
            "wan": "10.0.1.109"
        },
        "CreateIndex": 112,
        "ModifyIndex": 125
    },
...

This is in contrast with assuming DC of the node by the DC of the agent as agent can have only one DC.

I can offer a pull request if you are willing to accept it.

[brian-brazil]

I don't think we should be trying to accommodate anything but Consul itself. We already have a generic integration.

[ajankovic]

I don't think we should be trying to accommodate anything but Consul itself.

Totally agree with that statement. But the emulation part is irrelevant, it's there just for transparency.

The question still remains, why is discoverer assuming node's datacenter based on agent's datacenter when node is providing that information itself?

Or maybe in Consul terms, can the service have nodes that span multiple datacenters? If yes then the discoverer implementation is flawed. And I think I provided links to documentation that confirms this is true.