Feature request: Support Consul Connect

[roidelapluie]

Consul Connect is a new open source feature of consul that enables secure communications with TLS for services connected to consul.

https://www.hashicorp.com/blog/consul-1-2-service-mesh

[brian-brazil]

Can you explain more about what this is? We already support TLS connections for Consul SD.

[roidelapluie]

Consul Connect is a new feature of Consul (currently beta) that enables transparent TLS connection between services configured by Consul. It means that prometheus could transparently scrape metrics using tls, with a local proxy.

[roidelapluie]

See also: https://www.consul.io/intro/getting-started/connect.html

[brian-brazil]

I've read the docs, but they're not very illuminating. How does this work under the covers?

We don't support bespoke auth mechanisms as that would be a maintenance nightmare, so unless this is a standard http proxy (which we already support) it's unlikely this will be supported. Prometheus would also continue to need access to individual instances, which I'm not sure if this provides. If there's new metadata (which I presume there is) we can expose it via SD.

[roidelapluie]

Here it is, a small POC.

in consul.d, add prometheus.json

{                                                                                                                                         
   "service": {
      "connect": {
         "proxy": {
            "config": {
               "upstreams": [
                  {
                     "destination_name": "prometheus",
                     "local_bind_port": 1234
                  }
               ]
            }
         }
      },
      "name": "prometheus",
      "port": 9090
   }
}

Run:

consul agent -dev -config-dir=consul.d

Run prometheus.

Then, you can connect to http://127.0.0.1:1234 (not https); that will use tls behind the scene.

Multiple questions remain:

1. How can we fetch metrics of all the services? Because port 1234 currently will redirect you to one of the sane services.
2. how can we get the extra metadata to know which port to connect to?

So my assumption for the two answers before is that we would need to implement connect client directly inside prometheus. While that should not be that much work that would still be work.

Maybe we can also take this outside of prometheus and try to make a consul-connect file_sd provider but because there are so many prometheus users using consul that would be sad.

[brian-brazil]

So my assumption for the two answers before is that we would need to implement connect client directly inside prometheus. While that should not be that much work that would still be work.

I don't think it's appropriate to start implementing service mesh support inside Prometheus (we can't maintain SDs as-is, we don't need a whole new category of thing to maintain). If someone needs this they can always use the existing proxy_url.

Maybe we can also take this outside of prometheus and try to make a consul-connect file_sd provider but because there are so many prometheus users using consul that would be sad.

Presuming you can solve 1., this would likely be possible with consul_sd, some new metadata, and relabelling. You'd probably still need a connect-aware proxy though.

there are so many prometheus users using consul that would be sad.

That's kinda like saying that Prometheus supports EC2 SD, so why doesn't it support using RDS as a backing store. Discovery and scraping are different layers of the stack.

[roidelapluie]

I have opened hashicorp/consul#4298

If it gets implemented, I will write some documentation about that (& blog posts).

In the meantime let's keep this issue open ; maybe I will get other ideas.