consul SD should honor CONSUL_HTTP_ADDR env

[Wing924]

Proposal

Use case. Why is this important?

Currently, consul SD set server localhost:8500 by default.
I suggest change it to os.Getenv("CONSUL_HTTP_ADDR"), faillback to localhost:8500 if it's nil.

Because consuls are running as DaemonSet on k8s and they use host network, we must access them from not localhost but node's IP.
We can't set node's IP on ConfigMap because it's unknown before assign to a node.

ref. Running Consul - Kubernetes - Consul by HashiCorp

update

I suggest consul SD server defaults to "" (empty string).
Go Consul Client will handle the env variable:
https://github.com/hashicorp/consul/blob/master/api/api.go#L27
https://github.com/hashicorp/consul/blob/master/api/api.go#L338
it won't introduce breaking change.

[brian-brazil]

We only take config from one place for simplicity, which is the config file here. What I'd suggest is templating the config file on the host.

You might also want to look at k8 service discovery.

[Wing924]

We only take config from one place for simplicity, which is the config file here. What I'd suggest is templating the config file on the host.

For this case, I suggest consul SD server defaults to "" (empty string).
Go Consul Client will handle the env variable:
https://github.com/hashicorp/consul/blob/master/api/api.go#L27
https://github.com/hashicorp/consul/blob/master/api/api.go#L338
it won't introduce breaking change.

You might also want to look at k8 service discovery.

I also use k8s SD for metrics inside the k8s cluster while use consul for metrics on outside bare-metal.

[brian-brazil]

You can still use k8 SD for that, node mode in particular, to monitor your machine-level infrastructure.

[Wing924]

You can still use k8 SD for that, node mode in particular, to monitor your machine-level infrastructure.

I mean machines outside the k8s cluster, which is managed by consul for a long time.
k8s SD can't discovery those machines.

[brian-brazil]

So where's the problem then? Localhost will work on those machines.

[Wing924]

So where's the problem then? Localhost will work on those machines.

When the Prometheus are running on k8s while monitoring targets are on outside, it's not easy to write config.