Re-join to consul leader in a consul cluster

[mcortinas]

Hello,

I'm a newbie in working with prometheus.

My datacenter has a several consul server in a cluster and after read prometheus documentation the file prometheus.yaml just let me set only 1 consul server (if i'm not in a mistake... )

In my first setup I set a consul server NOT LEADER in prometheus.yaml and I received this trace log when I re-started the configuration

Apr 08 09:33:14 XXXXXXXXXXXX[22361]: time="2016-04-08T09:33:14Z" level=error msg="Error refreshing service list: Unexpected response code: 500 (No cluster leader)" source="consul.go:195"

I think maybe we need add in prometheus the feature to re-join to consul server leader in case the server configured in prometheus.yml was not the current consul cluster LEADER or the posibility configure an array of consul server.

Marc

[guanglinlv]

@mcortinas how about set a consul agent in prometheus. and let your consul agent re-join the consul cluster.

[amorken]

Hi!

The recommended deployment pattern for Consul is to run a lightweight client agent on each host, and let this client agent take care of the complexities of communicating with the Consul server quorum for you. See https://www.consul.io/intro/getting-started/agent.html for more information. I believe this is what guanliniv suggests above as well.

For Prometheus, this means that it should just talk to the Consul client agent on localhost, and implementing more logic for managing connections to the quorum is not really a good thing - it is better to leave this to the Consul agent.

Now, your particular problem ("no cluster leader") would not actually be solved by running Consul this way, because connecting directly to one of the cluster servers would still work perfectly well as long as it is a member of a Consul cluster with a functioning quorum and an elected leader. The Consul server would just act as an agent for you and forward traffic to the current leader. However, when there is no cluster leader, none of the servers in the cluster can help you. This is a problem with your Consul deployment, and needs to be solved first before you can get Prometheus or any other client to work with Consul.

Thanks,
AndersM

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.