"keep-alive" connections to consul

[drscre]

I just bumped into a situation where Prometheus v1.6.1 stopped receiving updates from Consul server.
This is probably due to a temporary networking failure between Prometheus and Consul.

netstat on Prometheus node shows ESTABLISHED connections to Consul.
netstat on Consul node shows no incoming connections from Prometheus.

I assume this is the consequence of using "blocking" api when you open a long-living connection to Consul and wait for changes to be pushed to you.
So it seems that Consul failed to push changes to Prometheus while network was down or maybe wait timeout exceeded. So he closed connection on his side, but that did not reach Prometheus.
On the other way, Prometheus still thinks everything is fine, since from his point of view "no changes" and "server dropped his end of connection" looks the same.

It would be nice to have an option to, maybe, periodically reconnect to consul server

P.S. I know that I am doing it wrong and should launch consul agent locally :-)

[or4cle]

Sounds similar to: #3314

[brian-brazil]

Yeah, sounds like a dupe of #3314

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.