Using k8s SD for alertmanager: doesn't cache address when API server goes away

[tomwilkie]

We've had this a couple of times now: when the API servers go away or Prometheus has problems speaking to them, Prometheus "forgets" where the alertmanagers are and we no longer get alerts.

I'm investigating.

This is on 2.2.1

[brian-brazil]

That sounds like a general SD issue. All SDs are meant to be resilient to this.

[tomwilkie]

We're using k8s SD for the targets, and they didn't disappear.

[brian-brazil]

Hmm, it should all be the same code.

[tomwilkie]

They are two separate instances of the same code, so its conceivable one got into a bad state and the other didn't.

[tomwilkie]

Heres the goroutines: https://gist.github.com/tomwilkie/054376c3559dd24281a333ab2ec82d01

[tomwilkie]

Yeah, looks like an SD issue (rather that a alerts issue) - here are the events, and they all just stopped at about 14:11, which is about when the master went down:

[tomwilkie]

Another interesting aside; the config was reloaded at 14:46, and a bunch of the k8s client goroutines are stuck since then (waiting for HasSynced).

So a new theory: API server went away, but everything was fine. Then our config got reloaded, and our alertmanagers got forgotten about - our alerts started clearing around 14:51PM, indicating the alert manager was forgotten at ~14:46PM.

[grobie]

We've experienced multiple issues with the K8s discovery stopping to work as well in 2.x releases. See the original description in #3810. It can be unbblocked by sending a SIGHUP. The latest occurance was today on a v2.2.1 server.

[brian-brazil]

Can you try the patch in #4013? It could explain some of this.

[brian-brazil]

Is 2.3.1 better?

[simonpasquier]

@tomwilkie are you still seeing this issue? v2.4.x included lots of improvements fixes on the SD side and in particular, the annoying #4124 should ™️ be fixed.