ReplicationController, ReplicaSet, Deployment, etc. support for Kubernetes SD

[therc]

I looked through the documentation, but there doesn't seem to be a way to get the desired size of a RC, RS, etc. The API server exports that. Basically, I want to use that number in alerts (is the expected number of pods up?), without having to hard-code the value in Prometheus configuration files.

More in general, the discovery code supports entities such as pods, nodes and services, but has nothing about the new ones added in recent versions of Kubernetes.

[jimmidyson]

Discovery just discovers targets to scrape & ingest metrics from. There is nothing currently exposing these metrics ATM. Sounds like you need a kubernetes exporter that will expose the metrics you require.

[therc]

Would it be in scope at some point in the future for SD to grab information like that and export it directly, if the configuration requests it? With Borgmon, you'd use the Shardinfo data from BNS:

https://docs.google.com/presentation/d/1X1rKozAUuF2MVc1YXElFWq9wkcWv3Axdldl8LOH9Vik/view#slide=id.g598ef96a6_0_118

This is a basic piece of information for a Kubernetes cluster that it doesn't make a lot of sense to have it as an external component requiring additional maintenance, configuration and meta-monitoring.

[brian-brazil]

Unless it's exposed by the SD as a target, it doesn't belong in SD.

[therc]

So, the only choice is... a new endpoint or set of metrics on the Kubernetes API server? I can't think of other places that don't involve an exporter.

I guess everyone hardcodes expected shard count in their configurations.

[brian-brazil]

That, or a separate exporter.

Coming from another direction if you alert on having no instances, and on errors/latency being okay that'd also cover you in many cases.

[jimmidyson]

That is how I've set stuff up, alert on service latencies rather than pod count. Going by the recommendations on alerting on what effects UX rather than e.g. arbitrary numbers of pods.

[therc]

Latency, which needs to be monitored anyway, works in many cases, but there are others where it doesn't apply or isn't easily instrumented, especially infrastructure pods and/or DaemonSets (DNS and, ahem... fluentd).

[fabxc]

All information about the state of Kubernetes need improved instrumentation within Kubernetes itself. Since Kubernetes is already using Prometheus for instrumentation, exporters are merely a workaround for a problem that should be fixed elsewhere.

We are currently trying to figure out whether this finds support within the Kubernetes team and will hopefully get to working on it soon.

[tcolgate]

It would be nice to be able to discover ingresses for the purposes of probing via the blackbox_exporter (much like how services can be used now).

[SuperQ]

@tcolgate Maybe let's file ingress discovery as a separate issue, it seems more clear-cut as a useful addition.

[brian-brazil]

I believe the new k8 SD covers all this.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.