Support certificates stored as Kubernetes secrets

[JosephSalisbury]

Hi!

At Giant Swarm, we're developing a platform for managing and running Kubernetes clusters. We run our control plane on Kubernetes, and use that to spin up Kubernetes clusters.

I'd like to use Prometheus to scrape these 'guest' Kubernetes clusters (api server, kubelet, node exporters, to begin with). As a customer requirement, we need to use https for the scrape itself.

Our operators provide a Kubernetes Service in the host cluster that points to the masters / workers of the guest cluster.

My plan is to add the Prometheus scraping annotation to these services, and have the Prometheus set up on the host cluster scrape them (we use the k8s service discovery, and i'd prefer to integrate with this). However, I'd need to specify the certificates for each cluster. The certificates are stored as Secrets in the host Kubernetes.

I don't see a way to mount these secrets to Prometheus easily. Customers can create and destroy clusters at will.

Does adding support for an annotation that points to the certificates - stored as a Kubernetes Secret -
make sense to you guys? Prometheus can then pull the secret, and use it for the TLS configuration.

e.g:

metadata:
  name: worker-xjjdo
  namespace: jfo39
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/certificates: "jfo39/prometheus-certificates"   // this is the magic

as an annotation for one of the worker's services, with the ca, crt, and key stored as data in the secret referenced. the actual name and format of the annotation is rough here. prometheus can then fetch the secret on scrape, and use the ca / crt / key as part of the tls connection.

Opening this to start a discussion, happy to supply a PR if this idea makes sense to you guys, or if you have any pointers to better ways to handle this.

[JosephSalisbury]

Give me a shout if any part of this is non-sensical, it's been a long day :D

[brian-brazil]

What's special about this way of storing secrets that we should support it and not every other way that people store secrets? We currently purposefully offer no interactions between relabelling and secrets, rather you must create a job per set of credentials.

It sounds to me that your use case is basically the same as running a Prometheus outside a k8 cluster, which is handled by providing the secerts in the config file. You'll already need a scrape_config per k8 cluster anyway in your setup.

[JosephSalisbury]

My thinking was to bring this functionality into prometheus, as opposed to writing some sidecar service to fetch the appropriate secrets and generate the prometheus config file - this felt like it would replicate a fair amount of the kubernetes sd work.

I get your point about needing a separate scrape_config per k8s cluster though, that makes sense.

[brian-brazil]

this felt like it would replicate a fair amount of the kubernetes sd work.

When you get into more sophisticated setups like yours there's a good chance that you'll need to do fancy stuff with the config file one way or the other. If it wasn't this, it'd be something else (e.g. per-k8 cluster rules).

What your proposing would only help some k8 users, Prometheus is platform-agnostic though so this would be the start of lots of additional logic to support a myriad of different ways of doing secret management across various platforms. We'd be basically talking creating a generic secret access library, and then tying that into SD&relabelling somehow. We can barely keep the service discovery subsystem maintained, another non-trivial integrate-the-world subsystem isn't something we can consider right now.

[JosephSalisbury]

Yeah, that makes sense to me. I consider this issue closed now.

I'll see if we can make our service open-source, in case other people do find a use for it - that feels like the best way to help out.

Cheers for your time! Have a nice day.

[brian-brazil]

You might consider proposing it as a feature in the CoreOS Prometheus Operator, I don't know if it would be accepted but it would be a more appropriate place to put it as that's focused on k8.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.