Scraping password/bearer protected targets found via service discovery

[smarterclayton]

Not all possible metrics targets are safe to expose without auth, and while a ScrapeConfig can carry bearer or user/password auth info for retrieving targets protected by auth, it's not currently possible for a service discovery endpoint to do the same via relabel configuration.

For Kubernetes and OpenShift, it would be extremely desirable to allow the existing Kubernetes endpoint collector to also retrieve a bearer token or username/password from the annotations of the service. While this does encourage some undesirable behavior (placing unencrypted secrets into Kubernetes annotations), in general even for security-conscious multi-tenant clusters this is reasonably safe, because the tenant can secure the metrics endpoint with a different "shared secret" than the rest of their service. We can't safely allow Prometheus to access Kubernetes secrets in a multi-tenant environment (that would allow Prometheus to escalate), and if the choices are between no metrics, unsecured metrics, and metrics that are visible to someone in the management console, most users would choose the latter.

Would Prometheus accept a PR to allow specific labels (possibly __bearer_token__, __basic_auth_username__, __basic_auth_password__, and maaaybe __proxy_url__) to be accepted from relabelling into the http client passed to the scraper, so that discovered services could also specify some minimal parameterization?

[smarterclayton]

@fabxc we discussed this at KubeCon briefly.

[brian-brazil]

Dupe of #1176

If you can create a shared secret that's insecure for everything, you can put that in a k8 config.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.