Support custom authentication with external prometheus instance

[michaellzc]

Feature Request

Background

We're running linkerd in a relatively large cluster, and the integrated Prometheus instance is pretty much useless unless you give it a ton of memory on a dedicated worker node. Moreover, why would I run my own Prometheus when cloud providers can do the heavy lifting.

What problem are you trying to solve?

Bring your own Prometheus instance that requires some sort of authentication, such as token-based (static), basic auth.

GET /prometheus/api/v1/query HTTP/1.1
Host: external.prometheus.com
Authorization: Bearer supersecuretoken

How should the problem be solved?

viz plugin should have a way to accept arbitrary HTTP headers

Add a new flag to the metrics-api entrypoint, something like prometheusAuthHeaders, and it is expecting a string array. For example, --prometheusAuthHeaders key1=val1,key2=val2 or --prometheusAuthHeaders key1=val1 --prometheusAuthHeaders key2=val2

linkerd2/viz/metrics-api/cmd/main.go

Lines 21 to 29 in 451602e

	cmd := flag.NewFlagSet("metrics-api", flag.ExitOnError)
	addr := cmd.String("addr", ":8085", "address to serve on")
	kubeConfigPath := cmd.String("kubeconfig", "", "path to kube config")
	prometheusURL := cmd.String("prometheus-url", "", "prometheus url")
	metricsAddr := cmd.String("metrics-addr", ":9995", "address to serve scrapable metrics on")
	controllerNamespace := cmd.String("controller-namespace", "linkerd", "namespace in which Linkerd is installed")
	ignoredNamespaces := cmd.String("ignore-namespaces", "kube-system", "comma separated list of namespaces to not list pods from")
	clusterDomain := cmd.String("cluster-domain", "cluster.local", "kubernetes cluster domain")

In the helm chart, we can just add a new value and map them to the metrics-api deployment command arguments.

prometheusAuthHeaders:
  Authorization: "Bearer supersecuretoken"
  More: "Custom Headers"

To include the auth headers, we will need to change how the prom API client is initialized.

This can be implemented using the RoundTrippers from the prometheus API client somewhere around here

linkerd2/viz/metrics-api/cmd/main.go

Lines 49 to 55 in 451602e

	var prometheusClient promApi.Client
	if *prometheusURL != "" {
	prometheusClient, err = promApi.NewClient(promApi.Config{Address: *prometheusURL})
	if err != nil {
	log.Fatal(err.Error())
	}
	}

Learn more about how the round trippers work prometheus/client_golang#817

Any alternatives you've considered?

Is there another way to solve this problem that isn't as good a solution?

No.

How would users interact with this feature?

If you can, explain how users will be able to use this. Maybe some sample CLI
output?

See the previous section.

Notes

I am open to work on it. It would be nice if the linkerd team can provide some guidance on the testing strategy.

[olix0r]

Presumably we'd have to store this in a secret, right? It wouldn't be appropriate for these values to be visible in the metrics-api pod manifest.

[krzysztofdrys]

I once worked in a setup, which had a similar problem. We had one central prometheus cluster and when it stored metrics, it added env label to them.

When someone (i.e. viz) needed to query this central prometheus, they needed to add proper env label.

What we ended up doing was using nginx and https://github.com/prometheus-community/prom-label-proxy

It went like this:

• viz would query nginx URL,
  nginx would add env parameter (&env={$env}) and send request to prom-label-proxy`
• prom-label-proxy would rewrite the query and send it to the actual prometheus.

What I am trying to say here, is that using nginx is a possible alternative to storying authorisation information in prometheus. So for for you setup:

• viz would query nginx URL,
• nginx would add authorisation info and pass the proxy the request to prometheus.

Well, this is just a possible alternative solution. :-)

[michaellzc]

Presumably we'd have to store this in a secret, right? It wouldn't be appropriate for these values to be visible in the metrics-api pod manifest.

Yeah, you're right. We need to come up with a convention that passes the secrets key-value pair to metrics API.

there're two really naive approach

Environment Variables

PROMETHEUS_AUTH_HEADERS: key1=val1,key2=val2

metrics-api can just parse the key-value pairs from the env var.

Config file

I don't think metrics-api uses anything like viper or cobra to config the API server, so we would have to introduce a new way to config the API server. Either specifically for the new auth headers or support all existing CLI flags.

The config file should be placed in a Secret and mount into the container.

/some/path/to/secrets/file

$ cat /some/path/to/secrets/file
key1=val1
key2=val2

/some/path/to/api/server/config/file.yaml

$ cat /some/path/to/api/config/yaml/file.yaml
prometheus:
  url: "https://prom"
  headers:
    key1: val1
    key2: val2

[michaellzc]

I once worked in a setup, which had a similar problem. We had one central prometheus cluster and when it stored metrics, it added env label to them.

When someone (i.e. viz) needed to query this central prometheus, they needed to add proper env label.

What we ended up doing was using nginx and prometheus-community/prom-label-proxy

It went like this:

• viz would query nginx URL,
  nginx would add env parameter (&env={$env}) and send request to prom-label-proxy`
• prom-label-proxy would rewrite the query and send it to the actual prometheus.

What I am trying to say here, is that using nginx is a possible alternative to storying authorisation information in prometheus. So for for you setup:

• viz would query nginx URL,
• nginx would add authorisation info and pass the proxy the request to prometheus.

Well, this is just a possible alternative solution. :-)

I was going to deploy such a workaround in our environment using http-proxy which adds the custom headers to all requests.

// prom-auth-proxy.linkerd-viz.svc.cluster.local:3000
const httpProxy = require('http-proxy');

httpProxy.createProxyServer({
  headers: {
    'Authorization': `Bearer sometoken`,
  },
  target: 'https://authenticated-prom-instance',
  changeOrigin: true
}).listen(3000);

However, it just doesn't feel right to me. The proxy server pretty much grants "root" access to the prom instance and the only way you can stop unauthorized access is by having a strict network policy between metrics-api pod and the proxy pod.

We are already feeding the metrics into our sysdig instance and able to visualize or set up alert rules over there, so being able to use linkerd dashboard is more of a nice-to-have feature for us. Who doesn't like pretty dashboard ;). Therefore, we're hoping the metrics-api can have out-of-the-box support for such a use case.

[winston0410]

Is there any follow up on this issue? Allowing Basic Auth would be really helpful, as it allows us to connect linkerd with external hosted prometheus, like the one from grafana cloud

[jack1902]

Added a 👍 above, but also in a similar position whereby my prometheus has basic-auth configured and i need to pass in a set of credentials.

I've been able to use https://username:password@.... to get linkerd-viz to use the basic-auth.

For anyone who might be using grafana-cloud, you need to generate a "viewer" token from https://grafana.com under your org.

The username is your tenant-id and the password is the api-key.

[winston0410]

@jack1902 Can you share the snippet that you are using? How do you pass the secret in args?

[jack1902]

@winston0410 so i'm using helm to deploy linkerd and linkerd-viz.

in terms of the helm-chart for linkerd-viz, i pass the following as values:

prometheusUrl: "https://YOUR_TENANT_USERNAME:API_KEY@PROMETHEUS_ENDPOINT/api/prom"
prometheus:
  enabled: false

grafana:
  enabled: false

installNamespace: false
grafanaUrl: "https://YOUR_TENANT.grafana.net/"

[winston0410]

@winston0410 so i'm using helm to deploy linkerd and linkerd-viz.

in terms of the helm-chart for linkerd-viz, i pass the following as values:

prometheusUrl: "https://YOUR_TENANT_USERNAME:API_KEY@PROMETHEUS_ENDPOINT/api/prom"

prometheus:

  enabled: false



grafana:

  enabled: false



installNamespace: false

grafanaUrl: "https://YOUR_TENANT.grafana.net/"

Hm but thats does expose your API KEY? It is not encrypted? Are you substituting the env before applying the helm release?