Skip to content

Prometheus multi-tenant Proxy. Needed to deploy Prometheus in a multi-tenant way


Notifications You must be signed in to change notification settings


Repository files navigation

Prometheus — Multi-tenant proxy

Build Status License: GPL v3

Twitter Join the chat at

This project aims to make it easy to deploy a Prometheus Server in a multi-tenant way.

This project has some reference from the prometheus label injector

The proxy enforces the namespace label in a given PromQL query while providing a basic auth layer.

What is it?

It is a simple golang proxy. It does basic or JWT auth, logs the requests, and serves as a Prometheus reverse proxy.

Actually, Prometheus does not check the auth of any request. By itself, it does not provide any multi-tenant mechanism. So, if you have untrusted tenants, you have to ensure a tenant uses its labels and does not use any other tenants' value.

For more security, only specific endpoints are proxied by default: /api/v1/series, /api/v1/query, and /api/v1/query_range (see --protected-endpoints).

The proxy also supports Amazon Managed Service for Prometheus.


To use this project, place the proxy in front of your Prometheus server instance, configure the auth proxy configuration and run it.

Run it

$ prometheus-multi-tenant-proxy run \
  --prometheus-endpoint http://localhost:9090 \
  --port 9091 \
  --auth-config ./my-auth-config.yaml \
  --reload-interval=5 \
  --unprotected-endpoints /-/healthy,/-/ready

Available arguments // environment variables to the run command:

  • --port // PROM_PROXY_PORT: Port used to expose this proxy.
  • --prometheus-endpoint // PROM_PROXY_PROMETHEUS_ENDPOINT: URL of your Prometheus instance.
  • --reload-interval // PROM_PROXY_RELOAD_INTERVAL: Interval in minutes to reload the auth config file.
  • --unprotected-endpoints // PROM_PROXY_UNPROTECTED_ENDPOINTS: Comma separated list of endpoints that do not require authentication.
  • --protected-endpoints // PROM_PROXY_PROTECTED_ENDPOINTS: Comma separated list of endpoints that are allowed after authentication. Pass an empty string to turn it off (i.e. to allow all endpoints).
  • --auth-type // PROM_PROXY_AUTH_TYPE: Type of authentication to use, one of basic, jwt
  • --auth-config // PROM_PROXY_AUTH_CONFIG: Authentication configuration.
    • for basic authentication: path to a configuration file following the Authn structure
    • for jwt authentication: either a path or an URL to a json containing a Json Web Keys Set (JWKS)
  • --aws // PROM_PROXY_USE_AWS: See below.

Use prometheus-multi-tenant-proxy run --help for more information.

Configure the proxy for basic authentication

The auth configuration is straightforward. Just create a YAML file my-auth-config.yaml with the following structure:

// Authn Contains a list of users
type Authn struct {
	Users []User `yaml:"users"`

// User Identifies a user including the tenant
type User struct {
	Username   string            `yaml:"username"`
	Password   string            `yaml:"password"`
	Namespace  string            `yaml:"namespace"`
	Namespaces []string          `yaml:"namespaces"`
	Labels     map[string]string `yaml:"labels"`

An example is available at configs/multiple.user.yaml file:

  - username: User-a
    password: pass-a
    namespace: tenant-a
  - username: User-b
    password: pass-b
    namespace: tenant-b

or if you need to allow multiple namespaces for a single user, an example is available at configs/multiple.namespaces.yaml file:

  - username: Happy
    password: Prometheus
    namespace: default
  - username: Sad
    password: Prometheus
    namespace: kube-system
  - username: Multiple
    password: Namespaces
    namespace: monitoring
      - default
      - kube-system
      - kube-public
  - username: Multiple
    password: NamespacesWithoutNamespace
      - default
      - kube-system
      - kube-public

A tenant can contain multiple users. But a user is tied to a single tenant.

Tenant definition usually contains a set of labels. Starting from v1.7.0 it's possible to add these labels to a new labels section to the user definition to inject these labels on queries for that user.

Example available at configs/sample.labels.yaml file:

  - username: Happy
    password: Prometheus
      app: happy
      team: america
  - username: Sad
    password: Prometheus
      namespace: kube-system
  - username: bored
    password: Prometheus
      - default
      - kube-system
      dep: system

Configure the proxy for JWT authentication

Under the hood, the proxy uses keyfunc to load keys (in JWKS format), and go-jwt for validating JWT tokens.

The Json Web Keys Set (JWKS) can be loaded either from a file or an URL, and will be reloaded automatically following the --reload-interval parameter.

An example of a valid JWKS containing both an HS256 (hmac, symmetric) and an RS256 (rsa, asymmetric) key is available at internal/app/prometheus-multi-tenant-proxy/.jwks_example.json. More examples are provided in the keyfunc readme. You can also use to generate valid JWKs.

Once the proxy is aware of one or more JWKS keys, it is ready to authorize requests based on signed JWT tokens. The token is extracted from one of two locations with the given precedence:

  1. the Authorization header, in the form Authorization: Bearer <TOKEN>, or, if not present,
  2. the Token header, in the form Token: <TOKEN>.

For the token to be valid, it must:

  • contain a kid (key ID) in the header that matches the kid of a known key in the JWKS,
  • contain a claim in the payload called namespaces, with zero or more values. For example:
      "namespaces": ["foo", "bar"]
  • contain a claim in the payload called labels, with zero or more values. For example:
      "labels": {
        "app": "happy",
        "team": "america"
  • have been signed with the key in the JWKS matching the kid found in the JWT header.

To test the proxy using JWT tokens, you can use the .jwks_example.json file above to run the proxy and generate a JWT token using Ensure you chose the HS256 algorithm and paste the following token:


To verify the signature, replace your-256-bit-secret with lala in the "verify signature" section. You can now use curl, for example:

curl -H "Authorization: Bearer $TOKEN" http://localhost:9092/api/v1/query\?query\=net_conntrack_dialer_conn_attempted_total

Proxy to Amazon Managed Service for Prometheus

All requests to an AWS managed prometheus service need a signature in the Authorization header, which is calculated based on the request URL, headers, and body. Since the proxy modifies the request on the fly, any existing signature will be invalidated. This is why prometheus-multi-tenant-proxy incorporates AWS signature v4.

To enable AWS signature, use either the --aws flag or set the environment variable PROM_PROXY_USE_AWS=true.

The credentials used for signing are taken from environment variables, such as AWS_ACCESS_KEY and AWS_SECRET_KEY (see AWS credentials environment variables for a full list). In case your prometheus service doesn't live in the us-east-1 region, you will also have to set either AWS_REGION or AWS_DEFAULT_REGION to the region's shorthand. Note that the AWS service is always set to aps, which is the shorthand for AWS Prometheus Service.

For more information, see:

Namespaces or labels

The proxy can be configured to use either namespaces and/or labels to query Prometheus. At least one must be configured, otherwise the proxy will not proxy the query to Prometheus. (It could lead to a security issue if the proxy is not configured to use namespaces or labels)

Deploy on Kubernetes using Helm

The proxy can be deployed on Kubernetes using Helm. The Helm chart is available at k8spin/prometheus-multi-tenant-proxy. Find the chart's documentation on its


$ helm repo add k8spin-prometheus-multi-tenant-proxy
$ helm repo update
$ helm upgrade --install prometheus-multi-tenant-proxy k8spin-prometheus-multi-tenant-proxy/prometheus-multi-tenant-proxy --set proxy.prometheusEndpoint=http://prometheus.monitoring.svc.cluster.local:9090

Example using flux

kind: HelmRepository
  name: prometheus-multi-tenant-proxy
  namespace: flux-system
    phase: seed
  interval: 1m0s
kind: HelmRelease
  name: prometheus-multi-tenant-proxy
  namespace: flux-system
  timeout: 30m
      retries: 3
      retries: 3
  interval: 1m
      chart: prometheus-multi-tenant-proxy
      version: "1.10.0"
        kind: HelmRepository
        name: prometheus-multi-tenant-proxy
        namespace: flux-system
      interval: 1m
  releaseName: prometheus-multi-tenant-proxy
  targetNamespace: monitoring
  storageNamespace: monitoring
  valuesFrom: []
      prometheusEndpoint: http://prometheus.monitoring.svc.cluster.local:9090
          authn: |
              - username: User-a
                password: pass-a
                namespace: tenant-a
              - username: User-b
                password: pass-b
                namespace: tenant-b

Build it

If you want to build it from this repository, follow the instructions below:

$ docker run -it --entrypoint /bin/bash --rm golang:1.20.5-bookworm
root@9b2da74fb4b8:/go# git clone
Cloning into 'prometheus-multi-tenant-proxy'...
remote: Enumerating objects: 403, done.
remote: Counting objects: 100% (84/84), done.
remote: Compressing objects: 100% (37/37), done.
remote: Total 403 (delta 55), reused 55 (delta 41), pack-reused 319
Receiving objects: 100% (403/403), 347.67 KiB | 2.92 MiB/s, done.
Resolving deltas: 100% (173/173), done.
root@9b2da74fb4b8:/go# cd prometheus-multi-tenant-proxy/cmd/prometheus-multi-tenant-proxy/
root@9b2da74fb4b8:/go/prometheus-multi-tenant-proxy/cmd/prometheus-multi-tenant-proxy# go build
go: downloading v2.25.6
go: downloading v2.1.0
go: downloading v5.0.0
go: downloading v0.44.0
go: downloading v0.7.0
go: downloading v3.0.1
go: downloading v0.26.0
go: downloading v1.0.0-rc.2
go: downloading v0.21.7
go: downloading v0.0.0-20210307161603-1c9aa721a97a
go: downloading v0.25.0
go: downloading v1.15.1
go: downloading v0.0.0-20230321023759-10a507213a29
go: downloading v0.0.0-20230301143203-a9d515a09cc2
go: downloading v0.20.3
go: downloading v1.5.0
go: downloading v1.3.1
go: downloading v1.11.3
go: downloading v1.2.0
go: downloading v1.14.0
go: downloading v1.14.0
go: downloading v2.0.2
go: downloading v0.0.0-20201216005158-039620a65673
go: downloading v1.0.1
go: downloading v2.2.0
go: downloading v0.3.0
go: downloading v0.42.0
go: downloading v0.9.0
go: downloading v1.30.0
go: downloading v0.22.3
go: downloading v0.22.1
go: downloading v0.9.1
go: downloading v2.4.0
go: downloading v0.21.4
go: downloading v0.21.2
go: downloading v0.20.8
go: downloading v1.2.3
go: downloading v2.1.0
go: downloading v0.0.0-20221122212121-6b5c0a4cb7fd
go: downloading v1.5.3
go: downloading v1.0.4
go: downloading v0.7.0
go: downloading v0.7.7
go: downloading v0.19.6
go: downloading v1.2.2
go: downloading v0.20.2
go: downloading v0.2.1
go: downloading v0.6.0
go: downloading v1.8.2
go: downloading v1.0.0
go: downloading v1.10.0
go: downloading v1.3.2
go: downloading v1.1.1
go: downloading v1.2.1
go: downloading v1.0.0
go: downloading v1.0.0
root@9b2da74fb4b8:/go/prometheus-multi-tenant-proxy/cmd/prometheus-multi-tenant-proxy# ./prometheus-multi-tenant-proxy
   Prometheus multi-tenant proxy - Makes your Prometheus server multi tenant

   Prometheus multi-tenant proxy [global options] command [command options] [arguments...]


   Angel Barrera <>
   Pau Rosello <>

   run      Runs the Prometheus multi-tenant proxy
   help, h  Shows a list of commands or help for one command

   --help, -h     show help
   --version, -v  print the version

Build the container image

If you want to build a container image with this proxy, run:

$ docker build -t prometheus-multi-tenant-proxy:local -f build/package/Dockerfile .

After built, just run it:

$ docker run --rm prometheus-multi-tenant-proxy:local

Using this project at work or in production?

See for what companies are doing with this project today.


The scripts and documentation in this project are released under the GNU GPLv3