Configure Prometheus, Grafana, AlertManager to run behind Traefik HTTP Proxy using a subpath

[ricsanfre]

Feature request

Configure Prometheus, Alertmanager and Grafana to run behind Traefik HTTP Proxy using a subpath making them available at https://monitoring.picluster.ricsanfre/grafana , https://monitoring.picluster.ricsanfre/prometheus and https://monitoring.picluster.ricsanfre/alertmanager instead of using separate domains for each component.

NOTE: Linkerd-viz dashboard integration with Grafana, only works if Grafana runs behind /grafana subpath, so this change will make that integration work.

References

• Configuring Grafana documentation
• Hook Grafana with Linkerd-viz-dashboard

[ricsanfre]

Configuring prometheus and alertmanager

From the command line parameters of Prometheus (prometheus -h), there are two options to control the path and route configuration.

• --web.external-url=<URL>

  The URL under which Prometheus is externally reachable (for example, if Prometheus is served via a reverse proxy). Used for generating relative and absolute links back to Prometheus itself. If the URL has a path portion, it will be used to prefix all HTTP endpoints served by Prometheus. If omitted, relevant URL components will be derived automatically.
  This is the address (URL) that is externally reachable by Prometheus (after reverse proxy).
  Used to generate relative or absolute links (on a page or in a request).

• --web.route-prefix=<path>

  Prefix for the internal routes of web endpoints. Defaults to path of –web.external-url.
  Route prefix for Prometheus internal web requests.
  Defaults to the path in --web.external-url.

Alertmanager has same options.

Using PrometheusOperator these options can be configured for both CRDs Prometheus and AlertManager specs:
externalUrl and routePrefix.

Configuring these parameters in kube-prometheus-stack chart

These are the values that need to be configured.
externalUrl is pointing to the URL the user is using and routePrefix is kept as /, so internal request not need to have the /prometheus prefix.

alertmanager:
  alertmanagerSpec:
    externalUrl: http://monitoring.picluster.ricsanfre.com/alertmanager/
    routePrefix: /
prometheus:
  prometheusSpec:
    externalUrl: http://monitoring.picluster.ricsanfre.com/prometheus/
    routePrefix: /

Configuring Traefik Ingress

Traefik ingress route need to be configured using a stripPrefix middleware, to remove path prefix before sending the request to the backend service.

---

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: stripprefix
  namespace: monitoring
spec:
  stripPrefix:
    prefixes:
      - "/prometheus"
      - "/alertmanager"
    forceSlash: false

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: prometheus
  namespace: monitoring
spec:
  entryPoints:
    - web
  routes:
  - kind: Rule
    match: Host(`monitoring.picluster.ricsanfre.com`) && PathPrefix(`/prometheus`)
    services:
    - name: kube-prometheus-stack-prometheus
      port: 9090
      namespace: monitoring
    middlewares:
      - name: stripprefix
        namespace: monitoring

  - kind: Rule
    match: Host(`monitoring.picluster.ricsanfre.com`) && PathPrefix(`/alertmanager`)
    services:
    - name: kube-prometheus-stack-alertmanager
      port: 9093
      namespace: monitoring
    middlewares:
      - name: stripprefix
        namespace: monitoring

[ricsanfre]

Configuring Grafana

When running Grafana behind a proxy, you need to configure the domain name to let Grafana know how to render links and redirects correctly.

In the Grafana configuration file, change server.domain to the domain name and if it is running behind a sub path set root_url and server_from_subpath

[server]
domain = example.com
root_url = %(protocol)s://%(domain)s:%(http_port)s/grafana/
serve_from_sub_path = true

Configuring these parameters in kube-prometheus-stack chart

When deploying grafana subchart from kube-prometheus-stack the following values must be provided:

grafana:
  grafana.ini:
    server:
      domain: monitoring.internal
      root_url: "%(protocol)s://%(domain)s:%(http_port)s/grafana/"
      serve_from_sub_path: true

Configuring Traefik Ingress

Traefik ingress rule to serve Grafana, prometheus and alertmanager from monitor.picluster.ricsanfre.com

---

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: stripprefix
  namespace: monitoring
spec:
  stripPrefix:
    prefixes:
      - "/prometheus"
      - "/alertmanager"
      - "/grafana"
    forceSlash: false

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: prometheus
  namespace: monitoring
spec:
  entryPoints:
    - web
  routes:
  - kind: Rule
    match: Host(`monitoring.picluster.ricsanfre.com`) && PathPrefix(`/prometheus`)
    services:
    - name: kube-prometheus-stack-prometheus
      port: 9090
      namespace: monitoring
    middlewares:
      - name: stripprefix
        namespace: monitoring

  - kind: Rule
    match: Host(`monitoring.picluster.ricsanfre.com`) && PathPrefix(`/alertmanager`)
    services:
    - name: kube-prometheus-stack-alertmanager
      port: 9093
      namespace: monitoring
    middlewares:
      - name: stripprefix
        namespace: monitoring
  - kind: Rule
    match: Host(`monitoring.picluster.ricsanfre.com`) && PathPrefix(`/grafana`)
    services:
    - name: kube-prometheus-stack-grafana
      port: 80
      namespace: monitoring
    middlewares:
      - name: stripprefix
        namespace: monitoring