Allow relative URL in web.external-url

[malcolmr]

I'm serving Prometheus behind a reverse-proxy under a /prometheus path, so in 0.16, I'd set web.external-url to /prometheus.

0.18 complains about that:

ERRO[0000] invalid Prometheus URL: /prometheus           source=main.go:65

Is it intended that web.external-url must be an absolute URL? (As /foo is a valid path-absolute relative URL.)

The help reads: "The URL under which Prometheus is externally reachable (for example, if Prometheus is served via a reverse proxy). Used for generating relative and absolute links back to Prometheus itself. If the URL has a path portion, it will be used to prefix all HTTP endpoints served by Prometheus. If omitted, relevant URL components will be derived automatically."

It seems like either the validation is too strict, or that last sentence is (now) incorrect.

[grobie]

That field expects an absolute URL. The highlighted sentence means if that parameter is not set at all, a default will be chosen, which uses / as path.

Is it not an option to use --web.external-url http://$(hostname):9090/prometheus?

[chancez]

I'm running into this myself because I'm running prometheus in Kubernetes and attempting to access it locally via kubectl proxy, which in effect, opens up a proxy locally to access my service under http://127.0.0.1:8001/api/v1/proxy/namespaces/default/services/prometheus/. I'd rather not change the hostname since that messes with other stuff. Being able to avoid the redirect prometheus does for external-url would be nice as well.

My main issue is that I need to be able to specify the prefix so this doesn't happen when I try to access it via chrome:

http://127.0.0.1:8001/static/js/graph_template.handlebar?_=1461656431100 Failed to load resource: the server responded with a status of 404 (Not Found)

[malcolmr]

I have added the hostname (and implicitly port), and it works fine, though I do have to make sure to omit it when I'm running locally, which is a bit of a pain (because the port is different behind the proxy).

I guess this is perhaps a feature request to revert to the behaviour (unexpectedly?) permitted in 0.16.

[grobie]

I'm open for allowing relative URLs as well.

[aaskey]

Set both web.external-url and web.route-prefix will work. Below is my config:

- "-web.external-url=http://localhost:8080/api/v1/proxy/namespaces/default/services/prometheus"
- "-web.route-prefix=/"

[tylux]

@chancez I know this is old but did you figure out the issue of /graph not loading correctly with that 404 error?

[thenayr]

@tylux It won't.

If you are accessing it through Kubernetes my best advice is to run kubectl port-forward .... to your Prometheus pod to access the Prometheus web-ui through localhost, or to setup a single static DNS entry that you access your Prometheus web-ui through (you'll probably need a Kubernetes service with Load Balancer type).

[tylux]

this particular instance is not inside Kubernetes, it is a standalone Prometheus box that is being proxy_passed from a Nginx box

been using the following snippet from #2193 and it mostly works other than that 404 on "static/js/graph_template.handlebar?_=1461656431100" resource on the /graphs page.

location /prometheus/ {
        proxy_set_header Accept-Encoding "";
        proxy_pass http://<real-prometheus-host>:9090/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        sub_filter_types text/html;
        sub_filter_once off;
        sub_filter '="/' '="/prometheus/';
        sub_filter '="/static/' '="/static/prometheus/';
    }

[discordianfish]

Can we maybe be a bit smarter about figuring out the web root ourselves?
At least for path in the served HTML we should be able to read the requested path from request.URL, right? You still need to configure -web.external-url for links in alerts though. Not sure if it makes sense for that to support relative URLs.

[montanaflynn]

We are also trying to set up a portal to prometheus web UI from behind an nginx reverse proxy.

Where do we set the config? I couldn't find any examples of setting these fields in the docs.

- "-web.external-url=..."
- "-web.route-prefix=/"

[linusguan]

@montanaflynn You set them as arguments when you run Prometheus, if you get it to work, do you mind sharing your settings?

[discordianfish]

I can't figure out how this is suppose to work either. I'm using the kubernetes api proxy and if I don't set external-url, it can't load the js files for /graph. If I set it to something (and break alternative ways to access this), that gets added additionally to the path prefix in the request. Unless I miss something, I believe external-url isn't doing what was intended.

I believe we could make this work for 90% of users without rewriting requests or flags in prometheus. From what I can tell, something in prometheus is already using the request path to figure out url prefixes. This is /graph snippet without external-url set:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
    <title>Prometheus Time Series Collection and Processing Server</title>
    <link rel="shortcut icon" href="/api/v1/namespaces/default/services/prometheus/proxy/static/img/favicon.ico?v=3afb3fffa3a29c3de865e1172fb740442e9d0133">
    <script src="/api/v1/namespaces/default/services/prometheus/proxy/static/vendor/js/jquery.min.js?v=3afb3fffa3a29c3de865e1172fb740442e9d0133"></script>
    <script src="/api/v1/namespaces/default/services/prometheus/proxy/static/vendor/bootstrap-3.3.1/js/bootstrap.min.js?v=3afb3fffa3a29c3de865e1172fb740442e9d0133"></script>

    <link type="text/css" rel="stylesheet" href="/api/v1/namespaces/default/services/prometheus/proxy/static/vendor/bootstrap-3.3.1/css/bootstrap.min.css?v=3afb3fffa3a29c3de865e1172fb740442e9d0133">
    <link type="text/css" rel="stylesheet" href="/api/v1/namespaces/default/services/prometheus/proxy/static/css/prometheus.css?v=3afb3fffa3a29c3de865e1172fb740442e9d0133">

    <script>
      var PATH_PREFIX = "";

As you see, the path are prefixed properly, this is why without setting anything, everything except the graph ui works. There I get this error: Failed to load :8001/static/js/graph_template.handlebar?v=3afb3fffa3a29c3de865e1172fb740442e9d0133&_=1499178246922. That's because we load this from js here: https://github.com/prometheus/prometheus/blob/master/web/ui/static/js/graph.js#L885 - and PATH_PREFIX isn't set.

Now setting external-url breaks everything because, while it fixes loading the handlebar template, it get added to the html links additionally to the correct, automatic prefixes.

The only thing I need to figure out to fix this is how the correct prefixing works.. I'm pretty surprised and have no idea yet..

[discordianfish]

Hope not the k8s apiserver proxy is doing this?

[linusguan]

@discordianfish I think even if you set the correct prefix by setting "-web.external-url=... -web.route-prefix=/", the /graph part is still not going to work, there is a temporary solution in another thread using nginx with custom rewrite rules.

see here: #2193

[fabxc]

@discordianfish oh yea, kube API server proxy fiddles around with your HTML IIRC.

[discordianfish]

That would explain all my and probably lot of other people' confusion. If it's really that and nothing on the prometheus side, it's impossible to fix this without introducing yet another flag which only sets a prefix for things we load in js..

[thenayr]

No, this isn't the Kube Proxy fault. Other software handles this perfectly fine (see Grafana/Kibana etc.)

[discordianfish]

@thenayr Do you know how this is suppose to work? We might lack some understanding how web applications handle this in general. I've opened kubernetes/kubernetes#48503 to clarify the apiserver proxy's behavior.

[thenayr]

@discordianfish I would model after what Kibana does for server.basePath.

A few references:

1 - kibana docs
2 - kibana docker image
3 - kibana k8s controller

In the 2nd/3rd links you can see how Kubernetes logging addon only specifies the base path as /api/v1/proxy/namespaces/kube-system/services/kibana-logging and it works across all clusters no matter what the root URL is.

[discordianfish]

This looks pretty much like what external-url is doing, no? If I set that, I get the prefix twice. Once due the prefixing and then something else (kube api's proxy or something in the 'middleware'). Either way, I still don't understand what is prefixing the URLs in the prometheus html when external-url is not set.

[thenayr]

Here's one example of where a script is being loaded over absolute path instead of external-url:

This is pulled in via ajax here

This particular piece causes the graph page to not load properly.

[discordianfish]

I know, as I said: We can fix that by setting the prefix path, but that also prefixes the HTML. In this specific case we would need to prefix the ajax stuff but not the HTML. But first we need to understand what is going on here and I don't have time for that deep dive right now. I've opened kubernetes/kubernetes#48503 and hope to get some clarification on the proxy behavior.

[thenayr]

Understood.

Dug a bit into the k8s proxy code and it looks like the following happens:

URL's that aren't "proxy aware" aka don't include the k8s proxy path (/api/v1/proxy...) in them will be rewritten to include them, as seen here

URL's that already include the proxy path are deemed "proxy aware" and will not be rewritten as seen here

I believe this would explain the current behavior of Prometheus going through kube-proxy with an external-url set to an absolute path.

The proxy looks for relative links that don't reference the proxy path, in the case of prometheus it is getting absolute URL's throughout, so it will rewrite them again to include the proxy path (but the ajax links work fine because those aren't rewritten).

[brancz]

External URL is what is used to render links in the HTML (not exclusively, but for the purpose of this explanation). And route prefix sets the routes the actual http routes are registered to which for the better or worse defaults to the path of the configured external URL (I've proposed to change this, but we decided not to). As the kube proxy strips the prefix of /v1/proxy/.../graph the request ends up being /graph. If I understand the problem correctly solution here is that you need to set external URL to the full proxy URL and route prefix to just "/" as described above. This is solved in the new alertmanager UI, but applying the same method would cause breaking changes to the Prometheus UI if we introduce it in the same way in the same way as the new Alertmanager UI does it.

We'll probably only be able to solve this confusion once we rewrite the Prometheus UI in the same way.

[discordianfish]

As @thenayr figured out, all we need to do is support relative URLs. The apiserver proxy will detect them as already prefixed and leave them alone.

[sylr]

FWIW I've been able to setup (after a lot of efforts) something pretty decent.

Network flow :

• internet
  • nginx ingress controller (https://prometheus.company.com/[prometheus-server]/graph)
    • oauth2_proxy (http://prometheus-company-com-oauth2-proxy:4180/[prometheus-server]/graph)
      • nginx reverse proxy (http://localhost:18080/prometheusX/graph)
        • prometheus-server-[k8s-euw-dev] (http://prometheus-server-k8s-euw-dev.monitoring.svc.cluster.local/graph)
        • prometheus-server-[k8s-euw-test] (http://prometheus-server-k8s-euw-dev.monitoring.svc.cluster.local/graph)
        • prometheus-server-[k8s-euw-prod] (http://prometheus-server-k8s-euw-dev.monitoring.svc.cluster.local/graph)

prometheus-company-com-oauth2-proxy deployment :

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "3"
  generation: 3
  labels:
    app: prometheus-company-com-oauth2-proxy
  name: prometheus-company-com-oauth2-proxy
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-company-com-oauth2-proxy
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: prometheus-company-com-oauth2-proxy
    spec:
      containers:
      - image: sylvainlectra/oauth2-proxy:0.12
        imagePullPolicy: IfNotPresent
        name: oauth2-proxy
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /conf
          name: oauth2-proxy-config-volume
      - image: nginx:1.13
        imagePullPolicy: IfNotPresent
        name: nginx
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/nginx/conf.d
          name: nginx-config-volume
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 420
          name: prometheus-company-com-oauth2-proxy-conf
        name: oauth2-proxy-config-volume
      - configMap:
          defaultMode: 420
          name: prometheus-company-com-nginx-conf
        name: nginx-config-volume
      - configMap:
          defaultMode: 420
          name: prometheus-company-com-oauth2-proxy-conf
        name: config-volume

ingress prometheus-company-com-oauth2-proxy

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/rewrite-target: /
    kubernetes.io/ingress.allow-http: "false"
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
  generation: 4
  name: prometheus-company-com-oauth2-proxy
  namespace: monitoring
spec:
  rules:
  - host: prometheus.company.com
    http:
      paths:
      - backend:
          serviceName: prometheus-company-com-oauth2-proxy
          servicePort: 4180
        path: /
  tls:
  - hosts:
    - prometheus.company.com
    secretName: wildcard-company-com-tls

oauth2_proxy conf (interesting bits):

http_address = "0.0.0.0:4180"
upstreams = [
    "http://localhost:18080/"
]

nginx reverse proxy conf:

server {
    listen *:18080;
    resolver 10.200.0.10 valid=10s;

    location ~* ^/(?<cluster>k8s[a-z0-9-]+) {
        rewrite ^/k8s[a-z0-9-]+/?$       /$cluster/graph    redirect;
        rewrite ^/k8s[a-z0-9-]+/(.*)$    /$1                break;
        proxy_pass http://prometheus-server-$cluster.monitoring.svc.cluster.local:80;

        sub_filter_types text/html;
        sub_filter_once off;
        sub_filter 'var PATH_PREFIX = "";' 'var PATH_PREFIX = "/$cluster";';
        sub_filter '="/' '="/$cluster/';
    }
}

Hope it helps.

Cheers.

[aduzsardi]

It's working like this without any issues

/path/to/prometheus --config.file="config.yml" --storage.tsdb.path="./data" --web.listen-address="127.0.0.1:9090" --web.external-url="http://MY_REVERSE_PROXY_DNS/prometheus/" --web.route-prefix="/"

/path/to/alertmanager --config.file="config.yml" --storage.path="./data" --web.external-url="http://MY_REVERSE_PROXY_DNS/alertmanager/" --web.route-prefix="/" --web.listen-address="127.0.0.1:9093" --mesh.listen-address=""

Nginx config

server {
  listen 80 default_server;

  location /alertmanager/ {
    proxy_pass http://127.0.0.1:9093/;
  }

  location /prometheus/ {
    proxy_pass http://127.0.0.1:9090/;
  }

  location /grafana/ {
    proxy_pass http://127.0.0.1:3000/;
  }

  location / {
    try_files $uri $uri/ =404;
  }

}

Hopefully this helps somebody :)

[RKdev]

I've been having the same issue. Web UI comes up via kubectl proxy, but the Graph button does nothing.
Not sure if this is a clue or not, but when I check Command-Line Flags in the ui --web.external-url is empty

[rafaelmagu]

I'm not sure if it's entirely related, but if I don't provide --web.external-url when starting Prometheus, it appears to ignore --web.route-prefix.

[rafaelmagu]

To better illustrate my comment, I have HAproxy on a bastion, with path-based routing for different backend services. For Prometheus, any request starting with /prometheus gets sent to the Prometheus backend, without rewrites.

The source for /prometheus/graph is rendered without the route prefix:

If I add --web.external-url=http://anything/prometheus to the command line and change --web.route-prefix to /, I'll get a 404 for /static content. If I then change my HAproxy config to drop /prometheus from the URLs before sending the requests to the backend, it works:

Note: by anything, I really do mean anything, even an invalid FQDN works with --web.external-url.

Alertmanager does not present this behaviour at all. Setting only --web.route-prefix=/alertmanager and configuring HAproxy to not rewrite URLs for the Alertmanager backend works as intended.

(I realise Alertmanager is more of a single page app, so this would not be an issue anyway)

[rafaelmagu]

With --web.external-url=http://anything/prometheus and --web.route-prefix=/prometheus, the page renders correctly without rewrites from HAproxy.

[deveshmehta]

Thanks Rafael for the details.

In my case the culprit is the recently added Kibana entry in the NGINX config which has a location /api which is conflicting with prometheus/api.

Will you be able to suggest the any solution/best practice to handle this.