when -web.route-prefix is set, resource and nav links are not prefixed

[bitplane]

What did you do?
Launch prometheus -web.route-prefix=/prometheus, set nginx to proxy http://localhost/prometheus to http://localhost:9090/prometheus

What did you expect to see?
A graph at http://localhost/prometheus/graph and the ability to navigate between pages.

What did you see instead? Under which circumstances?
Javascript doesn't load, page navigation doesn't work.

Static resources are links to href="/static/ rather than href="/prometheus/static/, nav menu and link to home also points to root rather than prefix:

            <li><a href="/status">Runtime &amp; Build Information</a></li>
            <li><a href="/flags">Command-Line Flags</a></li>
            etc

Environment

• System information:
  Linux 4.4.0-45-generic x86_64 / Ubuntu 16.04

• Prometheus version:
  prometheus_1.2.3+ds2-1_amd64.deb

[juliusv]

Yeah, that's a bit tricky. -web.route-prefix only changes the path prefix on which all endpoints are served, but it does not inject that knowledge into the HTML which loads the assets. See details in #1191.

Can you try instead: -web.external-url=http://localhost/prometheus

[bitplane]

My problem is that my hostname isn't fixed. When accessing the box externally I'm sometimes using a tunnel, so http://localhost:port/prometheus/ while internally it's http://internal-host/prometheus/, when I have get routing in place it'll be https://external-ip/prometheus and eventually https://domain.com/prometheus but only from places where I have my client cert installed, elsewhere I'll use a tunnel to the non-proxied url.

Unless there's something I've overlooked, I think it would work in all cases if we prefixed the path with web.external-url if there is one and use web.route-prefix if not. My go is pretty weak but I'll have a stab at this when I get a chance.

[jangaraj]

Try to rewrite/fix html links (no web.external-url or web.route-prefix). For example my nginx config (sub_filter rewrites html):

    location /prometheus/ {
        proxy_set_header Accept-Encoding "";
        proxy_pass http://<real-prometheus-host>:9090/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        sub_filter_types text/html;
        sub_filter_once off;
        sub_filter '="/' '="/prometheus/';
        sub_filter '="/static/' '="/static/prometheus/';
    }

[undeadops]

Solution proposed by @jangaraj works for me... don't really care for it being a solution the -web.route-prefix should be able to make it rewrite the static urls for CSS etc..

[s4s0l]

@bitplane FYI setting http://localhost/prometheus will not prevent it from working no matter what domain will you use

[tylux]

@jangaraj I tried out your nginx config, and it all seems to work except for the /graph page loading the query fields.

http:///static/js/graph_template.handlebar?_=1487884039083

when it should have /prometheus at the front

http:///prometheus/static/js/graph_template.handlebar?_=1487884039083

is that the config you are still running?

[jangaraj]

I see the same problem in my project https://github.com/monitoringartist/play.monitoringartist.com . Rewrite part

    sub_filter_types text/html;
    sub_filter_once off;
    sub_filter '="/' '="/prometheus/';
    sub_filter '="/static/' '="/static/prometheus/';

needs some improvement. Try to add another subfilter:

   sub_filter 'var PATH_PREFIX = "";' 'var PATH_PREFIX = "/prometheus";';

I'll test it later.

[tylux]

@jangaraj that additional sub_filter worked for me, Thanks!

[jangaraj]

OK, so current workaround is nginx proxy (no special prometheus settings web.external-url or web.route-prefix are required):

location /prometheus/ {
        proxy_set_header Accept-Encoding "";
        proxy_pass http://<real-prometheus-host>:9090/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        sub_filter_types text/html;
        sub_filter_once off;
        sub_filter '="/' '="/prometheus/';
        sub_filter '="/static/' '="/static/prometheus/';
        sub_filter 'var PATH_PREFIX = "";' 'var PATH_PREFIX = "/prometheus";';
}

[ageis]

I have the same problem, trying to run Prometheus through an nginx reverse proxy at /prometheus.

I've tried all the solutions and directives above and I am getting ERR_TOO_MANY_REDIRECTS back from nginx whenever I enable web.external-url or web.route-prefix. Any ideas?

[dhoppe]

I had the same problem and solved it with a sligthly different configuration.

[Unit]
Description=Prometheus
Wants=basic.target
After=basic.target network.target

[Service]
User=prometheus
Group=prometheus
ExecStart=/opt/prometheus/prometheus \
  -config.file=/etc/prometheus/prometheus.yml \
  -storage.local.path=/var/lib/prometheus \
  -web.console.libraries=/opt/prometheus/console_libraries \
  -web.console.templates=/opt/prometheus/consoles \
  -web.listen-address=127.0.0.1:9090 \
  -web.route-prefix=/prometheus

ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=always
RestartSec=42s

[Install]
WantedBy=multi-user.target

server {
    listen       80;
    server_name  centos-7.vagrant.dev;

    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }

    location /grafana/ {
        proxy_pass http://127.0.0.1:3000/;
    }

    location /prometheus/ {
        proxy_pass http://127.0.0.1:9090/prometheus/;

        proxy_set_header Accept-Encoding "";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        sub_filter_types text/html;
        sub_filter_once off;
        sub_filter '="/' '="/prometheus/';
        sub_filter '="/static/' '="/static/prometheus/';
        sub_filter 'var PATH_PREFIX = "";' 'var PATH_PREFIX = "/prometheus";';
    }

    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }
}

For some reason I had to append /prometheus, because the sub_filter alone would not work.

Unfortunately you also have to change the metrics_path for the Prometheus job. Otherwise Prometheus is not able to determine the status of the Prometheus endpoint.

[jangaraj]

@dhoppe You need to add path /prometheus into your proxy_pass only because you are using -web.route-prefix=/prometheus.

[dhoppe]

@jangaraj I know, but without it always jumps back from /prometheus to /graph and shows an error page. That is the reason why I added the -web.route-prefix.

[jangaraj]

@dhoppe I agree. Your workaround -web.route-prefix=/prometheus + /prometheus in proxy_pass is more user friendly. 👍

[stevepeak]

This configuration works. Thank you for your help 👍

[discordianfish]

Looks to me like #1791 alone isn't very useful without this getting fixed, right?

[brancz]

Note -web.route-prefix is only there to prefix the HTTP handlers when they are registered. -web.external-url in contrast is what is used to prefix the assets in the UI, and by default -web.external-url makes -web.route-prefix default to its path.

[fabxc]

@discordianfish getting what fixed? A summary of the findings from this thread would be appreciated as it's literally nginx configs been thrown around.

[discordianfish]

Oh I understood this issue being about assets not prefixed. If that is already fixed, I guess this should get closed?

[fabxc]

Okay, so that should happen afaics: https://github.com/prometheus/prometheus/blob/master/web/ui/templates/_base.html#L6-L14
I'll close here. If anyone still encounters any issues, please reopen.

[discordianfish]

Yes, but also see my comment here. I think this is broken right now in general: #1583 (comment)
Happy for input over there. Still a mystery to me what prefixes the path when not using external-url nor route-prefix.

[trentbullard]

when i implement @dhoppe final solution the web interface loads correctly, but when i open the Status>Targets page, prometheus shows (0/1 up) and the endpoint is http://localhost:9090/metrics

Shouldn't this be .../prometheus/metrics?

edit: added metrics_path: /prometheus/metrics to the yml. fixed

[mclamb]

Does the solution proposed by @jangaraj still work for everyone? I am trying to get path-based routing working without having to change any prometheus settings like web.external-url. The sub_filter proposals above work if I navigate to /prometheus/graph, but /prometheus alone gets redirected to /graph. I have also played around with rewrite rules such as the following to no avail:

rewrite ^/graph/(.*)$ /prometheus/$1 break;

Thanks!

[mclamb]

I found a comment by @sylr in #1583 that seems to resolve my issue with the /graph redirect, in combination with the solution by @jangaraj above. The full location clause is:

location /prometheus/ {
        proxy_set_header Accept-Encoding "";
        proxy_pass http://<real-prometheus-host>:9090/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        sub_filter_types text/html;
        sub_filter_once off;
        sub_filter '="/' '="/prometheus/';
        ## APPEARS TO BE UNNECESSARY? sub_filter '="/static/' '="/static/prometheus/';
        sub_filter 'var PATH_PREFIX = "";' 'var PATH_PREFIX = "/prometheus";';

        rewrite ^/prometheus/?$ /prometheus/graph redirect;
        rewrite ^/prometheus/(.*)$ /$1 break;
}

Can anyone else confirm that these are the minimal settings required to get path-based routing working properly without resorting to editing prometheus config itself (i.e setting web.external-url)?

Thanks

[kvishweshwar]

@mclamb, Yes it is working properly for Prometheus, without having to have any kind of configuration changes, but not working for Grafana.

Thanks!

[kvishweshwar]

#5908 this may be useful to resolve Grafana related issues.

[AJ15416]

Does anyone have an apache equivalent to @mclamb 's solution? I am still not able to solve the /prometheus redirecting to /graph problem. I have the following and I am able to get everything except the redirect working.

ProxyPass /prometheus/ "http://prometheus:80/"
RedirectMatch /prometheus /prometheus/
<Location /prometheus/>
        RequestHeader unset Accept-Encoding
        AddOutputFilterByType SUBSTITUTE text/html
        Substitute "s|=\"/|=\"/prometheus/"
        Substitute "s|var PATH_PREFIX = \"\";|var PATH_PREFIX = \"/prometheus/\";"
</Location>

This is the portion I am having trouble with

        rewrite ^/prometheus/?$ /prometheus/graph redirect;
        rewrite ^/prometheus/(.*)$ /$1 break;

I've tried working with RewriteEngine, Rewrite, and just throwing another RedirectMatch in there, but nothing seems to get the job done.

[foscoj]

I found a comment by @sylr in #1583 that seems to resolve my issue with the /graph redirect, in combination with the solution by @jangaraj above. The full location clause is:

location /prometheus/ {
        proxy_set_header Accept-Encoding "";
        proxy_pass http://<real-prometheus-host>:9090/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        sub_filter_types text/html;
        sub_filter_once off;
        sub_filter '="/' '="/prometheus/';
        ## APPEARS TO BE UNNECESSARY? sub_filter '="/static/' '="/static/prometheus/';
        sub_filter 'var PATH_PREFIX = "";' 'var PATH_PREFIX = "/prometheus";';

        rewrite ^/prometheus/?$ /prometheus/graph redirect;
        rewrite ^/prometheus/(.*)$ /$1 break;
}

Can anyone else confirm that these are the minimal settings required to get path-based routing working properly without resorting to editing prometheus config itself (i.e setting web.external-url)?

Thanks

Thanks, still working for prometheus via nginx!

[fchiorascu]

I found a comment by @sylr in #1583 that seems to resolve my issue with the /graph redirect, in combination with the solution by @jangaraj above. The full location clause is:

location /prometheus/ {
        proxy_set_header Accept-Encoding "";
        proxy_pass http://<real-prometheus-host>:9090/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        sub_filter_types text/html;
        sub_filter_once off;
        sub_filter '="/' '="/prometheus/';
        ## APPEARS TO BE UNNECESSARY? sub_filter '="/static/' '="/static/prometheus/';
        sub_filter 'var PATH_PREFIX = "";' 'var PATH_PREFIX = "/prometheus";';

        rewrite ^/prometheus/?$ /prometheus/graph redirect;
        rewrite ^/prometheus/(.*)$ /$1 break;
}

Can anyone else confirm that these are the minimal settings required to get path-based routing working properly without resorting to editing prometheus config itself (i.e setting web.external-url)?

Thanks

For me this is not working at all.

[fchiorascu]

#4925