2.3.0 breaks our reverse proxy setup

[dominikschulz]

Bug Report

What did you do?

We upgraded our Prometheus instances from 2.2.1 to 2.3.0.

What did you expect to see?

The Webinterface should still be working as before.

What did you see instead? Under which circumstances?

All static assets failed to load due to a wrong path prefix.

Environment

We run several Prometheus instances behind an nginx reverse proxy that handles authentication (among other duties).
Our Prometheus instances are available on URLs like these (where "foo" and "bar" are the names of different instances):

• https://www.example.org/p8s/foo
• https://www.example.org/p8s/bar

We are running Prometheus with these (relevant) commandline arguments:

• --web.external-url=https://www.example.org/p8s/foo
• --web.route-prefix=/

Our nginx.conf contains a location stanza for each Instance like this:

# ensure trailing slash
location = /p8s/foo {
  return 302 /p8s/foo/;
}
location /p8s/foo/ {
  proxy_pass http://prometheus-foo/;
  # more proxy header options
}

Note: The trailing slash will remove the leading part of the request URL matching the location name (i.e. https://www.example.org/p8s/foo/graph will hit the backend as http://<backend>/graph). See proxy_pass. This is not strictly necessary for Prometheus but makes much sense with the other services also serviced by this nginx and makes our lives a lot easier.

This setup worked perfectly fine with 2.2.x but broke with 2.3.0.

Example static asset URL with 2.2.1:
https://www.example.org/p8s/foo/static/vendor/js/jquery.min.js

Example static asset URL with 2.3.0:
https://www.example.org/static/vendor/js/jquery.min.js

I was suspecting some relation to #4025 but couldn't spot an issue with that, yet.

[brian-brazil]

Hmm, #4025 is incorrect and should be reverted. The route prefix is internal to the proxy<->Prometheus, and shouldn't affect how html is rendered.

[fabxc]

That PR is not the final code in 2.3 – there was another change (cd2820e) that supposedly fixed a bug #4025 introduced. Did that fix just a single case? Should both be reverted?

[dominikschulz]

The commit you mentioned doesn't fix #4025 for our setup.
For ours to work we'd really need pathPrefix to return opts.ExternalURL.Path and not opts.RoutePrefix (or /).

[brian-brazil]

#4031 needs to be evaluated separately.

[brian-brazil]

#4256 fixes this.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.