Links to /route-prefix/api/ but only responds on /api/

[candlerb]

What did you do?

Run prometheus with these flags:

--web.external-url=https://example.com/prometheus
--web.route-prefix=/prometheus

and behind an Apache reverse proxy

        ProxyPass /prometheus http://10.85.179.50:9090/prometheus
        ProxyPassReverse /prometheus http://10.85.179.50:9090/prometheus

Opened the page at https://example.com/prometheus/graph

What did you expect to see?

Usual prometheus front page

What did you see instead? Under which circumstances?

Usual prometheus front page but with red box: "Error loading available metrics!"

Opening the Javascript console showed that it got a 404 error trying to access
https://example.com/prometheus/api/v1/label/__name__/values?_=1502894473124

Testing this directly from the proxy:

root@wrn-mon2:~# curl -Lv 10.85.179.50:9090/prometheus/api/v1/label/__name__/values?_=1502894473124*   Trying 10.85.179.50...
* Connected to 10.85.179.50 (10.85.179.50) port 9090 (#0)
> GET /prometheus/api/v1/label/__name__/values?_=1502894473124 HTTP/1.1
> Host: 10.85.179.50:9090
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Wed, 16 Aug 2017 14:53:58 GMT
< Content-Length: 19
<
404 page not found
* Connection #0 to host 10.85.179.50 left intact

BUT:

root@wrn-mon2:~# curl -Lv 10.85.179.50:9090/api/v1/label/__name__/values?_=1502894473124
*   Trying 10.85.179.50...
* Connected to 10.85.179.50 (10.85.179.50) port 9090 (#0)
> GET /api/v1/label/__name__/values?_=1502894473124 HTTP/1.1
> Host: 10.85.179.50:9090
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Access-Control-Allow-Headers: Accept, Authorization, Content-Type, Origin
< Access-Control-Allow-Methods: GET, OPTIONS
< Access-Control-Allow-Origin: *
< Access-Control-Expose-Headers: Date
< Content-Type: application/json
< Date: Wed, 16 Aug 2017 14:54:23 GMT
< Transfer-Encoding: chunked
<
{"status":"success","data":["go_gc_duration_seconds","go_gc_duration_seconds_count"
...

So it appears that the api endpoint doesn't honour the route-prefix.

Environment

• System information:

  Linux 4.4.0-87-generic x86_64

• Prometheus version:

  prometheus, version 2.0.0-beta.1 (branch: HEAD, revision: 4dcb465029f39a2a3030333a81753f8a40ec90ce)
    build user:       root@d3f9974fac5a
    build date:       20170811-12:12:36
    go version:       go1.8.3
  

• Alertmanager version:

  n/a

• Prometheus configuration file:

  default

• Alertmanager configuration file:

  n/a

• Logs:

  n/a

[candlerb]

Aside: I could change my proxy to strip off the prefix when proxying. However, the static assets (e.g. javascript) are served under the /prometheus/ prefix and not at the top level.

root@wrn-mon2:~# curl http://10.85.179.50:9090/prometheus/static/vendor/js/jquery.min.js?v=4dcb465029f39a2a3030333a81753f8a40ec90ce
/*! jQuery v1.11.2 | (c) 2005, 2014 jQuery Foundation, Inc. | jquery.org/license */
!function(a,b){"object"==typeof module&&"object"==typeof module.exports
...

root@wrn-mon2:~# curl http://10.85.179.50:9090/static/vendor/js/jquery.min.js?v=4dcb465029f39a2a3030333a81753f8a40ec90ce
404 page not found

[candlerb]

This suggests a nasty proxy workaround which does appear to work!

        ProxyPass /prometheus/api http://10.85.179.50:9090/api
        ProxyPassReverse /prometheus/api http://10.85.179.50:9090/api
        ProxyPass /prometheus http://10.85.179.50:9090/prometheus
        ProxyPassReverse /prometheus http://10.85.179.50:9090/prometheus

[maximecharron]

@candlerb do you know if it worked in version 1.x?

I'll try to take a look tonight or tomorrow.

[candlerb]

do you know if it worked in version 1.x?

Sorry I don't, I have only been working with 2.0 beta.

[williamstein]

For what it is worth, I just hit this issue with the 2.0 beta, found it here, then build the dev-2.0 branch from source (the merged PR mentioned above), and it fixed the problem.

[candlerb]

Thanks. I will install 2.0.0-beta5, released 2 minutes ago :-)

[candlerb]

Looks good - thank you!

[gouthamve]

Closed in #3137

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.