Support for reverse proxy that changes path

[neelance]

I am trying to access Prometheus via the Kubernetes reverse proxy (kubectl proxy). The resulting URL is http://localhost:8001/api/v1/proxy/namespaces/prod/services/prometheus/graph which gets proxied to the path /graph on the Prometheus server. Problem is that the JS can not load its resources, since it tries to use http://localhost:8001/static/js/ instead of http://localhost:8001/api/v1/proxy/namespaces/prod/services/prometheus/static/js/.

I found the -web.external-url flag, but it does not solve my problem. If I do -web.external-url=/api/v1/proxy/namespaces/prod/services/prometheus/ then http://localhost:8001/api/v1/proxy/namespaces/prod/services/prometheus/graph does not work any more, because it would need to be http://localhost:8001/api/v1/proxy/namespaces/prod/services/prometheus/api/v1/proxy/namespaces/prod/services/prometheus/graph. So I can't solve the issue with the flag because it expects that the reverse proxy does not change the path, which the Kubernetes proxy does.

[juliusv]

@neelance Interesting, https://cloud.google.com/container-engine/docs/kubectl/proxy says it's for connecting to the Kubernetes API server, but I guess people use it for connecting to other things as well? So it basically only works for web pages that only have relative paths in links and no absolute ones?

[juliusv]

The problem with making all links relative in Prometheus is that at least console template library functionality from https://github.com/prometheus/prometheus/blob/master/console_libraries/prom.lib can be included from arbitrary path depths, so it would add complexity to make it come up with the right relative links somehow.

@neelance Is being able to use something through kubectl proxy considered pretty much essential when using Kubernetes, or more of a nice-to-have?

[neelance]

@juliusv kubectl proxy is very convenient, because it gives you access to internal web interfaces of your cluster with authentication provided by Kubernetes.

I think there is no need to make all paths relative, the approach with -web.external-url is essentially the right one. The only problem is that it also affects the routes of the HTTP server in Prometheus instead of just using the path in JS for building absolute URLs. Why is the changing of the routes necessary? Advanced reverse proxies can map the path to anything, including /. More simple reverse proxies like the one of Kubernetes are fixed to a mapping to /. Are there reverse proxies which can only do a 1-to-1 mapping?

[juliusv]

Not sure how much more complex it usually is to set up a proxy that alters the path like you suggest, but there's another reason why we include the path prefix in the HTTP routes right now: if we didn't, it would be impossible to use the Prometheus server UI directly, even though you often only need to use a proxy in certain situations (like when you're accessing it from a wall screen in a restricted dashboard VPN). One idea would be to additionally handle all routes without the prefix, but that could cause collisions between the chosen path prefix and a route.

So if we want to support proxies that change the path, we will have to add a separate flag to turn off path prefixes for the actual HTTP routes and only include the path prefix in links back to Prometheus. I can send a PR and then people can discuss whether they like the idea or think that it's one flag too much :)

Other ideas welcome.

[chancez]

Im sort of a fan of how Grafana handles all of this: https://github.com/grafana/grafana/blob/master/conf/defaults.ini#L24-L48

Basically you domain and enforce_domain is effectively web.external_url and #1193 web.prefix-routes. However I'm not convinced the above PR alone is enough for all use-cases, for example when you don't want to redirect, but do want both the to set a prefix on both the server-side and front-end (not just the frontend, which is what the above PR allows).

I'd also like to say the current approach Prometheus takes is a bit confusing, but Grafana's configuration is very obvious in how it would behave (to me at least) so it would be worth trying to approach this similarly.

Also, Grafana's approach is pretty flexible because you can interpolate, or just hardcode the root_url. I don't really think interpolation is needed, but being able to override the whole URL seen by the browser in the config, without the redirect logic would be great.

[matthiasr]

Just capturing the TL;DR of a discussion in the Kubernetes slack here:

This is not so much an issue with kubectl proxy because that opens a layer 4 path to Prometheus. The problem arises when using the proxy built into the Kubernetes api-server. It allows access to services running in Kubernetes through a URL like https://some-host//api/v1/proxy/namespaces/default/services/prometheus without needing to cross with or set up any ingresses or LoadBalancers (which cost $$$ on cloud providers). This is the main and by default only way to access services running in a cluster provisioned using any of the standard methods in a cloud provider. I think it is pretty important to make this work.

[jnardiello]

Absolutely +1 to what @matthiasr just mentioned. I spent the whole day trying to understand what was the issue here. Having this fixed it's essential in order to even try prometheus on k8s

[fabxc]

Okay, so ultimately distinguishing between frontend and backend path rewriting both seem necessary.

@chancez did I get your objection correctly, that the missing use case in #1193 is that one might want different path prefixes for the frontend and backend paths?
Would that be solved by having two different flags – -web.external-url for URLs generated in the HTML output and another -web.route-prefix for prefixes to the actual internal routes?

@matthiasr @juliusv

[chancez]

@fabxc I think so. I'm mostly concerned that the frontend, always needs to have it's hyperlink's updated so the basepath is correct if you change the internal route prefix/basepath. I'm not 100% sure they need both need their own flags right now though. A flag which updates the basepaths for hyperlinks in the web frontend + the routes in the backend will probably be enough. I'll let you try and see what makes sense.

[fabxc]

@chancez But that's essentially what we have right now. We take the path of the external URL and use it for the resource links in the HTML. We also take the path of the external URL and prepend it to the routes internally.
The latter causes the conflict described in the initial comment by @neelance.

@juliusv suggested adding a boolean flag optinally switching the internal prefixing on and off.
My suggestion was to make it the explicit path rather than boolean. This would be more powerful and support all weird use cases of path modifying proxies. Users would have to potentially duplicate the path in -web.external-url and -web.route-prefix, which seems fine.

We should get this resolved in the next few days.

[matthiasr]

We could add `-web.route-prefix` but default to the path from `-web.external-url` – that way we have the full flexibility but keep the current behavior if the former is not specified.

[fabxc]

Good idea!
That would even allow us to add it post 1.0.0

On Sun, Jul 3, 2016, 6:26 PM Matthias Rampke notifications@github.com
wrote:

We could add -web.route-prefix but default to the path from
-web.external-url – that way we have the full flexibility but keep the
current behavior if the former is not specified.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1191 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AEuA8hr_g0b-xlsSkgWOTuE8nmJMSuybks5qR-KjgaJpZM4GYlve
.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.