Gradio static files do not follow the root_path behind a Jupyter proxy on k8s

[louisbrulenaudet]

Describe the bug

Hi everyone,

I am experiencing a problem when running Gradio 4.16.0 behind a Jupyter proxy on a k8s cluster specifying a root_path, static files do not follow this path and my browser console returns a 404 error.

Below is the code:

demo.launch(
    server_port=5300, 
    root_path="/proxy/absolute/5300/", 
    share=True, 
    inline=False
)

The link is still accessible and the components can be viewed without style.

Thank you in advance, I remain at your disposal for any clarification.

Best regards,
Louis

Have you searched existing issues? 🔎

• I have searched and found no existing issues

Reproduction

import gradio as gr

def greet(name):
    return "Hello " + name + "!"

demo = gr.Interface(
    fn=greet, 
    inputs="text", 
    outputs="text"
)

demo.launch(
    server_port=5300, 
    root_path="/proxy/absolute/5300/", 
    share=True, 
    inline=False
)

Screenshot

No response

Logs

No response

System Info

gradio==4.16.0 on Python 3.9

Severity

Blocking usage of gradio

[abidlabs]

@louisbrulenaudet could you please try installing this version of gradio and seeing if the issue is resolved?

pip install https://gradio-builds.s3.amazonaws.com/9b8810ff9af4d9a50032752af09cefcf2ef7a7ac/gradio-4.18.0-py3-none-any.whl

[pseudotensor]

@abidlabs Does it make sense that 3.50.2 had no k8 issue but 4.16+ does. But only 4.18.0 had the reverse proxy issue. I'm worried that the PR solved the recent regression from 4.17.0->4.18.0, but not this one.

[abidlabs]

To be honest I'm not totally sure. I have not tested on a k8 cluster. However, I did fix an issue related to root_path so I think this issue is fixed. Would be awesome if someone can test the above-mentioned on a k8 cluster and confirm

[louisbrulenaudet]

Hi @abidlabs, @pseudotensor,

First of all, I'd like to thank you for your help. I've just tried version 4.19.0 released today and the problem persists.

Looking through the element inspector, it turns out that the index.css and theme.css files don't have the right path. These are respectively :

• theme.css = //domain.com/proxy/absolute/5300/proxy/absolute/5300/theme.css
• style.css = /assets/index-8d4a258a.css

The code remains the same.

I remain at your disposal for further investigations and thank you in advance for your help.

Yours sincerely
Louis

[abidlabs]

Ok reopening this issue and looking into it!

[skye0402]

I can add to this: 4.16 (and 4.17 which was buggy for the accordions) works on my K8S. Since then it stopped working (4.18+).

The browser tries to load theme.css from the backend service url which it shouldn't (it's behind the approuter, so it can't access it). For the moment I stay on 4.16.

[skye0402]

@abidlabs - Any updates? Currently i need to stay on 4.16. The error came with 4.18.

[abidlabs]

Hi folks, looking into this issue. First of all, could someone test with the latest version of gradio:

pip install https://gradio-builds.s3.amazonaws.com/1a4b089e78d1efff47b4606daebde02e9b246ed4/gradio-4.20.1-py3-none-any.whl

and let me know if the issue persists? We've made some fixes to how the root path is resolved, although I suspect it might not be a complete fix just yet.

[skye0402]

@abidlabs - is it the same like 4.20.1 on PyPi? I used this just now as it's easier to deploy. Result is as shown in the picture.

You see it's the page without any styling and therefore unusable. So it looks like from 4.18 onwards still. So info and theme.css seem to be the bad guys here while the rest works as it should.

[abidlabs]

@skye0402 they're actually different versions. There's a fix to the root url on main, which is not in 4.20.1 just yet. That being said, from the console log, it doesn't look like it would have fixed the issue.

Can I ask, what is the correct url for your gradio app? I.e. what url is it being served on, if not https://apascalingtest-service:8020?

[skye0402]

@abidlabs thank you for working on this! I have used the link you provided and rebuilt the image just to be sure I don't miss anything. Indeed, it is unchanged, please see the screenshot. Url of the app is https://scalingtest.c-290ae5b.kyma.shoot.live.k8s-hana.ondemand.com - it runs behind an approuter, so a pod accepts the request from the above url and forwards it to the pod that runs the gradio app. That pod has that internal hostname that you saw from the screenshot (https://apascalingtest-service:8020/) - all of that works wonderfully until 4.16/4.17.
I can provide you or your team with access to the app or even the K8S cluster if that helps - let me know.

[abidlabs]

Thanks @skye0402 for the details! Can you tell us a bit more about this approuter? What library / framework are you using? Trying to understand what could cause this issue.

One workaround we could do is to let users manually specify the full root_url. I.e. you could pass in https://scalingtest.c-290ae5b.kyma.shoot.live.k8s-hana.ondemand.com/ as a parameter in your Gradio app, or set it via an environment variable, and this url would take priority over Gradio trying to determine the root url automatically.

The problem, of course, is that if your URL changes, you'd have to reset the parameter / env variable. Just wondering, would this be useful for your use case? Same question for you: @pseudotensor @louisbrulenaudet

[abidlabs]

One other request, please: could you try installing gradio from this PR: #7641 (you'll see the pip command to do that in the PR)

And then visit your Gradio app, and look at the logs that are printed to the terminal on the machine running the Gradio app? I've added a few lines that should print something like:

request.url ...
x_forwarded_host ...
x_forwarded_host_lower ...

and can you tell me what you see?

[skye0402]

@abidlabs certainly! The approuter is https://www.npmjs.com/package/@sap/approuter
Injecting the hosted url as parameter would be an option - of course it's not so nice as having it independent as it was before.

Here is the result using the gradio with logging:

/usr/local/lib/python3.11/site-packages/gradio_client/documentation.py:102: UserWarning: Could not get documentation group for <function set_static_paths at 0x7f1fb0228400>: No known documentation group for module 'gradio.utils'
warnings.warn(f"Could not get documentation group for {cls}: {exc}")
Running on local URL: http://apascalingtest-app-568c7bd85d-fv2bm:7860
request.url http://apascalingtest-app-568c7bd85d-fv2bm:7860/
x_forwarded_host None
x_forwarded_host_lower None
To create a public link, set `share=True` in `launch()`.
request.url http://apascalingtest-service:8020/
x_forwarded_host scalingtest.c-290ae5b.kyma.shoot.live.k8s-hana.ondemand.com
x_forwarded_host_lower scalingtest.c-290ae5b.kyma.shoot.live.k8s-hana.ondemand.com

I'm not an expert but I think this looks promising! 😃
P.S.: The first output might be before authentication is completed from a user perspective. From that time onwards the last 3 lines repeat with every new request:

/usr/local/lib/python3.11/site-packages/gradio_client/documentation.py:102: UserWarning: Could not get documentation group for <function set_static_paths at 0x7fbdc9828400>: No known documentation group for module 'gradio.utils'
warnings.warn(f"Could not get documentation group for {cls}: {exc}")
Running on local URL: http://apascalingtest-app-568c7bd85d-nm89v:7860
request.url http://apascalingtest-app-568c7bd85d-nm89v:7860/
x_forwarded_host None
x_forwarded_host_lower None
To create a public link, set `share=True` in `launch()`.
request.url http://apascalingtest-service:8020/
x_forwarded_host scalingtest.c-290ae5b.kyma.shoot.live.k8s-hana.ondemand.com
x_forwarded_host_lower scalingtest.c-290ae5b.kyma.shoot.live.k8s-hana.ondemand.com
request.url http://apascalingtest-service:8020/
x_forwarded_host scalingtest.c-290ae5b.kyma.shoot.live.k8s-hana.ondemand.com
x_forwarded_host_lower scalingtest.c-290ae5b.kyma.shoot.live.k8s-hana.ondemand.com
request.url http://apascalingtest-service:8020/
x_forwarded_host scalingtest.c-290ae5b.kyma.shoot.live.k8s-hana.ondemand.com
x_forwarded_host_lower scalingtest.c-290ae5b.kyma.shoot.live.k8s-hana.ondemand.com
request.url http://apascalingtest-service:8020/
x_forwarded_host scalingtest.c-290ae5b.kyma.shoot.live.k8s-hana.ondemand.com
x_forwarded_host_lower scalingtest.c-290ae5b.kyma.shoot.live.k8s-hana.ondemand.com

... meaning every 3 lines are output when refreshing the page from the browser.

[abidlabs]

Thanks @skye0402 this is very helpful, and we should be able to issue a fix using the x_forwarded_host header. Will put something up soon

[abidlabs]

@skye0402 (or @pseudotensor @louisbrulenaudet) would you be able to try the PR again and let me know if Gradio apps are working for you now?

You can install:

pip install https://gradio-builds.s3.amazonaws.com/3f6ced361dfac11fe1545fb16ae643f1e93e0af1/gradio-4.20.1-py3-none-any.whl

[skye0402]

@abidlabs Magnificent! That's a job well done (and I can upgrade to latest Gradio soon). Below is the same app as screenshot like before (but working this time). Thank you very much!

[abidlabs]

Thanks for all your help!