Support multiple workers / instances behind load balancer / reverse proxy (for example Gunicorn or similar)

[aersam]

This is a proposal after having discussed an issue in #1193

Currently nicegui cannot be used behind gunicorn or another reverse proxy without affinity support. This may lead to confusion and also I think it's straightforward to remove that limitation.

Reverse Proxies are quite common with FastAPI/Python since otherwise the Python GIL limits you quite a bit and a single long running request can block others (if not using async/await, sometimes you can't)

I think it would be possible for nicegui to implement support for running behind a reverse proxy without too much effort, but maybe I'm wrong. This is because once the websocket connection is established, only a single worker is involved anyway.
One would need persist the client(id/data) to disk on initial page render and then read it from there if needed. After the websocket handshake, everything works with the websocket which implies staying on the same worker. So only one hand-over between workers/processes is required.

Do you think that's something to consider? Or do I simply things too much?

[rodja]

Thanks for bringing this up @aersam. We would love to remove this limitation. But it might not be so simple.

One would need persist the client(id/data) to disk on initial page render and then read it from there if needed.

The data part may become tricky. Let us consider a simple scenario with a load balancer / reverse proxy in the front and two workers / instances named A and B in the back. The correct page-builder function (decorated with @ui.page) is executed when requesting the initial page. For example on worker A. While execution, arbitrary state can be created on the worker. For example opening an usb camara device or creating a database connection. After loading the html response the client establishes a websocket connection but may be routed to worker B by the load balancer. I do not know any way to transfer the state from A to B.

Last week @falkoschindler voiced a possible alternative approach: what if the page builder does not execute the decorated function directly but only serves a super tiny, generic html page which establishes a websocket connection. Then the the page builder is executed after the websocket connection is established. We have not found the time to investigate this approach. Anybody interested in giving it a shot?

[aersam]

I am unsure how much time I find for it, but will do this sooner or later as we will need it

[rodja]

If we're running the page builder on incoming websocket instead of incoming http request, we need to abandon the ability to return Starlette/FastAPI responses. And we won't have access to the request parameters any longer.

Another problem with multiple workers is the loss of global state. For example in a simple chat app, only the users on the same worker would see each other.

[aersam]

I think people are aware of the loss of global state if working with multiple workers, that's the same thing in every other framework, too.

Access to request parameter and not being able to return Responses is a big thing to me, however. It's the only meaningful place to do authentication, so I guess we need a different solution.

What if one could add a parameter sideeffect=False to ui.page which would allow to transfer the state between worker on init? Since the Id is strictly increment only, I guess it should work to just reexecute it on another worker. The first worker would somehow need to be able to remove it's state after some timeout since he won't receive the connection

Or allow something more involved which allows splitting the logic in two handlers:

@ui.page('/documentation')
def documentation_page(page) -> None:
    @page.pre_render(req: Request) ## no access to events/client. used for SEO/access to parameters etc
    def pre_render():
        ui.html("Loading...") # html would be removed on render
        if not req.bla
            return Response() # if a response object is returned, no connection wanted
        return None # maybe allow a json-dumpable dict that is passed to render?, even on other worker

    @page.render()
    def render():
        add_head_html()
        menu = side_menu()
        add_header(menu)
        ui.add_head_html('<style>html {scroll-behavior: auto;}</style>')
        with ui.column().classes('w-full p-8 lg:p-16 max-w-[1250px] mx-auto'):
            section_heading('Reference, Demos and more', '*NiceGUI* Documentation')
            documentation.create_full()

But looks too complicated... Would need to be optional to use

[rodja]

If we could auto-discover and reach all other workers we may be able to route the websocket stream to the correct worker.

Or we simply write a tutorial on how to use a load balancer like Traefik which supports session affinity instead of plain gunicorn.

[aersam]

That being said I do not have a good solution yet and I don't think there is an easy solution

[rodja]

I'm not too confident you never run into a blocking problem in a larger production app.

Sure, it's not easy. But I think it's still the best option for running a larger production app. In our robotics framework RoSys which uses NiceGUI as a basis we can activate asyncio warnings for tasks which block to long: https://github.com/zauberzeug/rosys/blob/main/rosys/analysis/asyncio_warnings.py

Also because many libraries just don't support async/await, just imagine some people doing some unwise pandas stuff.

Pandas operations are mostly not I/O bound but cpu bound and need to be executed in a separate process (see run.cpu_bound in RoSys).

For scenarios like these, gunicorn is a big help

I disagree. Gunicorn covers up the inherent problem of having blocked the event loop. It will still affect everything running on the specific worker. Hence a portion of your users will still experience the lag.

[v479038280]

If we can learn from streamlit about how to handle such delay?

[aersam]

@rodja: Well you're right in that's the proper way of doing things. I just have a quite a special group of people doing python that's not mainly devs but more coming from a data background without too much knowledge on resource management. Gunicorn is a bit of a duplicate assurance to me that no stupid thing like a memory leak or so blows up everything. If I have 4 workers and one is blown up, gunicorn will restart it and it will save me lot's of tickets, but not all. Of course it's not a solution, it's a workaround.

However I must admit that I did not see how complex this will be and therefore it's maybe just simpler to use a real load balancer with affinity support. What you should be doing however is to have a page to direct users to in case they experience the handshake error, to have some guidance there

[rodja]

Thanks for explaining @aersam. I totally understand the issue but am not sure how to fix it. I've created a new entry in our FAQs: https://github.com/zauberzeug/nicegui/wiki/FAQs#how-to-avoid-the-reloading-because-handshake-failed-javascript-error-message-which-results-in-a-full-page-reload

[Stefan2016]

Thanks for explaining @aersam. I totally understand the issue but am not sure how to fix it. I've created a new entry in our FAQs: https://github.com/zauberzeug/nicegui/wiki/FAQs#how-to-avoid-the-reloading-because-handshake-failed-javascript-error-message-which-results-in-a-full-page-reload

Do you have experience setting up session affinity with ngnix and nicegui? Would be interesting to see an example.

[rodja]

This feature request is more about supporting multiple workers for proxies which do not support session affinity. But nginx does. See #3038 for a discussion about such a setup.