Lite: Websocket queueing

[whitphx]

Description

• The WebSocket emulation implementation on WebWorker is based on shinylive.
• The queueing module in Python uses in-server HTTP communications but it doesn't work on the Gradio-lite's architecture, so this PR also includes the changes on it (67c432d is it for now, while it might be replaced with better solution).
  • Probably we should extract only these changes on the Python part to another PR.
  • Fix the Queue to call API endpoints without internal HTTP routing #5165 is for it.

🎯 PRs Should Target Issues

Before your create a PR, please check to see if there is an existing issue for this change. If not, please create an issue before you create this PR, unless the fix is very small.

Not adhering to this guideline will result in the PR being closed.

Tests

1. PRs will only be merged if tests pass on CI. To run the tests locally, please set up your Gradio environment locally and run the tests: bash scripts/run_all_tests.sh

2. You may need to run the linters: bash scripts/format_backend.sh and bash scripts/format_frontend.sh

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
gradio	✅ Ready (Inspect)	Visit Preview	Aug 9, 2023 9:34am

[gradio-pr-bot]

🦄 change detected

This Pull Request includes changes to the following packages.

Package	Version
@gradio/app	minor
@gradio/client	minor
@gradio/wasm	minor
gradio	minor

• Maintainers can select this checkbox to manually select packages to update.

With the following changelog entry.

Lite: Websocket queueing

Maintainers or the PR author can modify the PR title to modify this entry.

Something isn't right?

• Maintainers can change the version label to modify the version bump.
• If the bot has failed to detect any changes, or if this pull request needs to update multiple packages to different versions or requires a more comprehensive changelog entry, maintainers can update the changelog file directly.

[gradio-pr-bot]

All the demos for this PR have been deployed at https://huggingface.co/spaces/gradio-pr-deploys/pr-5124-all-demos

---

You can install the changes in this PR by running:

pip install https://gradio-builds.s3.amazonaws.com/b8e1f4fd6311f56f1467a9a34d71e298e85d1ac6/gradio-3.39.0-py3-none-any.whl

[gradio-pr-bot]

🪼 branch checks and previews

•	Name	Status	URL
	Storybook	ready!	Storybook preview
	Visual tests	all good!	Build review
🦄	Changes	detected!	Details

[whitphx]

The queue works on Gradio-lite.

[aliabd]

@whitphx like we discussed in sync, I ran into a few issues:

• Demos with queue enabled are hanging forever, but there's no error on the console. weirdly, only demos with demo.load are working for me. Everything else hangs.

Here's two simple examples:

import gradio as gr

demo = gr.Interface(lambda x:x, "text", "text")

if __name__ == "__main__":
    demo.queue().launch()

import gradio as gr

def greet(inp):
	return "hello " + inp + "!"

with gr.Blocks() as demo:
	text = gr.Textbox()
	output = gr.Textbox()
	button = gr.Button("Submit")
	button.click(greet, inputs=text, outputs=output)


if __name__ == "__main__":
	demo.queue().launch()

• There's an issue with executing files on the App.svelte. If I make a change on the editor and click execute, I get this error:

Uncaught (in promise) PythonError: Traceback (most recent call last):
  File "<exec>", line 120, in _run_script
  File "app.py", line 12, in <module>
    dep = demo.load(greet_with_time, None, text, every=1)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.11/site-packages/gradio/blocks.py", line 1704, in load
    dep, dep_index = self.set_event_trigger(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.11/site-packages/gradio/blocks.py", line 326, in set_event_trigger
    "outputs": [block._id for block in outputs],
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.11/site-packages/gradio/blocks.py", line 326, in <listcomp>
    "outputs": [block._id for block in outputs],
                ^^^^^^^^^
AttributeError: 'Markdown' object has no attribute '_id'

    at new_error (pyodide.asm.js:9:12519)
    at pyodide.asm.wasm
    at pyodide.asm.wasm
    at Module.callPyObjectKwargs (pyodide.asm.js:9:81856)
    at Module.callPyObject (pyodide.asm.js:9:82066)
    at Function.apply (pyodide.asm.js:9:97147)
    at Object.apply (pyodide.asm.js:9:95381)
    at self.onmessage (webworker.js?type=classic&worker_file:333:795)

[whitphx]

@aliabd Thank you! Will work on these!

[whitphx]

Note:

About the first problem

Demos with queue enabled are hanging forever, but there's no error on the console. weirdly, only demos with demo.load are working for me. Everything else hangs.

This is because startup-events is not called now in the Wasm env:
https://github.com/gradio-app/gradio/blob/main/gradio/blocks.py#L2041-L2046

-> I will fix here to call startup_events().

About the second problem

There's an issue with executing files on the App.svelte. If I make a change on the editor and click execute, I get this error: AttributeError: 'Markdown' object has no attribute '_id'

id is set here: https://github.com/gradio-app/gradio/blob/main/gradio/blocks.py#L141
Looks like multi-thread, async, or some Wasm-specific nature affect this with a naive usage of a global Context class. Luckily, found #5625 solves this exact problem 🎉

-> I will merge the latest main which already has this fix.

[whitphx]

@aliabd The bugs should have been fixed 👍
But I have some things to ask for better code. Appreciate your thoughts, thanks!

[whitphx]

This code is indented to cancel the tasks created at the following places, which normally run in an event loop in the server thread and stopped by self.server.cancel(), however, in the case of Wasm, run in an event loop in the main thread.

• gradio/gradio/routes.py

  Lines 573 to 576 in ea0e00b

  	task = run_coro_in_background(
  	blocks._queue.process_events, [event], False
  	)
  	set_task_name(task, event.session_hash, event.fn_index, batch=False)

• gradio/gradio/queueing.py

  Lines 136 to 138 in ea0e00b

  	task = run_coro_in_background(self.process_events, events, batch)
  	run_coro_in_background(self.broadcast_live_estimations)
  	set_task_name(task, events[0].session_hash, events[0].fn_index, batch)

---

The current code here is ad-hoc. @aliabd do you have any ideas about where we should put this code snippet better or what code snippet we can write?
(I firstly thought I could write this code at queueing.Queue.close(), but it seemed the Queue is expected to be resume()-ed, so I finally didn't. Was it correct?)

[aliabd]

@whitphx Blocks.close() only makes sense in the context of a server, because it's used to free up ports/terminate the server.

Really, the only use case is in a notebook setting, because users can be running multiple demos sequentially in cells and could use it to reduce bloat once a demo is done. Or to run gr.close_all() which closes every port an Interface is running on. We do also use it in our tests. But there's no use case for it inside a demo or anything like that (we don't even document it anywhere). So it doesn't make sense in the wasm context, since there's no server or ports anyway. So I think if it's called inside wasm it shouldn't do anything (or potentially it could do something with app). You do call it in wasm_utils.py but I'm not sure what it's doing there? (left a comment)

Same with queue().close() which is only meant to be used in the context of threads.

[aliabd]

Let me know if this makes sense or if I'm missing something

[whitphx]

Thank you for the review!

My original intension was to "clean up" all the resources and running tasks bound to the running app and blocks when a user runs new app code including Blocks.launch() with run_code() or run_file().
The clean-up here should be something like what happens when a normal Gradio app is terminated, and I thought Blocks.close() was it.

Also, wasm_utils.register_app() is called with a new FastAPI app object every time a user calls Blocks.launch().

So I thought putting the clean-up code in Blocks.close() and calling it from wasm_utils.register_app() before registering the new app object would be a good solution.
If it doesn't make sense, it's ok if we can put this task cancellation code somewhere and call it on every Blocks.launch() call as written above. Appreciate it if you have some suggestions on it as well.

For the semantics of the server-context you wrote:
Even in the Wasm ver., there is a FastAPI app object, and if we see it as a part of the "server", things may make sense.

[whitphx]

For example, putting this task cancellation code to wasm_utils.register_app() directly also works, instead of modifying Blocks.close() and calling it from register_app(), like

def register_app(_app):
    global app

    # HERE!
    for task in asyncio.all_tasks():
        task_name = task.get_name()
        if re.match(
            r"\w+_\d+", task_name
        ):  # This regex matches the task names set by `utils.set_task_name()`
            task.cancel()

    app = _app

Blocks.close() looked like a general purpose cleanup method paired with Blocks.launch(), so I thought using it makes the code more robust for future changes rather than adding Wasm-specific cleanup code, but it's ok to decline it if it doesn't make sense from your aspect. Please give an advice :D

[whitphx]

@aliabd Thanks!

Can't we just cancel all the tasks?

There are some tasks which must not be cancelled, such as

gradio/js/wasm/src/webworker/index.ts

Line 109 in abb5e9d

async def _call_asgi_app_from_js(scope, receive, send):

Such tasks are mainly executed outside Gradio to manage the Gradio server across multiple run_file() or run_code() executions.
Also, I wanted to reduce risks of unexpected side effects in the future.

I'm fine with doing it your way

So, let me take this path, thanks!
I added a comment explaining this 👍

[whitphx]

@abidlabs @aliabd Also, I created a new commit modifying the async task management to cancel the tasks without depending on the task names: 5ee60a7
which seems to be more robust.
What do you think? If the previous one is better, I will revert this.

[aliabd]

@whitphx I really like this, much cleaner! and no regex :)

But this is causing an error on the console.

PyodideTask exception was never retrieved
pyodide.asm.js:9 future: <PyodideTask finished name='41bqrtb8ssv_0' coro=<Queue.process_events() done, defined at /lib/python3.11/site-packages/gradio/queueing.py:425> exception=ValueError('[<gradio.queueing.Event object at 0x78dcda8>] is not in list')>

[whitphx]

@aliabd Thanks!

I added a fix: 9c65c01

---

As I commented in the code, this is an existing bug.
The thrown error is not printed in the normal Python env (Why exceptions in asnycio are late or do not appear at all? - Stack Overflow) but it is in the Pyodide/Wasm env due to its unique event loop design.
Even in the normal env, we can confirm the same exception is raised, for example when reloading an app using gr.load(every=...) by adding a callback to the task created here to print the exception as below.

                     blocks._queue.process_events, [event], False
                 )
                 set_task_name(task, event.session_hash, event.fn_index, batch=False)
+                def task_done_callback(task):
+                    if task.exception():
+                        print(task.exception())
+                task.add_done_callback(task_done_callback)

python run.py
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
[<gradio.queueing.Event object at 0x12dafc2e0>] is not in list

This implies we should fix all the code using run_coro_in_background() to handle exceptions (as a separate ticket of course). The Stack Overflow answer says

You should never do fire-and-forget technique for created tasks.

-> Created an issue: #5784

[aliabd]

ah got it!

[whitphx]

@aliabid94 Is this comment correct about the purpose of calling the HTTP endpoint startup-events? (Ref: #1969)

[aliabd]

@whitphx Blocks.close() only makes sense in the context of a server, because it's used to free up ports/terminate the server.

Really, the only use case is in a notebook setting, because users can be running multiple demos sequentially in cells and could use it to reduce bloat once a demo is done. Or to run gr.close_all() which closes every port an Interface is running on. We do also use it in our tests. But there's no use case for it inside a demo or anything like that (we don't even document it anywhere). So it doesn't make sense in the wasm context, since there's no server or ports anyway. So I think if it's called inside wasm it shouldn't do anything (or potentially it could do something with app). You do call it in wasm_utils.py but I'm not sure what it's doing there? (left a comment)

Same with queue().close() which is only meant to be used in the context of threads.

[aliabd]

Let me know if this makes sense or if I'm missing something

[aliabd]

Both bugs are indeed gone! All the queue apps I'm trying are working great :) Will just do some more testing and then I'm fine with merging this. Nice work @whitphx!

[aliabd]

gradio-ui / functional-test is failing because of an issue we had that's fixed on main. You should just update the branch.

[whitphx]

While these are not direct answers to your comment, I tried to explain what the "server" mechanism of the Wasm ver. looks like:

• As you wrote, In the Wasm ver. there is no HTTP server such as uvicorn. However, the FastAPI app is still working and there is also an HTTP-server-like mechanism instead.
  • In normal Gradio, the FastAPI app communicates with the HTTP server, uvicorn, via ASGI protocol (and uvicorn communicates with HTTP clients.)
  • In the Wasm ver., similarly, the FastAPI app communicates with the JS code running in a WebWorker via ASGI protocol. Pyodide automatically converts Python objects into JS objects and vice versa, so such inter-language ASGI communication is possible by writing the ASGI server in JS (its code is here).
    • To do this, the FastAPI app object must be accessible from a global scope so that the JS code can access it via Pyodide's pyodide.runPython(). wasm_utils.register_app() and wasm_utils.get_registered_app() are the ad-hoc mechanism for it by registering the app object from Blocks.launch() and obtaining it from the JS scope.
• In other words, in the Wasm ver., we replaced uvicorn with our own HTTP server implementation which communicates directly with the JS code without actual HTTP requests because the FastAPI app runs on Pyodide in a WebWorker and the frontend app has to communicate with it via the worker-to-main-thread messaging API, not normal HTTP.

[aliabd]

Thanks @whitphx for the great work here!

I tested everything:

• queueing is working as expected in a wasm environment.
• queueing is working as expected in a non-wasm environment. (here's a space with demos reflecting last commit)

This is good to merge! Please just update the pr from main, and run the linter so the backend tests pass (sh scripts/format_backend.sh)

[whitphx]

Thank you!
All tests passed.
Some deploy jobs failed but it seems because this PR is from my personal repo.

Will merge it.

[whitphx]

(here's a space with demos reflecting last commit)

omg, Nice space for manual tests, btw!