Add support for async functions and async generators to gr.ChatInterface

[abidlabs]

Added async support for all of the routes: the submit function, the example function, and the API functions. Test with any async function or generator, e.g.:

async def slow_echo(message, history):
    for i in range(len(message)):
        time.sleep(0.05)
        yield "You typed: " + message[: i+1]

demo = gr.ChatInterface(slow_echo, examples=["abcdef"], cache_examples=True).queue()

demo.launch()

Closes: #5115

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
gradio	✅ Ready (Inspect)	Visit Preview	Aug 8, 2023 7:44pm

[gradio-pr-bot]

🦄 change detected

This Pull Request includes changes to the following packages.

Package	Version
gradio	minor

• Maintainers can select this checkbox to manually select packages to update.

With the following changelog entry.

Add support for async functions and async generators to gr.ChatInterface

Maintainers or the PR author can modify the PR title to modify this entry.

Something isn't right?

• Maintainers can change the version label to modify the version bump.
• If the bot has failed to detect any changes, or if this pull request needs to update multiple packages to different versions or requires a more comprehensive changelog entry, maintainers can update the changelog file directly.

[gradio-pr-bot]

All the demos for this PR have been deployed at https://huggingface.co/spaces/gradio-pr-deploys/pr-5116-all-demos

---

You can install the changes in this PR by running:

pip install https://gradio-builds.s3.amazonaws.com/e2fb94d7f53fdfc3050a910de93c5a3b5a080530/gradio-3.39.0-py3-none-any.whl

[freddyaboulton]

Since we're switching everything to coroutines, any possible blocking code that's not async needs to be run in a separate thread otherwise we'll block the event loop. So for example use anyio.to_thread.run_sync to run the non-async code.

Might be better to utils.SyncToAsyncIterator for the generators to match what we do in blocks.

Otherwise this looks good though!

[abidlabs]

Ah nice I was going to ask why we do utils.SyncToAsyncIterator in blocks.py

[abidlabs]

Ok I've updated to use SyncToAsyncIterator @freddyaboulton. Let me know if you see anything else or if this is good to merge.

[freddyaboulton]

We need to run the non-coroutine in a separate thread otherwise we'll block the event loop. This is because the _submit_fn is now async.

If you run this code on your branch, two events can't run concurrently anymore

import random
import gradio as gr
import time

def random_response(message, history):
    time.sleep(5)
    return random.choice(["Yes", "No"])

demo = gr.ChatInterface(random_response)

if __name__ == "__main__":
    demo.launch()

Note how the textbox submit event has to wait:

compare to main:

[abidlabs]

Ah nice thanks @freddyaboulton, will fix

[gradio-pr-bot]

🎉 Chromatic build completed!

There are 0 visual changes to review.
There are 0 failed tests to fix.

• Storybook Preview
• Build Review

[abidlabs]

Now should be fixed. Tested with regular functions as well as with iterators (the latter requires enabling queuing and setting concurrency count > 1), and non-async functions no longer block.

[freddyaboulton]

Thanks for the enhancement @abidlabs !!

[abidlabs]

Appreciate the guidance around non-blocking thanks @freddyaboulton!