Fix audio streaming output issues in 4.0

[aliabid94]

Streaming audio out broke in 4.0, as can be seen in the stream_audio_out demo. The waveform generator does not support a streaming audio source, so did a fallback onto a regular audio component. Test this via stream_audio_out demo.

There was another issue that whenever the DOM re-rendered for any reason, the waveform would have to regenerate, causing it to fetch the audio source on every DOM re-render. This is because normalize_files caused a new JSON to be passed to the AudioPlayer, so any DOM change caused a re-render. Now only re-renders if the URL changes. Test this via demo below while watching the Network Chrome inspector:

import gradio as gr


def greet(audio):
    for i in range(10):
        import time
        time.sleep(1)
        yield i


with gr.Blocks() as demo:
    name = gr.Audio(label="Name")
    output = gr.Textbox(label="Output Box")
    greet_btn = gr.Button("Greet")
    greet_btn.click(fn=greet, inputs=name, outputs=output, api_name="greet")

if __name__ == "__main__":
    demo.launch()

[gradio-pr-bot]

🪼 branch checks and previews

•	Name	Status	URL
	Spaces	ready!	Spaces preview
	Website	ready!	Website preview
	Storybook	ready!	Storybook preview
	Visual tests	all good!	Build review
🦄	Changes	detected!	Details

---

Install Gradio from this PR

pip install https://gradio-builds.s3.amazonaws.com/5be7b1fd77670086e48a546d3259cfafe665c3a6/gradio-4.1.2-py3-none-any.whl

Install Gradio Python Client from this PR

pip install "gradio-client @ git+https://github.com/gradio-app/gradio@5be7b1fd77670086e48a546d3259cfafe665c3a6#subdirectory=client/python"

[gradio-pr-bot]

🦄 change detected

This Pull Request includes changes to the following packages.

Package	Version
@gradio/audio	minor
gradio	minor

• Maintainers can select this checkbox to manually select packages to update.

With the following changelog entry.

Fix audio streaming output issues in 4.0

Maintainers or the PR author can modify the PR title to modify this entry.

Something isn't right?

• Maintainers can change the version label to modify the version bump.
• If the bot has failed to detect any changes, or if this pull request needs to update multiple packages to different versions or requires a more comprehensive changelog entry, maintainers can update the changelog file directly.

[abidlabs]

Will take a look! But could we add some tests to prevent regressions in the future? I think an e2e playwright test is warranted here

[abidlabs]

There was another issue that whenever the DOM re-rendered for any reason, the waveform would have to regenerate, causing it to fetch the audio source on every DOM re-render. This is because normalize_files caused a new JSON to be passed to the AudioPlayer, so any DOM change caused a re-render. Now only re-renders if the URL changes

Ah nice, I had noticed this issue in the Spaces deploy before

[abidlabs]

Test this via stream_audio_out demo

The output streaming isn't quite working for me. I get the behavior that some other users have described, which is that the audio file doesn't play until the full audio has been streamed. I've tried a few different things:

• Trying audio files of different lengths
• Hitting the play button manually vs. waiting for it to autoplay
• Decreasing the chunk size in the demo

This is the 60 second audio file I'm trying: https://www2.cs.uic.edu/~i101/SoundFiles/CantinaBand60.wav

[abidlabs]

Synced with @aliabid94, I was testing wrong. LGTM once we can fix the static checks and potentially add tests

[abidlabs]

Don't love the fallback UI, but we should get this in so that streaming works again. Perhaps @hannahblair you could take a look at the UI in a separate PR?

[aliabid94]

Adding playwright tests to this is going to be quite tricky because streaming is handled by the browser, not any javascript, so I'm not sure what we can specifically test is being done correctly in streaming. As far as the JS is aware, the HTML is set to a single source the same way a non-streaming audio file is.

[aliabid94]

@hannahblair if you have ideas to make the streaming audio player prettier than just using the default player, might be something you can take a look at!

[abidlabs]

Sounds good, let's merge this in for now

[hannahblair]

Thanks so much for handling this @aliabid94!

Let me have a think about how I can address the fallback UI for this and get back to you.