Apply color labels to regions of Audio component

[stevenhillis]

• I have searched to see if a similar issue already exists.

Is your feature request related to a problem? Please describe.
I am working with audio data. I don't have an intuitive way to label regions of that audio in a way that ties directly back to the audio playback.

Describe the solution you'd like
I would like to be able to specify a list of tuples like [(start_time_seconds, end_time_seconds, region_label)] for an Audio component. I would like to be able to specify this at instantiation, and I would also like to be able to interact with it in a block through callbacks to update my regions. I don't mind if colors are assigned automatically, but it would also be nice to specify a mapping between a region_label and a color. I think seaborn has this sort of behavior pretty well figured out. I would like a key displayed somewhere on the component, and I might like the region_label to appear on_hover over a region. When handling input, I'd like a text box to appear above the selected region for users to enter the region_label.

Additional context

• Potential use cases:
  • Input:
    • Allow users to specify smaller regions of long audios to learn from or process
  • Output:
    • Speaker Diarization
    • Language Identification
    • VAD

Here's a dumb photoshop of what I'm wanting:

[abidlabs]

Hi @stevenhillis thanks for the issue! This totally makes sense, although I'm not sure if we should adapt the existing Audio component or create a new component. Will leave as an open issue for now and update once we've discussed some more internally

[stevenhillis]

Hi @abidlabs, bumping this. Which way are the internal winds blowing?

[abidlabs]

Hi @stevenhillis recently we've had a lot of requests for new components or new variations of components like this one. We’re planning on making a framework to make it easy to create custom Gradio components — this is on our roadmap but is still 1-2 months out. So I think we’ll hold off on implementing this component ourselves but will work in making it easy for an interested user to create such a component themselves

cc @pngwn

[pngwn]

The Audio component does need overhauling as I don't think it is really good enough right now but as @abidlabs said that won't be happening immediately. When we do overhaul we will need to consider what kinds of features it should support. Gradio components should be solving the 80% case, while the remaining 'niche' cases can be solved in userland with the upcoming customisation options/APIs.

I do not know where this features sits in the that context but happy to have that conversation. There is a tracking issue here with relatively little detail but I'd love to get your thoughts on what the Audio component should and shouldn't support and how we can improve the usability (both for developers and end users).

[abidlabs]

Hi! We've now made it possible for Gradio users to create their own custom components -- meaning that you can write some Python and JavaScript (Svelte), and publish it as a Gradio component. You can use it in your own Gradio apps, or share it so that anyone can use it in their Gradio apps. Here are some examples of custom Gradio components:

• A "Rich Textbox" that allows you to write bold/italics/colored text
• A "Folium Map Viewer" component that allows you to use interactive maps

You can see the source code for those components by clicking the "Files" icon and then clicking "src". The complete source code for the backend and frontend is visible. In particular, its very fast if you want to build off an existing component. We've put together a Guide: https://www.gradio.app/guides/five-minute-guide, and we're happy to help if you'd like to put together a custom component for this issue

[abidlabs]

I'll go ahead and close this issue since we are not planning to include this in the core Gradio library. But happy to help if you are interested in making this a custom Gradio component (feel free to ask questions in this issue).