Lite: Make the Examples component display media files using pseudo HTTP requests to the Wasm server #5627

whitphx · 2023-09-20T11:35:53Z

Description

Alternative implementation of the media components in the Wasm mode, based on the discussions in #4849.

🎯 PRs Should Target Issues

Before your create a PR, please check to see if there is an existing issue for this change. If not, please create an issue before you create this PR, unless the fix is very small.

Not adhering to this guideline will result in the PR being closed.

Tests

PRs will only be merged if tests pass on CI. To run the tests locally, please set up your Gradio environment locally and run the tests: bash scripts/run_all_tests.sh
You may need to run the linters: bash scripts/format_backend.sh and bash scripts/format_frontend.sh

gradio-pr-bot · 2023-09-20T11:36:08Z

🪼 branch checks and previews

•	Name	Status	URL
	Spaces	ready!	Spaces preview
	Website	ready!	Website preview
	Storybook	ready!	Storybook preview
	Visual tests	all good!	Build review
🦄	Changes	detected!	Details

Install Gradio from this PR

pip install https://gradio-builds.s3.amazonaws.com/8b08528552cb42c837bb694b2fbd88e717c468fc/gradio-3.47.1-py3-none-any.whl

Install Gradio Python Client from this PR

pip install "gradio-client @ git+https://github.com/gradio-app/gradio@8b08528552cb42c837bb694b2fbd88e717c468fc#subdirectory=client/python"

gradio-pr-bot · 2023-09-20T11:36:53Z

🦄 change detected

This Pull Request includes changes to the following packages.

Package	Version
`@gradio/app`	`minor`
`@gradio/audio`	`minor`
`@gradio/image`	`minor`
`@gradio/video`	`minor`
`@gradio/wasm`	`minor`
`gradio`	`minor`

Maintainers can select this checkbox to manually select packages to update.

With the following changelog entry.

Lite: Make the Examples component display media files using pseudo HTTP requests to the Wasm server

Maintainers or the PR author can modify the PR title to modify this entry.

Something isn't right?

Maintainers can change the version label to modify the version bump.
If the bot has failed to detect any changes, or if this pull request needs to update multiple packages to different versions or requires a more comprehensive changelog entry, maintainers can update the changelog file directly.

whitphx · 2023-09-20T11:39:36Z

js/image/example/Image.svelte

-	class:table={type === "table"}
-	class:gallery={type === "gallery"}
-	class:selected
+	class={[


The class: directive can't be used to components 😵‍💫

Any advice? Sorry I'm not so familiar with Svelte convention.

Also these styles defined at the parent component don't work...

Yes, svelte scopes styles to ensure component encapsulation.

The best approach here would be not to pass the classes down but to reimplement the styles to apply to a parent span or div, then the image can just have a 100% width/ height and the container can take care of the styling. I don't think we are doing anything too fancy with the example media so this should work well and will avoid the need to pass down the classes.

If we do need to apply style to the image from this component we can use a descendant global selector which will allow us to target the img but not leak out styles:

css.img-wrap :global(> img) { ... }

Thanks! Will try to modify them.

pngwn

Left a few comments, let me know if you have any questions!

pngwn · 2023-10-03T09:06:58Z

js/image/example/Image.svelte

-	class:table={type === "table"}
-	class:gallery={type === "gallery"}
-	class:selected
+	class={[


Yes, svelte scopes styles to ensure component encapsulation.

The best approach here would be not to pass the classes down but to reimplement the styles to apply to a parent span or div, then the image can just have a 100% width/ height and the container can take care of the styling. I don't think we are doing anything too fancy with the example media so this should work well and will avoid the need to pass down the classes.

If we do need to apply style to the image from this component we can use a descendant global selector which will allow us to target the img but not leak out styles:

css.img-wrap :global(> img) { ... }

pngwn · 2023-10-03T09:08:51Z

js/wasm/svelte/Image.svelte

Lets move this file to @gradio/image so it is available with our other image components, we can add an extra named export to the static subpath. The wasm stuff is an implementation detail that users don't need to be aware of.

OK, I would follow your choice,
while based on the same understanding below I made a different choice here at this point; more or less, this component is an interface encapsulating the Wasm details from users as you know and both make sense to put this in @gradio/image and @gradio/wasm, while my preference was to put Wasm-related implementations in @gradio/wasm. If you prefer to omit the wasm name from all import statements in these modules, I understand it's also an option.

The wasm stuff is an implementation detail that users don't need to be aware of.

When we create components for other media we should do the same, put them alongside the components they relate to. The main reason for this is that when custom components launch these will be user facing, so it would be nice if they don't need to think about wasm/ not wasm when building.

Really makes sense. Thanks

Does the shared subpath make more sense?
For example, in the case of @gradio/video, VideoPreview in the static subpath imports Player in shared (static->shared dependecy direction).

js/wasm/svelte/context.ts

js/wasm/svelte/file-url.ts

js/wasm/svelte/index.ts

whitphx · 2023-10-05T09:07:50Z

@pngwn I just pushed the changes related to the previous review. Can you check it when you have time? Thanks!
(I will be still working on Audio/Video components as well)

… in @gradio/image/example/Image.svelte

…L is not HTTP or refering to remote

whitphx · 2023-10-12T07:30:09Z

Currently the local dev app (js/app/src/lite/dev/App.svelte) can't add/edit media files.
So, to test the media files capability of this PR, copy code like below to js/app/src/lite/dev/App.svelte so it downloads and mount the media files programatically.

	// Source from https://huggingface.co/spaces/gradio/blocks_inputs/tree/main
	fetch("https://huggingface.co/spaces/gradio/blocks_inputs/resolve/main/lion.jpg")
		.then((response) => response.blob())
		.then((blob) => blob.arrayBuffer())
		.then((arrayBuffer) => {
			controller.write("lion.jpg", new Uint8Array(arrayBuffer), {});
			controller.write(
				"app.py",
				`
import gradio as gr
import os

def combine(a, b):
		return a + " " + b

def mirror(x):
		return x

with gr.Blocks() as demo:

		txt = gr.Textbox(label="Input", lines=2)
		txt_2 = gr.Textbox(label="Input 2")
		txt_3 = gr.Textbox(value="", label="Output")
		btn = gr.Button(value="Submit")
		btn.click(combine, inputs=[txt, txt_2], outputs=[txt_3])

		with gr.Row():
				im = gr.Image()
				im_2 = gr.Image()

		btn = gr.Button(value="Mirror Image")
		btn.click(mirror, inputs=[im], outputs=[im_2])

		gr.Markdown("## Text Examples")
		gr.Examples(
				[["hi", "Adam"], ["hello", "Eve"]],
				[txt, txt_2],
				txt_3,
				combine,
				#cache_examples=True,
		)
		gr.Markdown("## Image Examples")
		gr.Examples(
				examples=[os.path.join(os.path.dirname(__file__), "lion.jpg")],
				inputs=im,
				outputs=im_2,
				fn=mirror,
				#cache_examples=True,
		)

if __name__ == "__main__":
		demo.launch()
`,
				{}
			);
			controller.run_file("app.py");
		});

pngwn

Other than my comment, everything looks great! Thanks @whitphx!

pngwn · 2023-10-16T08:30:42Z

js/audio/interactive/Audio.svelte

This is the only media component where the interactive version has been changed rather than the example. Is there a reason for that?

It's just because of the codebase structure.
In the case of Audio, example/Audio.svelte doesn't have the <audio /> tag as it only shows the example file name, and modifying interactive/Audio.svelte was needed to make audio apps work instead.
In turn, for example in the case of Video, all Video components uses shared/Player.svelte under the hood, so I just modified it.

FYI, I benchmarked the following 3 spaces to make sure the pair of a media input component and gr.Examples() works.

https://huggingface.co/spaces/gradio/blocks_inputs (for Image)

https://huggingface.co/spaces/gradio/video_component (for Video)

https://huggingface.co/spaces/gradio/main_note (for Audio)

So there might be missing parts. Will catch them in another PR if found.

whitphx · 2023-10-16T15:23:15Z

@pngwn Thank you!!!

whitphx mentioned this pull request Sep 20, 2023

Lite: Make the Examples component display media files correctly #4849

Closed

whitphx commented Sep 20, 2023

View reviewed changes

pngwn reviewed Oct 3, 2023

View reviewed changes

whitphx force-pushed the wasm-example-media-files branch from 9048daf to b26e835 Compare October 5, 2023 07:22

whitphx mentioned this pull request Oct 6, 2023

Lite: Fix Examples.create() to be a normal func so it can be called in the Wasm env #5821

Merged

whitphx and others added 5 commits October 9, 2023 17:53

Create a Wasm-compatible <Image> component in @gradio/lite and use it…

3c349f0

… in @gradio/image/example/Image.svelte

add changeset

8f5de8a

Fix Image styling

60655b1

Move js/wasm/svelte/Image.svelte -> js/image/static/Image.svelte

506c490

Shorten CSS class name

7013f32

whitphx force-pushed the wasm-example-media-files branch 2 times, most recently from 686a749 to fe178a7 Compare October 9, 2023 09:05

whitphx and others added 2 commits October 9, 2023 19:11

Create Wasm-compatible video components

e37c8e8

add changeset

dfe571a

whitphx force-pushed the wasm-example-media-files branch from fe178a7 to db1967f Compare October 9, 2023 10:12

Move the Wasm-compatible Image from static to shared

8b650d8

whitphx force-pushed the wasm-example-media-files branch from 2a04628 to 12aba77 Compare October 9, 2023 10:15

whitphx requested a review from pngwn October 9, 2023 10:18

whitphx marked this pull request as ready for review October 9, 2023 10:18

whitphx added 2 commits October 9, 2023 19:49

Create Wasm-compatible audio components

d7a99e0

Fix resolve_wasm_src() to return the passed src as-is when the UR…

f53a85e

…L is not HTTP or refering to remote

whitphx force-pushed the wasm-example-media-files branch from 5b46c91 to f53a85e Compare October 9, 2023 10:49

gradio-pr-bot and others added 2 commits October 9, 2023 10:51

add changeset

1a8c688

Fix resolve_wasm_src()

ee1dd20

Merge branch 'main' into wasm-example-media-files

25e43c7

pngwn approved these changes Oct 16, 2023

View reviewed changes

Merge branch 'main' into wasm-example-media-files

0aaa445

whitphx merged commit b67115e into main Oct 16, 2023
19 of 20 checks passed

whitphx deleted the wasm-example-media-files branch October 16, 2023 16:26

This was referenced Oct 16, 2023

chore: update versions #5831

Merged

chore: update versions (beta) #5769

Merged

aliabd mentioned this pull request Oct 16, 2023

Fix styling issues with Audio, Image and Video components #5934

Merged

whitphx mentioned this pull request Oct 19, 2023

Update static/AudioPlayerto use ../shared/Audio instead of the native <audio> for Wasm support #6004

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lite: Make the Examples component display media files using pseudo HTTP requests to the Wasm server #5627

Lite: Make the Examples component display media files using pseudo HTTP requests to the Wasm server #5627

whitphx commented Sep 20, 2023 •

edited

gradio-pr-bot commented Sep 20, 2023 •

edited

gradio-pr-bot commented Sep 20, 2023 •

edited

Something isn't right?

whitphx Sep 20, 2023

whitphx Sep 21, 2023

pngwn Oct 3, 2023

whitphx Oct 3, 2023 •

edited

pngwn left a comment

pngwn Oct 3, 2023

pngwn Oct 3, 2023

whitphx Oct 3, 2023 •

edited

pngwn Oct 3, 2023

whitphx Oct 3, 2023

whitphx Oct 9, 2023 •

edited

whitphx commented Oct 5, 2023

whitphx commented Oct 12, 2023

pngwn left a comment

pngwn Oct 16, 2023

whitphx Oct 16, 2023 •

edited

whitphx commented Oct 16, 2023

Lite: Make the Examples component display media files using pseudo HTTP requests to the Wasm server #5627

Lite: Make the Examples component display media files using pseudo HTTP requests to the Wasm server #5627

Conversation

whitphx commented Sep 20, 2023 • edited

Description

🎯 PRs Should Target Issues

Tests

gradio-pr-bot commented Sep 20, 2023 • edited

🪼 branch checks and previews

gradio-pr-bot commented Sep 20, 2023 • edited

🦄 change detected

This Pull Request includes changes to the following packages.

With the following changelog entry.

Something isn't right?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

whitphx Oct 3, 2023 • edited

Choose a reason for hiding this comment

pngwn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

whitphx Oct 3, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

whitphx Oct 9, 2023 • edited

Choose a reason for hiding this comment

whitphx commented Oct 5, 2023

whitphx commented Oct 12, 2023

pngwn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

whitphx Oct 16, 2023 • edited

Choose a reason for hiding this comment

whitphx commented Oct 16, 2023

whitphx commented Sep 20, 2023 •

edited

gradio-pr-bot commented Sep 20, 2023 •

edited

gradio-pr-bot commented Sep 20, 2023 •

edited

whitphx Oct 3, 2023 •

edited

whitphx Oct 3, 2023 •

edited

whitphx Oct 9, 2023 •

edited

whitphx Oct 16, 2023 •

edited