Allow start/pause of streaming image input. Only access the webcam while it's needed #7228

freddyaboulton · 2024-01-31T00:54:56Z

Description

Closes: #3642
Closes: #7230

If the webcam is an allowed image source, gradio will access the webcam as soon as the component is mounted on the page. There is also no way to stop gradio from accessing the webcam. This is #3642

When the webcam is an allowed source, users have to click into the image component to grant webcam access. This is what it looks like in that case:

After the webcam stops recording or taking a picture, gradio will stop using the webcam and revert back to the ^previous state.

Additionally, when gr.Image(streaming=true), the webcam component will behave similarly to video recording. There will be a record and stop button at the bottom that users can use to stop the stream of images.

Webcam Demo no streaming

Webcam Demo with streaming

Test streaming with this demo

import gradio as gr
import numpy as np

def flip(im):
    return np.flipud(im)

with gr.Blocks() as demo:
    with gr.Row():
        input_img = gr.Image(sources=["webcam"])
        output_img = gr.Image()
        input_img.stream(flip, input_img, output_img)
demo.launch(share=True)

if __name__ == "__main__":
    demo.launch()

EDIT:

Also fixes Interface streaming blocking the output, fixes: #7021

🎯 PRs Should Target Issues

Before your create a PR, please check to see if there is an existing issue for this change. If not, please create an issue before you create this PR, unless the fix is very small.

Not adhering to this guideline will result in the PR being closed.

Tests

PRs will only be merged if tests pass on CI. To run the tests locally, please set up your Gradio environment locally and run the tests: bash scripts/run_all_tests.sh
You may need to run the linters: bash scripts/format_backend.sh and bash scripts/format_frontend.sh

gradio-pr-bot · 2024-01-31T00:56:56Z

🪼 branch checks and previews

•	Name	Status	URL
	Spaces	building...
	Website	ready!	Website preview
	Storybook	ready!	Storybook preview
	Visual tests	building...
🦄	Changes	detected!	Details

gradio-pr-bot · 2024-01-31T00:57:20Z

🦄 change detected

This Pull Request includes changes to the following packages.

Package	Version
`@gradio/image`	`minor`
`gradio`	`minor`

Maintainers can select this checkbox to manually select packages to update.

With the following changelog entry.

Allow start/pause of streaming image input. Only access the webcam while it's needed

Maintainers or the PR author can modify the PR title to modify this entry.

Something isn't right?

Maintainers can change the version label to modify the version bump.
If the bot has failed to detect any changes, or if this pull request needs to update multiple packages to different versions or requires a more comprehensive changelog entry, maintainers can update the changelog file directly.

freddyaboulton · 2024-01-31T00:58:00Z

js/image/shared/Webcam.svelte

@@ -180,19 +203,34 @@
 		event.stopPropagation();
 		options_open = false;
 	}
+
+	$: console.log("webcam_accessed", webcam_accessed);


will delete 🤦

freddyaboulton · 2024-01-31T00:58:37Z

js/image/shared/Webcam.svelte

+				<WebcamIcon />
+			</span>
+			<button on:click={async () => await access_webcam()}
+				>{"Click to access webcam"}</button


This should probably be camel case for consistency

aliabid94 · 2024-01-31T18:08:28Z

I could've sworn I made the change to set the progress tracker visibility to False when streaming from a webcam

abidlabs · 2024-01-31T18:11:58Z

I could've sworn I made the change to set the progress tracker visibility to False when streaming from a webcam

I'm pretty sure its a regression. We should do this for any streaming event, e.g. #7230 as well

abidlabs · 2024-01-31T18:13:49Z

This looks great to me @freddyaboulton! My only feedback is that the clickable area to give access to the webcam is very small. I kept clicking the icon and was confused why it wasn't working. I'd suggest making this whole area clickable:

freddyaboulton · 2024-01-31T19:28:52Z

@aliabid94 Yea I remember too. I think it was this PR - #6709

Maybe we accidentally merged a regression.

aliabid94 · 2024-01-31T20:25:33Z

Is the demo supposed to be using an audio component? Btw when I was running locally I did not see the progress bar but also I got this error after a while when streaming a webcam:
ValueError: Cannot process this value as an Image, it is of type: <class 'tuple'>

aliabid94 · 2024-01-31T21:14:48Z

Btw when I was running locally I did not see the progress bar

Actually there was a visible progress bar only when using streaming from Interface, fixed in this PR in this commit: db019e4

freddyaboulton · 2024-01-31T22:15:26Z

Actually there was a visible progress bar only when using streaming from Interface, fixed in this PR in this commit: db019e4

Nice - confirmed it works! 🔥

Is the demo supposed to be using an audio component?

Yea that's my bad - copy/pasted the wrong demo. I updated the PR description to include the correct one but also including it here

import gradio as gr
import numpy as np

def flip(im):
    return np.flipud(im)

with gr.Blocks() as demo:
    with gr.Row():
        input_img = gr.Image(sources=["webcam"])
        output_img = gr.Image()
        input_img.stream(flip, input_img, output_img)
demo.launch(share=True)

if __name__ == "__main__":
    demo.launch()

My only feedback is that the clickable area to give access to the webcam is very small.

Agreed - just pushed up a fix! Will wait for CI to pass before opening it up for broader review.

aliabid94 · 2024-02-01T21:59:49Z

js/image/shared/Webcam.svelte

+		const func = mode === "image" ? take_picture : take_recording;
+		func();


let's avoid abbreviations like func. Personally think this is more readable tbh

Suggested change

const func = mode === "image" ? take_picture : take_recording;

func();

if (mode === "image") {

take_picture();

} else {

take_recording();

}

aliabid94 · 2024-02-01T22:07:44Z

js/image/shared/Webcam.svelte

@@ -253,6 +282,10 @@
 		height: var(--size-full);
 	}

+	.invisible {


nit, but we use the class name hide throughout the codebase for display: none;, for consistency would be nice

aliabid94

LGTM great work

dawoodkhan82 · 2024-02-01T22:31:30Z

Looks great @freddyaboulton. Small nit, I notice after taking a picture the input component quickly flashes back to the "click to access webcam" state before showing the taken picture.

abidlabs · 2024-02-02T00:21:39Z

Maybe also fixes: #7230?

freddyaboulton · 2024-02-02T17:55:14Z

Thanks for the reviews everyone! Will merge the comments and flicker have been addressed.

freddyaboulton commented Jan 31, 2024

View reviewed changes

freddyaboulton force-pushed the record-button-for-image-streaming branch from db019e4 to 1395b21 Compare January 31, 2024 22:12

freddyaboulton marked this pull request as ready for review January 31, 2024 22:35

freddyaboulton requested review from pngwn, aliabid94, dawoodkhan82, abidlabs, aliabd and hannahblair January 31, 2024 22:35

freddyaboulton mentioned this pull request Feb 1, 2024

Output flickering in stream mode #7230

Closed

1 task

aliabid94 reviewed Feb 1, 2024

View reviewed changes

aliabid94 approved these changes Feb 1, 2024

View reviewed changes

freddyaboulton and others added 5 commits February 2, 2024 10:47

Implementation

e762835

add changeset

4c8f7d3

remove progress display when streaming from Interface

79e6db3

Make entire icon clickable

9105fb6

lint

cc414c1

Address changes

9255ecd

freddyaboulton force-pushed the record-button-for-image-streaming branch from b26b2c5 to 9255ecd Compare February 2, 2024 16:22

freddyaboulton added 2 commits February 2, 2024 11:36

Fix story

14e8236

Edit

0ef5c7f

Merge branch 'main' into record-button-for-image-streaming

8ed0504

freddyaboulton enabled auto-merge (squash) February 2, 2024 17:59

freddyaboulton merged commit 2e6672c into main Feb 2, 2024
8 checks passed

freddyaboulton deleted the record-button-for-image-streaming branch February 2, 2024 18:26

pngwn mentioned this pull request Feb 2, 2024

chore: update versions #7177

Merged

freddyaboulton mentioned this pull request Feb 5, 2024

Add back source selection interaction test for image #7308

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow start/pause of streaming image input. Only access the webcam while it's needed #7228

Allow start/pause of streaming image input. Only access the webcam while it's needed #7228

freddyaboulton commented Jan 31, 2024 •

edited

gradio-pr-bot commented Jan 31, 2024 •

edited

gradio-pr-bot commented Jan 31, 2024 •

edited

Something isn't right?

freddyaboulton Jan 31, 2024

freddyaboulton Jan 31, 2024

aliabid94 commented Jan 31, 2024

abidlabs commented Jan 31, 2024

abidlabs commented Jan 31, 2024

freddyaboulton commented Jan 31, 2024

aliabid94 commented Jan 31, 2024

aliabid94 commented Jan 31, 2024

freddyaboulton commented Jan 31, 2024

aliabid94 Feb 1, 2024

aliabid94 Feb 1, 2024

aliabid94 left a comment

dawoodkhan82 commented Feb 1, 2024

abidlabs commented Feb 2, 2024

freddyaboulton commented Feb 2, 2024

		const func = mode === "image" ? take_picture : take_recording;
		func();

-		const func = mode === "image" ? take_picture : take_recording;
-		func();
+		if (mode === "image") {
+		  take_picture();
+		} else {
+		  take_recording();
+		}

Allow start/pause of streaming image input. Only access the webcam while it's needed #7228

Allow start/pause of streaming image input. Only access the webcam while it's needed #7228

Conversation

freddyaboulton commented Jan 31, 2024 • edited

Description

Webcam Demo no streaming

Webcam Demo with streaming

🎯 PRs Should Target Issues

Tests

gradio-pr-bot commented Jan 31, 2024 • edited

🪼 branch checks and previews

gradio-pr-bot commented Jan 31, 2024 • edited

🦄 change detected

This Pull Request includes changes to the following packages.

With the following changelog entry.

Something isn't right?

freddyaboulton Jan 31, 2024

Choose a reason for hiding this comment

freddyaboulton Jan 31, 2024

Choose a reason for hiding this comment

aliabid94 commented Jan 31, 2024

abidlabs commented Jan 31, 2024

abidlabs commented Jan 31, 2024

freddyaboulton commented Jan 31, 2024

aliabid94 commented Jan 31, 2024

aliabid94 commented Jan 31, 2024

freddyaboulton commented Jan 31, 2024

aliabid94 Feb 1, 2024

Choose a reason for hiding this comment

aliabid94 Feb 1, 2024

Choose a reason for hiding this comment

aliabid94 left a comment

Choose a reason for hiding this comment

dawoodkhan82 commented Feb 1, 2024

abidlabs commented Feb 2, 2024

freddyaboulton commented Feb 2, 2024

freddyaboulton commented Jan 31, 2024 •

edited

gradio-pr-bot commented Jan 31, 2024 •

edited

gradio-pr-bot commented Jan 31, 2024 •

edited