Feature request: stream images from video in realtime

[tonydavis629]

• I have searched to see if a similar issue already exists.

Is your feature request related to a problem? Please describe.
It is not possible to stream images from a video in real time with gradio. I want to be able to upload a video, process a frame, and return the frame in real time. Currently Interface does not return images in real time, no matter how fast your iterative output is, it's probably at 2 or 3 frames per second.

Minimal example:

import gradio as gr
import cv2

def process_video(input_video):
    cap = cv2.VideoCapture(input_video)

    output_path = "output.mp4"

    fps = int(cap.get(cv2.CAP_PROP_FPS))
    width  = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
    height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))

    video = cv2.VideoWriter(output_path, cv2.VideoWriter_fourcc(*"mp4v"), fps, (width, height))

    iterating, frame = cap.read()
    while iterating:

        # flip frame vertically
        frame = cv2.flip(frame, 0)
        display_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

        video.write(frame)
        yield display_frame, None

        iterating, frame = cap.read()

    video.release()
    yield display_frame, output_path

with gr.Blocks() as demo:
    with gr.Row():
        input_video = gr.Video(label="input")
        processed_frames = gr.Image(label="last frame")
        output_video = gr.Video(label="output")

    with gr.Row():
        examples = gr.Examples(["parrot.mp4"], inputs=input_video)
        process_video_btn = gr.Button("process video")

    process_video_btn.click(process_video, input_video, [processed_frames, output_video])

demo.queue()
demo.launch()

Describe the solution you'd like
When running this example, there should be a way to specify the frame rate returned by the Interface.

Additional context
#1637

[freddyaboulton]

Maybe the same approach as #5077 can be used

[abidlabs]

Yup I agree, @aliabid94 would you be able to see if there's something similar we can implement for streaming a Video?

[aliabid94]

We could definitely have streaming video, but frame by frame will probably always be laggy - you'd ideally returns a chunk of video content. Would that work for you @tonydavis629? I can go ahead and implement streaming video, where you yield chunks of video content. If you really wanted to, your video content could just be a single frame I suppose, though I'd bet this would be quite slow.

[RustX2802]

Hey guys, do you have any update regarding webcam stream?
I tried to pass a function which does the real-time video processing and visualization. I set the input to None and just let inferring the processed frame from the camera as output. The code seems working but not as intended. I mean, the visualization opens locally but not within Gradio canvas. Here is the minimal example:

def process_frames():
    cam_stream.start()
    while True:
        frame = cam_stream.read()
        predictions = anomaly_detection(frame)
        if predictions is None:
            yield None
        else:
            output_image = visualizer.visualize_image(predictions)
            yield output_image
        time.sleep(0.1)
        if cv2.waitKey(1) & 0xFF == ord("q"):
            break
    cam_stream.stop()
    cv2.destroyAllWindows()
iface = gr.Interface(
    fn=process_frames,
    inputs=None,
    outputs=gr.outputs.Image(type="numpy", label="Segmentation Result"),
)
iface.queue()
iface.launch()

Do you have any suggestions in this regard?

[JonasVerbickas]

We could definitely have streaming video, but frame by frame will probably always be laggy - you'd ideally returns a chunk of video content. Would that work for you @tonydavis629? I can go ahead and implement streaming video, where you yield chunks of video content. If you really wanted to, your video content could just be a single frame I suppose, though I'd bet this would be quite slow.

Have you made any progress on this?
I want to allow users to upload larger video files and these take longer to run inference on.
Currently, I utilize a gradio progress bar to indicate that the video is being worked on, but the user experience would be so much better if frames were shown as they were generated.
I would think this feature would interest anyone using gradio and producing video outputs.

[jacobdang]

This would be highly useful. Just wondering is there any plan to add this feature? Currently I am doing a robot project, and trying to display the ros node output to the gradio image component. But it cannot be done in real-time or a streaming fashion.

[raghavduddala]

@abidlabs or @aliabid94 any update on this would be really helpful, as I too want to run inference for a livestream of video feed coming in.

[freddyaboulton]

Working on this now @raghavduddala