can I do real-time editing/filtering on the web stream?

[YubinXie]

I was wondering if I can apply some filters/mask on the streaming video? This requires access to the frames in real-time. Is it possible to do this on this package?

I have tried to capture frame one by one using cv2: clean the previous one and display the next one. However, what I actually see is firstly the first image disappears and the output space of this cell also disappears; then the next image comes and increases the output space. Overall, you will the whole page is shaking up and down due to the clean and display step.

It would be great if I can do the modification on the streaming on this package! Thanks.

[martinRenou]

I made an example using skimage instead of opencv, but I guess you can achieve something similar with opencv: https://github.com/QuantStack/quantstack-talks/blob/master/2018-11-14-PyParis-widgets/notebooks/6.ipywebrtc.ipynb

In the example I process images took from the chat room, but you can do the same taking images from a CameraStream or a VideoStream.

[martinRenou]

So basically the process is the following: Create your input stream (CameraStream for example) an create an ImageRecorder.

camera = CameraStream()
recorder = ImageRecorder(stream=camera)

Then you implement your image processing logic, taking recorder.image.value as input (it's a binary value). And you can use the Image widget (from ipywidgets) as output:

out = Image()

This way you can update the out.value in your image processing logic. And display the out widget under the cell.

[YubinXie]

Great! Thanks!

[MetaDev]

Thanks for providing this example, this set me up on the right path.
However, the loop is quite slow, the webcam image is lagging even without processing.
This is probably due to the back and fort conversion with PIL?:

im_in = PIL.Image.open(io.BytesIO(image_recorder.image.value))
im_array = np.array(im_in)[...,:3]
f = io.BytesIO()
im_out.save(f, format='png')

Is there any way around it?

[martinRenou]

If you do a quick profiling of your code, I suspect the slow part is the saving of the image.

[MetaDev]

I commented out all code except,

im_in = PIL.Image.open(io.BytesIO(image_recorder.image.value))
im_array = np.array(im_in)[...,:3]

and it is still slow.
comparing webcamstream and imagestream of that webcam I see a very strong drop in FPS in the imagestream.

[martinRenou]

Oh I see, I'm not sure using an ImageStream is a good idea. Try using the ImageRecorder directly on the CameraStream

[MetaDev]

I'm sorry I meant ImageRecorder with webcam as stream, maybe I should use videorecorder?

[martinRenou]

The VideoRecorder would be faster, and then you could try extracting each frame of the video using OpenCV or maybe PIL if it can.

[MetaDev]

Is there documentation on how to that? It is not really clear from the api how to access the video frame.

…

On Wed, Jul 17, 2019 at 11:52 AM martinRenou ***@***.***> wrote: The VideoRecorder would be faster, and then you could try extracting each frame of the video using OpenCV or maybe PIL if it can. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#50?email_source=notifications&email_token=ABXGUECM6SWTAXESRTPAK73P73TWXA5CNFSM4GZZMZVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2DVOJI#issuecomment-512186149>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABXGUEAY476YCX352TCR4C3P73TWXANCNFSM4GZZMZVA> .

[martinRenou]

Using the VideoRecorder you could maybe save the video to a file, then use OpenCV to load the file and extract the frames? You would need to look at OpenCV documentation for doing this.

[MetaDev]

Yeah, but once I save to a file, than I am no longer processing real-time. I am just wondering, because the webcam stream is really smooth, so it would be nice to access the webcam frames without having a drop in frame rate. Thanks for the quick response :).

…

On Wed, Jul 17, 2019 at 11:58 AM martinRenou ***@***.***> wrote: Using the VideoRecorder you could maybe save the video to a file, then use OpenCV to load the file and extract the frames? You would need to look at OpenCV documentation for doing this. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#50?email_source=notifications&email_token=ABXGUEH7OKZFSLQ7AXOJ5VTP73UMFA5CNFSM4GZZMZVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2DV5PY#issuecomment-512188095>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABXGUEFY6NL6D2ZSIZTCOZ3P73UMFANCNFSM4GZZMZVA> .

[martinRenou]

than I am no longer processing real-time

Indeed, the VideoRecorder is for recording videos out of the stream. For now, the only way you can do it is with the ImageRecorder, even if it is slow...

There might be some room for improvement in terms of performances here.

[wolterlw]

Is it possible to create a VideoRecorder variant that acts anything like OpenCV's VideoCapture(0)
so that one can process the frames as a numpy array and then display them using 3rd-party tools, like bokeh for example?

[martinRenou]

My guess is that you can dump the video output of the VideoRecorder into a video file and use OpenCV's VideoCapture on this file?

[wolterlw]

@martinRenou no, there are much better ways of doing that without ipywebrtc.
So the use case I believe many people have is to make life demos that execute on remote servers but display results in a browser. In my particular case I perform hand pose estimation and it works much faster on a server, but it's quite complicated to stream video from the webcam there yourself.