Python Frontend/Client to receive Pixel Streaming output

[devrim]

Component your feature request relates to

• Signalling server
• Frontend library
• Frontend UI library
• Matchmaker
• SFU
• This repo

Problem your feature solves? Please describe.
A Python Frontend that connects to Signalling Server and receives Pixel Streaming output just like the JS client.

Additional context
It'd be amazing to consume Pixel Streaming output in Python to be consumed by OpenCV and the like to train AI/ML models. Is it possible? If yes, how can one go about implementing it?

[lukehb]

Hi @devrim,

First of all I have relabelled your issue from an enhancement to a question to set your expectations here. We have no intent of shipping a python client.

However, I am happy to answer your question and I hope it will help you and others who are wanting to make custom clients for Pixel Streaming.

In general a custom client will have to do the following things:

• Use WebSocket to communicate to the existing signalling server using the player messages (I recommend you study the signalling protocol document to learn what messages and their order that you will have to implement).
• Use WebRTC to take the signalling messages, offer/answer, iceCandidate to establish a P2P WebRTC connection between your WebRTC client library and the UE Pixel Streaming plugin
• You will have to implement some way to render your video track to screen using your chosen WebRTC library. This is library and language dependent.

As for Python in particular I do not have any recommendations for WebRTC libraries to use, but a quick Google reveals you have some choices.

As for the second part of your question, which is consuming the Pixel Streaming video output by some other library (e.g. OpenCV), well again, this depends on the WebRTC library you use, but conceptually you should be able to attach a video sink (this is the WebRTC terminology) to your video track or your peer connection and that video sink will give you the raw decoded frames, which you can then pass to OpenCV or whatever other library you like.

Hope this helps!