Add streaming capability

[wolfv]

It could be interesting to add real audio / video streaming capability.

I am not sure how that would work in detail, but this: https://github.com/jlaine/aiortc could be a reusable implementation of the webrtc protocol in Python.

The idea would be that you could generate sound in the backend (e.g. DSP filters or similar stuff) and then stream the result to the browser / frontend.

[martinRenou]

I guess we can use the RTCDataChannel on the front-end (See this webrtc example)
EDIT: What it does is that it sends data chunks by chunks

[wolfv]

I think the PeerConnection might be more directly relevant: https://webrtc.github.io/samples/src/content/peerconnection/audio/

[martinRenou]

I guess you are right :)

[maartenbreddels]

ping @izaid who was using websockets, but I'm not sure there is a standard we could support.

Basically all the magic behind ipywebrtc.chat implements the PeerConnection.
It creates a WebRTCRoomMqtt object, which finds peers using a MQTT server. Based on the messages, it sets up a RTCPeerConnection which is wrapped by WebRTCPeer, which contains the local and remove streams.

[maartenbreddels]

I think the implementation can be done better, for instance, it is now not possible to connect to a remote peer if you have the 'connection info' (basically a piece of text that goes in a RTCSessionDescription to describe the connection).

You can imagine a remote camera, for which you have this info in a text file. From that description you should be able to connect to it, but currently, that's not how it is implemented.

[izaid]

Hey all -- thanks for pinging me, this is very cool. I want to have a more in-depth look in the next little while.

Just FYI, the reason we are using websockets is because they are pretty well supported from a non-Javascript environment, so for things like embedded devices, etc. There are a couple pretty good C / C++ websocket libraries. I looked hard for a decent WebRTC library in C / C++, but came up real short.

Anyway, that's just stuff specific to me.

[maartenbreddels]

Actually, if we support a stream over a websocket, ipywebrtc can be a way to convert to webrtc.

Does any IP webcam work with a websocket?

[wolfv]

I am no expert but it seems like Google provides a native (C++) protocol implementation with examples on the WebRTC website: https://webrtc.org/native-code/native-apis/

Could be interesting to test that out.

[izaid]

@maartenbreddels I'd be interested in trying this. I don't know if any IP webcam works with a websocket, but it's relatively easy to get websockets going and just stream frames encoded on the fly.

@wolfv Yeah, I'm aware of the Google repository. I found that very hard to get working and ultimately not worth the effort, others may have different experiences.

[Raska]

Hello,
I was wondering if the discussion was still topical about returning chunks during live stream to use it directly in jupyter notebook. A way such as pyaudio does...
Thank you for your work and your answer.
Best regards.