Custom GStreamer media source and renderer in libopenwebrtc-gst library

[superdump]

Given the vast number of GStreamer audio/video source/sink elements, it is not feasible and does not seem sensible to require implementation of them all in OpenWebRTC, nor for people to maintain a fork of OpenWebRTC to do so for their element.

At the same time, it is not desirable to have to ship all GStreamer API in our releases for people who are not interested in it.

I propose that we implement custom OwrMediaSource and OwrMediaRenderer classes that have a property on them to set a GstElement. We should define how the element is expected to work (what caps are supported, what queries, events, meta etc are supposed to be handled) and build that into a libopenwebrtc-gst library. This library, when shipping, should then include GStreamer API alongside it in order to allow people to make use of it.

[sdroege]

Sounds good to me. What special behaviour requirements do you have in mind for the sources/sinks? Shouldn't just any work that behaves like a normal GStreamer sink/source?

[superdump]

I was thinking mostly of caps and what will be handled by owr / by the element/application. I haven't thought through the smaller details yet so it's more an idea. Maybe it will be quite generic and there won't really be requirements. :)

[philn]

Sounds good to me as well :)

[clementperon]

And i think that could also help to support compressed media source as well no ?

[sdroege]

Not much as that mostly requires changes inside the other parts of openwebrtc

[clementperon]

Yes, of course just that set/access to the caps of the src would help instead of trying to deduct them after.

[hlevring]

Your work is much appreciated, really fantastic!!
Just wanted to know what timeframe you expect before it will be possible to pass custom video source to OpenWebRTC ?

[superdump]

It's not currently top priority. Perhaps @philn is interested in putting this together. It would be nice to do it so that the custom gst source/renderer classes basically just add a property to set the element on the appropriate context member. I think that context member already exists, but not totally sure. It should be a small amount of code changes at least.

[philn]

@hlevring feel free to test my branch and give some feedback :)
https://github.com/philn/openwebrtc/tree/owr-gst

[ck3mp3r]

All roads lead to #221. Am I correct in assuming that the precompiled libs don't have the support for the sub-library enabled? Hence I'm going to have to compile them myself for iOS etc?

[superdump]

That is correct.

[ck3mp3r]

Any particular reason this isn't enabled by default?

[superdump]

Because we want to only require that people know the OpenWebRTC API by default rather than letting the GStreamer API seep through.

Some standard use cases should be integrated into the core of OpenWebRTC such as being able to provide or retrieve compressed or uncompressed buffers in well-defined formats to do with whatever you like.

Also, playing back from a file (I have made a branch for this but I think it still needs a bit of work on the API and internally) as well as recording to a file.

Another interesting use case is the ability to replay pcap files.

And then if you want to do something more advanced, you will need to build the extra library with the extra API for custom GStreamer sources and renderers. Perhaps we or someone will contribute everything needed to make binary releases of that additional library as a separate framework that depends on the OpenWebRTC framework and also adds the GStreamer API.

[ck3mp3r]

That makes perfect sense.
My use case is exactly that: access to the audio and video buffers to be able to write them to disk etc.
I'm just starting to wrap my head around GST and OWR, I'm looking for a way to tap into the video/audio coming in from a call to be able to further use it.
Any examples you can point out?
Whatever solution I come up with I'll gladly share back here.

[superdump]

I think probably something like the OwrURISourceAgent ( #436 ) but an OwrRecordingAgent perhaps. It should use GStreamer's encodebin element internally, it could have a file path property for where to record to, some preset properties to configure what GstPresets for encodebin to use, some way to add / remove sources used for recording where one source maps to one output track. Something along those lines perhaps.

[ck3mp3r]

I'll try the extension lib for now. My other alternative is Multipeer Connectivity and plain GST...

[ck3mp3r]

How do I use --enable-owr-gst with cerbero?

[superdump]

For now you can just hack it in for your platform(s) of interest.

[ck3mp3r]

I figured as much, still looking for the right place in the config files...

[superdump]

recipes/openwebrtc.recipe is the file to modify. You should be able to just add --enable-owr-gst there.

[ck3mp3r]

Hi, thanks I found this from a comment you made elsewhere already. Will give it a try. Thanks for your help!

[ck3mp3r]

Just a further note: if you use --enable-owr-gst you also need to use --disable-owr-static.
''configure: error: owr-gst is not supported with owr-static''

[superdump]

I think I know why we did that (static builds require listing all the symbols beforehand and we didn't want to have varying API depending on what was built.) But it will mean that it won't work on iOS. We will have to put together some guide or something for how to build OWR-gst for use on all platforms.

[ck3mp3r]

Oh, that just put a spanner in the works for me... looks like I'll have to revert to multipeer connectivity and plain GST for now...

[superdump]

It's not so much work to get it working. Just that the owr-gst symbols need to be added to the symbols list in libopenwebrtc.exp and that needs committing to say a local git branch. Then the openwebrtc.recipe needs modifying to point to that repository (change the origin remote to file:///Users/name/path/to/repo) and the commit to origin/branchname. And then rebuild for OS X and iOS I think.

[ck3mp3r]

I'm not familiar with all of the details, let alone the api. I need to build a proof of concept and I am trying to future proof it by doing it with open standards...
Is there any way we can achieve this together? I am sure I am not the only one interested in using openwebrtc with the features gst gives us out of the box.

[ck3mp3r]

When you say symbols you are just referring to the functions defined in the headers being added to the list?

Are there any other changes needed in cerbero apart from "--disable-owr-static --enable-owr-gst --disable-tests"?

[superdump]

I think that should be sufficient. Do you still have issues?

[ck3mp3r]

I've cloned the repo and am working on it. See the reference above. I'll let you know how I get on! Thanks!

[ck3mp3r]

Finally managed to build everything. Not sure if it will work yet but I have a different issue now. How do I use cocoa pods to integrate my built framework? Or is it better to remove the cocoa pod dependency on openwebrtc and just link via Xcode directly?

[ck3mp3r]

I modified everything like we discussed, built the frameworks and no owr_gst_*.h files to be seen anywhere in the output framework.... any ideas? Are there more settings I need to change apart from the build flags and also symbol catalog?

Edit:
Turns out that the iPhone framework has the header files required but not the darwin build, is that intentional, considering I edited the main recipe?

[superdump]

I need to investigate this thoroughly. I think we should make a separate framework for the extra GStreamer-bits so that people who don't want that don't need to install it.

[ck3mp3r]

@superdump that makes sense, but for now I think I'll just see how things work on iOS. It would be great to be able to use the owr_gst bits on the mac as well though...
I'll keep on checking back, or do you have a rough idea as to when you'd think you'd have it on a branch or something?

[superdump]

I want to do it for the next release (0.4) which I am hoping to do within the coming month or thereabouts, mostly depending on fixing #451. Ideally also #385 but I'm not so hopeful about that one currently, unless someone does it upstream.

[ck3mp3r]

@superdump any ETA on this?