Negotiate GL textures and render them in player example #182

ceyusa · 2019-01-23T16:03:30Z

This PR goes above PR #196

It needs to update the CI GStreamer setup, because it requires a new version of glib and gstreamer-gl library.

It contains

Port to webrender 0.58
Port to gstreamer git head version (0.13.0) -- this implied a complete rewrite of servosrc
Add a new trait in Frame to hold a reference to the backend's buffer
Update glutin version tu current git head
Add a method to set the GL parameters in the backend's player
Use glsinkbin in gstreamer player in GL parameters are negotiated
A lot of changes in example player to render gl textures is --gl argument is passed

Fixes: #166

backends/gstreamer/src/audio_decoder.rs

backends/gstreamer/src/source.rs

ceyusa · 2019-01-28T15:09:02Z

Right now, WebRTC GStreamer backend doesn't compile: there are multiple errors of this type

error[E0623]: lifetime mismatch                                                                                                                                                                                    
  --> backends/gstreamer/src/media_capture.rs:20:36                                                                                                                                                                
   |                                                                                                                                                                                                               
17 |         builder: gst::caps::Builder,                                                                                                                                                                          
   |                  ------------------ this parameter and the return type are declared with different lifetimes...                                                                                               
18 |     ) -> Option<gst::caps::Builder> {                                                                                                                                                                         
   |          --------------------------                                                                                                                                                                           
19 |         match self {                                                                                                                                                                                          
20 |             Constrain::Value(v) => Some(builder.field(name, v)),                                                                                                                                              
   |                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...but data from `builder` is returned here

backends/gstreamer/src/media_capture.rs

ceyusa · 2019-02-05T10:35:30Z

Last Friday I confirmed my suspicion that the media's frame should keep a living reference to the associated GStreamer buffer, otherwise the buffer gets unreffed and the texture turns invalid.

Since media's frame is backend's agnostic, it would be hard to keep a reference to a opaque structure, so I tried out to turn frame into a trait.

But soon I got hit by FrameRenderer trait, which function render() receives a Frame, which now turned into a trait, FrameRenderer violates the rules of trait object safety. And I'm kind of trapped there.

ceyusa · 2019-02-13T18:39:40Z

@ferjm @jdm r?

ferjm

Nice! Thank you so much for doing this work.

As mentioned on IRC, we need to figure out the codepath for OSX before being able to merge this.

I'd love to hear @sdroege final feedback as well :)

backends/gstreamer/src/source.rs

backends/gstreamer/src/player.rs

sdroege · 2019-02-14T11:10:33Z

I'd love to hear @sdroege final feedback as well :)

Won't get to that today but I'll put it on my list.

backends/gstreamer/Cargo.toml


        Port to git version of gstreamer (0.13.0)

This implies * to use glib::GString for media_info data * gst::Buffer::from_slice() doesn't fail * New returns as Result simplifyng code * Rewrite ServoSrc element using subclassing * add life times to media_capture caps builder * enable NLL for gstreamer crate


        Use expect() for programming errors

For programming errors it is better to panic using expect(). This patch does this for GObject's set_property and get_property.


        Add Buffer trait to Frame

And implement it in GStreamer backend. The purpose of this trait is to keep alive the reference of the backend's buffer while it is used.


        Port to webrender 0.58


        Use glutin git version

Since the current unreleased version has merged the required API to get the used EGL Display.


        Add player API to set GL context and display


        Handle --gl argument in example

The purpose of this argument is to select if the user wants to render textures or images. *This is just a mock up*, textures are not handled yet.


        Set app's GL context to player

Move upper the player_wrapper instantiation in order to set the gl usage as soon as possible.


        Enable player example to render textures

If buffer contains a GL texture it will return the texture id


        Add an image key initializator


        Add FrameProvider for textures as external images


        Use glsinkbin and handle bus sync messages

If gl display and context are set, the glsinkbin is used, otherwise only appsink. Now GL textures are negotiated and rendered.


        Use dyn to mark Player trait object in example


        Texture upload example only works for linux

ceyusa · 2019-02-20T18:57:57Z

I guess all (or most of) the issues were solved :)

sdroege · 2019-02-20T21:20:17Z

I guess all (or most of) the issues were solved :)

🎊 Nothing else from my side, I guess :)

ferjm · 2019-02-21T09:56:06Z

@bors-servo r+

Manishearth · 2019-02-21T10:12:55Z

@bors-servo r=ferjm

bors-servo · 2019-02-21T10:12:55Z

📌 Commit 6a3be48 has been approved by ferjm

bors-servo · 2019-02-21T10:12:58Z

⌛ Testing commit 6a3be48 with merge 888668f...


        Auto merge of #182 - ceyusa:gl-textures, r=ferjm

Negotiate GL textures and render them in player example This PR goes above PR #196 It needs to update the CI GStreamer setup, because it requires a new version of glib and gstreamer-gl library. It contains * Port to webrender 0.58 * Port to gstreamer git head version (0.13.0) -- this implied a complete rewrite of servosrc * Add a new trait in Frame to hold a reference to the backend's buffer * Update glutin version tu current git head * Add a method to set the GL parameters in the backend's player * Use glsinkbin in gstreamer player in GL parameters are negotiated * A lot of changes in example player to render gl textures is --gl argument is passed Fixes: #166

bors-servo · 2019-02-21T10:23:46Z

☀️ Test successful - checks-travis
Approved by: ferjm
Pushing 888668f to master...