gstreamer-imx / gleffects [opengl] interaction

[Buanderie]

Is it possible to use gstreamer-imx plugins along with opengl plugins from the gst-plugins-bad (1.4.5) to apply opengl effects and/or opengl shaders ?

Using the following pipeline:
gst-launch-1.0 imxv4l2videosrc imx-capture-mode=10 ! imxg2dvideotransform ! video/x-raw, width=800, height=600, format=RGBA ! gleffects effect=0 ! imxeglvivsink
I get very low framerate...

While using this pipeline:
gst-launch-1.0 imxv4l2videosrc imx-capture-mode=10 ! imxg2dvideotransform ! video/x-raw, width=800, height=600, format=RGBA ! gleffects effect=0 ! imxipuvideotransform ! video/x-raw, format=I420 ! imxeglvivsink
I get the following error:
ERROR: from element /GstPipeline:pipeline0/GstGLEffects:gleffects0: Failed to download video frame

Also, using this pipeline:
gst-launch-1.0 imxv4l2videosrc imx-capture-mode=10 ! imxg2dvideotransform ! video/x-raw, width=800, height=600, format=RGBA ! imxeglvivsink
I get high framerate, with low latency, which is what I want...

The thing is, I would like to apply image processing and/or effects to my video before output... And since I'm already using DMA memory through gstreamer-imx plugins, the idea was to use shaders for image processing...

Is there something I'm missing, here ? How gstreamer-imx is supposed to be interfaced with gstreamer opengl plugins ? Should I switch to more recent version of the opengl plugins ? (I'm using 1.4.5 on my system, I know they moved the opengl plugins from bad to base in the newer releases...)

Thanks for your help

[Buanderie]

Again using glshader... This pipeline:
gst-launch-1.0 imxv4l2videosrc imx-capture-mode=10 ! imxg2dvideotransform ! video/x-raw, width=800, height=600, format=RGBA ! glshader location=/usr/share/gray_shader.fs ! imxeglvivsink

gives very poor results...
GST_DEBUG gives me some clue about what is going on...
0:00:16.483679336 24303 0x1278d50 LOG glupload gstglupload.c:310:gst_gl_upload_perform_with_buffer:<glupload0> Attempting viv direct upload 0:00:16.483803002 24303 0x1278d50 LOG glupload gstglupload.c:319:gst_gl_upload_perform_with_buffer:<glupload0> Attempting upload with raw data 0:00:16.489012002 24303 0x1278d50 TRACE glupload gstglupload.c:546:_upload_memory: uploading with textures:4,0,1991323946 dimensions:800x600 0:00:17.107529669 24303 0x1278d50 LOG glupload gstglupload.c:310:gst_gl_upload_perform_with_buffer:<glupload0> Attempting viv direct upload 0:00:17.107674336 24303 0x1278d50 LOG glupload gstglupload.c:319:gst_gl_upload_perform_with_buffer:<glupload0> Attempting upload with raw data 0:00:17.112876669 24303 0x1278d50 TRACE glupload gstglupload.c:546:_upload_memory: uploading with textures:4,0,1991323946 dimensions:800x600 WARNING: from element /GstPipeline:pipeline0/GstImxEglVivSink:imxeglvivsink0: A lot of buffers are being dropped.

Seems like glshader is copying from the physical memory to CPU then back to physmem... Taking a lot of time, indeed.

rawling through the bad plugin sourcecode, I found this:
`GST_LOG_OBJECT (upload, "Attempting viv direct upload");
if (gst_is_physical_buffer (buffer)) {
if (gst_gl_viv_direct_bind_gstbuffer (upload->context, upload->priv->tex_id, &upload->in_info, buffer)) {
*tex_id = upload->priv->tex_id;
return TRUE;
}
}

raw_data_upload:
GST_LOG_OBJECT (upload, "Attempting upload with raw data");
/* GstVideoMeta map */
if (!gst_video_frame_map (&upload->priv->frame, &upload->in_info, buffer,
GST_MAP_READ)) {
GST_ERROR_OBJECT (upload, "Failed to map memory");
return FALSE;
}`

I know this is not gstreamer-imx related, now... Just if you could give me some clue about what to do. Maybe upgrading plugins-bad woud do the trick ? All my gstreamer yocto packages are tied to 1.4.5 version, so that would be a lot of work to make it work with another version.

[BMValeev]

Have you solved that issue yet?

[Buanderie]

Hello, no I did not.
As a matter of fact, my objective was to write a custom GLSL shader video filter. So that's what I did. I wrote a custom element based on gstreamer-imx imxeglvivsink that renders my texture (bound using VIV OpenGL extensions) to the framebuffer. Then I memcpied the framebuffer using mmap and fast NEON memcpy to ouput the frame to the other side of my element (since glReadPixels was WAY too slow). Ugly, but it did the trick...

[BMValeev]

Then I memcpied the framebuffer using mmap and fast NEON memcpy to ouput the frame to the other side of my element (since glReadPixels was WAY too slow).

Could you please send a link about that NEON based memcpy, I cant found information more informative than https://stackoverflow.com/questions/11161237/fast-arm-neon-memcpy
Have you used fb or wayland? I have found that wayland doesn't work properly.

[adwaitpatil]

Hello, I am not not sure if this is relevant at all but it might help.

On my Raspberry Pi (I don't have an NXP board right now) I usually put the 'glupload' element before the 'gleffects' element in my pipeline and it works.

Example pipeline:
gst-launch-1.0 -vvv v4l2src ! "image/jpeg, width=640, height=480" ! omxmjpegdec ! "video/x-raw(memory:GLMemory), format=RGBA, width=640, height=480" ! glupload ! gleffects effect=fisheye ! glimagesink sync=false

[BMValeev]

Hello, I am not not sure if this is relevant at all but it might help.

On my Raspberry Pi (I don't have an NXP board right now) I usually put the 'glupload' element before the 'gleffects' element in my pipeline and it works.

Example pipeline:
gst-launch-1.0 -vvv v4l2src ! "image/jpeg, width=640, height=480" ! omxmjpegdec ! "video/x-raw(memory:GLMemory), format=RGBA, width=640, height=480" ! glupload ! gleffects effect=fisheye ! glimagesink sync=false

Sorry, but that is not connected. Main reason basically were DMA based "unique" memory allocator in gstreamer-imx. Gstreamer-core supports 4 types of memory allocation, and when you try to connect glupload and imxv4l2videosrc, it checks common allocator type. Because of unique allocation type, glupload does not support it, and gstreamer use cpu based memory copy. For that reason as result I added own plugin based on eglvivsink, and used as shader configurator. Also waf was configured wrong in standard yocto layer. I've to corrected that.

[Talkless]

For that reason as result I added own plugin based on eglvivsink

@BMValeev have you considered contributing that code to gstreamer-imx to make it compatible with glupload by default?