flutterpi_gstreamer_video_player and Asus TinkerBoard 2

[bojidartonchev]

Hello,

I am currently trying to get gstreamer working on Asus TinkerBoard 2 RK3399 using yocto, but I have some problems playing an .mp4 video through flutter app using the flutterpi_gstreamer_video_player plugin.

[locales] Warning: The system has no configured locale. The default "C" locale may or may not be supported by the app.
keyboard.c: Could not load keyboard configuration from "/etc/default/keyboard". Default keyboard config will be used. load_file: No such file or directory
gl_renderer.c: Could not resolve EGL/GL symbol "eglQueryDmaBufFormatsEXT"
gl_renderer.c: Could not resolve EGL/GL symbol "eglQueryDmaBufModifiersEXT"
plugins/gstreamer_video_player/player.c: Failed to set custom HTTP headers because gstreamer source element has no 'extra-headers' property.
plugins/gstreamer_video_player/player.c: gstreamer warning: Delayed linking failed. (debug info: gst/parse/grammar.y(546): gst_parse_no_more_pads (): /GstPipeline:pipeline0/GstURIDecodeBin:src:
failed delayed linking some pad of GstURIDecodeBin named src to some pad of GstAppSink named sink)
plugins/gstreamer_video_player/player.c: gstreamer error: code: 1, domain: gst-stream-error-quark, msg: Internal data stream error. (debug info: ../gst-plugins-good-1.18.5/gst/isomp4/qtdemux.c(6545): gst_qtdemux_loop (): /GstPipeline:pipeline0/GstURIDecodeBin:src/GstDecodeBin:decodebin0/GstQTDemux:qtdemux0:
streaming stopped, reason not-linked (-1))

I am initializing the VideoController with an asset:

_controller = VideoPlayerController.asset("assets/videos/movie.mp4")
	 ..initialize().then((_) {
        _controller.play();
        setState(() {});
      });

And this is the GST_DEBUG=3 output:

0:00:00.128624221   597   0x5593b2c070 WARN                 basesrc gstbasesrc.c:3688:gst_base_src_start_complete:<source> pad not activated yet
0:00:00.137615430   597   0x5593b2c070 WARN                 basesrc gstbasesrc.c:3688:gst_base_src_start_complete:<source> pad not activated yet
0:00:00.172598517   597   0x5593c4cf00 WARN                 qtdemux qtdemux_types.c:244:qtdemux_type_get: unknown QuickTime node type .TIM
0:00:00.172639350   597   0x5593c4cf00 WARN                 qtdemux qtdemux_types.c:244:qtdemux_type_get: unknown QuickTime node type .TSC
0:00:00.172652475   597   0x5593c4cf00 WARN                 qtdemux qtdemux_types.c:244:qtdemux_type_get: unknown QuickTime node type .TSZ
0:00:00.172802392   597   0x5593c4cf00 WARN                 qtdemux qtdemux.c:3066:qtdemux_parse_trex:<qtdemux0> failed to find fragment defaults for stream 1
0:00:00.173055850   597   0x5593c4cf00 WARN                 qtdemux qtdemux.c:3066:qtdemux_parse_trex:<qtdemux0> failed to find fragment defaults for stream 2
0:00:00.233019023   597   0x7ef000b000 WARN                 default gst/parse/grammar.y:546:gst_parse_no_more_pads:<src> warning: Delayed linking failed.
0:00:00.233072690   597   0x7ef000b000 WARN                 default gst/parse/grammar.y:546:gst_parse_no_more_pads:<src> warning: failed delayed linking some pad of GstURIDecodeBin named src to some pad of GstAppSink named sink
0:00:00.234496607   597   0x5593c4cf00 WARN                 qtdemux qtdemux.c:6545:gst_qtdemux_loop:<qtdemux0> error: Internal data stream error.
0:00:00.234537440   597   0x5593c4cf00 WARN                 qtdemux qtdemux.c:6545:gst_qtdemux_loop:<qtdemux0> error: streaming stopped, reason not-linked (-1)

I have tried launching the following pipeline, since I believe this is what your plugin does with gst-launch-1.0 from terminal and there are no errors. (I am still using appsink, so I can't see any video output, but it seems playing).
gst-launch-1.0 uridecodebin uri=file:///usr/share/flutter/kiosk/data/flutter_assets/assets/videos/movie.mp4 ! video/x-raw ! appsink sync=true

EGL version:

Using display 0x55c0f9bff0 with EGL version 1.4
===================================
EGL information:
  version: "1.4 Midgard-"r18p0-01rel0""
  vendor: "ARM"
  client extensions: "EGL_EXT_client_extensions EGL_EXT_platform_base EGL_KHR_client_get_all_proc_addresses EGL_KHR_platform_gbm EGL_MESA_platform_gbm EGL_KHR_platform_wayland EGL_EXT_platform_wayland"
  display extensions: "EGL_WL_bind_wayland_display EGL_KHR_partial_update EGL_KHR_image_pixmap EGL_EXT_image_dma_buf_import EGL_KHR_config_attribs EGL_KHR_image EGL_KHR_image_base EGL_KHR_fence_sync EGL_KHR_wait_sync EGL_KHR_gl_colorspace EGL_KHR_get_all_proc_addresses EGL_IMG_context_priority EGL_ARM_pixmap_multisample_discard EGL_ARM_implicit_external_sync EGL_KHR_gl_texture_2D_image EGL_KHR_gl_renderbuffer_image EGL_KHR_create_context EGL_KHR_surfaceless_context EGL_KHR_gl_texture_cubemap_image EGL_EXT_create_context_robustness"
===================================
OpenGL ES 2.x information:
  version: "OpenGL ES 3.2 v1.r18p0-01rel0.db8fd8841edf74cef96ef24ce665edac"
  shading language version: "OpenGL ES GLSL ES 3.20"
  vendor: "ARM"
  renderer: "Mali-T860"

Any suggestions would be appreciated!

[ardera]

Can you try gst-play-1.0 /usr/share/flutter/kiosk/...?

This seems like something with the video is wrong, or something with gstreamer. You could try using a newer gstreamer (building it yourself). But it's weird that gst-launch-1.0 uridecodebin ... works.

Though that's not exactly what flutter-pi is doing, flutter-pi is using a playbin since some time ago, you could launch it like this: gst-launch-1.0 playbin uri=file:///usr/...

[bojidartonchev]

Both gst-play-1.0 and gst-launch-1.0 with playbin seems playing.

root@rockchip-rk3399-tinker_board_2:~# gst-launch-1.0 playbin uri=file:///usr/share/flutter/kiosk/data/flutter_assets/assets/videos/testvideo.mp4
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
error: XDG_RUNTIME_DIR not set in the environment.
WARNING: from element /GstWaylandSink:waylandsink0: Could not initialise Wayland output
Additional debug info:
../gst-plugins-bad-1.18.5/ext/wayland/gstwaylandsink.c(503): gst_wayland_sink_find_display (): /GstWaylandSink:waylandsink0:
Failed to create GstWlDisplay: 'Failed to connect to the wayland display '(default)''
error: XDG_RUNTIME_DIR not set in the environment.
Got context from element 'sink': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayEGL\)\ gldisplayegl0";
arm_release_ver of this libmali is 'r18p0-01rel0', rk_so_ver is '5'.
error: XDG_RUNTIME_DIR not set in the environment.
Redistribute latency...
error: XDG_RUNTIME_DIR not set in the environment.
error: XDG_RUNTIME_DIR not set in the environment.
error: XDG_RUNTIME_DIR not set in the environment.
Redistribute latency...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
0:00:05.2 / 0:00:30.0 (17.4 %)

Also I forgot to mention that I had to patch gl_renderer.c in order to create the context properly, I am not sure if it is connected.

diff --git a/src/gl_renderer.c b/src/gl_renderer.c
index 41cf0d3..f03c21b 100644
--- a/src/gl_renderer.c
+++ b/src/gl_renderer.c
@@ -565,9 +565,9 @@ EGLContext gl_renderer_create_context(struct gl_renderer *renderer) {
     EGLContext context;
 
     ASSERT_NOT_NULL(renderer);
-
+    static const EGLint context_attribs[] = { EGL_CONTEXT_CLIENT_VERSION, 2, EGL_NONE };
     pthread_mutex_lock(&renderer->root_context_lock);
-    context = eglCreateContext(renderer->egl_display, renderer->forced_egl_config, renderer->root_context, NULL);
+    context = eglCreateContext(renderer->egl_display, renderer->forced_egl_config, renderer->root_context, context_attribs);
     pthread_mutex_unlock(&renderer->root_context_lock);
 
     return context;

[ardera]

Okay, can you try with branch fix/mali?

Implemented the fix for the missing context attribs there, and I attempted to fix what I think is the issue with the failed delayed linking ... problem

[bojidartonchev]

With the mentioned branch, there is no failed delayed linking error. But the video is still not playing.

Now I have the following error:
plugins/gstreamer_video_player/frame.c: Video format is not supported by EGL: YU12 (modifier: 0).

Video format is
pix_fmt: yuv420p

Tried converting the video to yuv444p
plugins/gstreamer_video_player/frame.c: Video format is not supported by EGL: YU24 (modifier: 0).

I added some logs to track the problem and it seems the format->modifier is not initialized, but is expected to be DRM_FORMAT_MOD_LINEAR in the following block:

bool external_only;
for_each_format_in_frame_interface(i, format, interface) {
    if (format->format == drm_format) {
        LOG_ERROR(
            "Test video format is supported by EGL: %" DRM_FOURCC_FORMAT " (modifier: %" PRIu64 ").\n",
            DRM_FOURCC_ARGS(drm_format),
            (uint64_t) format->modifier
        );

        if(format->modifier == DRM_FORMAT_MOD_LINEAR)
        {
            external_only = format->external_only;
            goto format_supported;
        }
    }
}

And I got this:
plugins/gstreamer_video_player/frame.c: Test video format is supported by EGL: YU12 (modifier: 72057594037927935).

If I remove the modifier check, the next failing thing is buffer object allocation:
plugins/gstreamer_video_player/frame.c: Couldn't create GBM BO to copy video frame into.

Did some research and tried the following patch

diff --git a/src/plugins/gstreamer_video_player/frame.c b/src/plugins/gstreamer_video_player/frame.c
index c722a58..b31be60 100644
--- a/src/plugins/gstreamer_video_player/frame.c
+++ b/src/plugins/gstreamer_video_player/frame.c
@@ -441,7 +441,7 @@ UNUSED int dup_gst_memory_as_dmabuf(struct gbm_device *gbm_device, GstMemory *me
         return -1;
     }
 
-    bo = gbm_bo_create(gbm_device, map_info.size, 1, GBM_FORMAT_R8, GBM_BO_USE_LINEAR);
+    bo = gbm_bo_create(gbm_device, map_info.size, 1, get_pixfmt_info(PIXFMT_ARGB8888)->gbm_format, GBM_BO_USE_LINEAR);
     if (bo == NULL) {
         LOG_ERROR("Couldn't create GBM BO to copy video frame into.\n");
         goto fail_unmap_buffer;
@@ -889,7 +889,7 @@ struct video_frame *frame_new(struct frame_interface *interface, GstSample *samp
 
     bool external_only;
     for_each_format_in_frame_interface(i, format, interface) {
-        if (format->format == drm_format && format->modifier == DRM_FORMAT_MOD_LINEAR) {
+        if (format->format == drm_format) {
             external_only = format->external_only;
             goto format_supported;
         }

Now I am getting error EGL_BAD_MATCH from eglCreateImageKHR
plugins/gstreamer_video_player/frame.c: Couldn't create EGL image from video sample. eglCreateImageKHR: 12297

Anyway I was just randomly trying some stuff and I am not sure if that is the right direction, but decided to share with you.

Best regards!

[ardera]

The general problem in your case is that your EGL driver does not support EXT_image_dma_buf_import_modifiers, which provides eglQueryDmaBufFormatsEXT + another proc, that's why these messages are printed.

gl_renderer.c: Could not resolve EGL/GL symbol "eglQueryDmaBufFormatsEXT"
gl_renderer.c: Could not resolve EGL/GL symbol "eglQueryDmaBufModifiersEXT"

Those two procedures it couldn't resolve are normally used for querying what pixel formats and format modifiers EGL supports for import. If those are not there, flutter-pi is basically blind and can't really tell what buffers it can import. (It configures gstreamer to decode the video frames into one of the supported formats)

In the previous patch, I changed it so that flutter-pi in that case will fall back to a set of commonly supported pixel formats (and DRM_FORMAT_MOD_INVALID as the modifier, which basically means no explicit modifier), which weston does too when EXT_image_dma_buf_import_modifiers is not supported

But apparently even then, some of those pixel formats seem to not be supported (that's where the EGL_BAD_MATCH comes from, I think (after you fixed my two other bugs)). I don't really know what to do in this case. Manually create an empty buffer for every pixel format / modifier combination and test if importing works? There's infinitely-many combinations though. Convert to RGB and uploading using glTexImage2D? Probably slow, though seems like your GPU doesn't support dmabufs anyway, so there's at least one memory copy anyway. I could also hardcode the supported formats & modifiers, for this specific hw, though I don't know what those are. Ideally I'd really like to have a TinkerBoard myself and make it work there lol.

[bojidartonchev]

Is there a way to enable EXT_image_dma_buf_import_modifiers?

As I see, it requires EGL_KHR_image_base and EGL_EXT_image_dma_buf_import which I currently have according to kmscube log.

EGL information:
  version: "1.4 Midgard-"r18p0-01rel0""
  vendor: "ARM"
  client extensions: "EGL_EXT_client_extensions EGL_EXT_platform_base EGL_KHR_client_get_all_proc_addresses EGL_KHR_platform_gbm EGL_MESA_platform_gbm EGL_KHR_platform_wayland EGL_EXT_platform_wayland"
  display extensions: "EGL_WL_bind_wayland_display EGL_KHR_partial_update EGL_KHR_image_pixmap EGL_EXT_image_dma_buf_import EGL_KHR_config_attribs EGL_KHR_image EGL_KHR_image_base EGL_KHR_fence_sync EGL_KHR_wait_sync EGL_KHR_gl_colorspace EGL_KHR_get_all_proc_addresses EGL_IMG_context_priority EGL_ARM_pixmap_multisample_discard EGL_ARM_implicit_external_sync EGL_KHR_gl_texture_2D_image EGL_KHR_gl_renderbuffer_image EGL_KHR_create_context EGL_KHR_surfaceless_context EGL_KHR_gl_texture_cubemap_image EGL_EXT_create_context_robustness"

If not, Here is mentioned that there is way to achieve conversion using glEGLImageTargetTexture2DOES but I am not sure how to apply that in my case.

[bojidartonchev]

Quick update.

I got it working with several patches:

First I added a videoconvert to the pipeline:
static const char *default_pipeline_descr = "uridecodebin name=\"src\" ! video/x-raw ! videoconvert ! video/x-raw,format=RGBA ! appsink sync=true name=\"sink\"";

Second I added DRM_FORMAT_ABGR8888 to the fallback formats.

I had to skip adding caps to the sink, since we converted the video.

This is my patch right now on the fix/mali branch:

diff --git a/src/plugins/gstreamer_video_player/frame.c b/src/plugins/gstreamer_video_player/frame.c
index c722a58..62d2b9c 100644
--- a/src/plugins/gstreamer_video_player/frame.c
+++ b/src/plugins/gstreamer_video_player/frame.c
@@ -298,6 +298,7 @@ struct frame_interface *frame_interface_new(struct gl_renderer *renderer) {
         /// TODO: For each format, try creating a GBM bo and importing, to see if it
         ///  actually works.
         static const struct egl_modified_format fallback_formats[] = {
+            {DRM_FORMAT_ABGR8888, DRM_FORMAT_MOD_INVALID, false},
             {DRM_FORMAT_ARGB8888, DRM_FORMAT_MOD_INVALID, false},
             {DRM_FORMAT_XRGB8888, DRM_FORMAT_MOD_INVALID, false},
             {DRM_FORMAT_YUYV, DRM_FORMAT_MOD_INVALID, false},
@@ -441,7 +442,7 @@ UNUSED int dup_gst_memory_as_dmabuf(struct gbm_device *gbm_device, GstMemory *me
         return -1;
     }
 
-    bo = gbm_bo_create(gbm_device, map_info.size, 1, GBM_FORMAT_R8, GBM_BO_USE_LINEAR);
+    bo = gbm_bo_create(gbm_device, map_info.size, 1, get_pixfmt_info(PIXFMT_ARGB8888)->gbm_format, GBM_BO_USE_LINEAR);
     if (bo == NULL) {
         LOG_ERROR("Couldn't create GBM BO to copy video frame into.\n");
         goto fail_unmap_buffer;
@@ -889,7 +890,7 @@ struct video_frame *frame_new(struct frame_interface *interface, GstSample *samp
 
     bool external_only;
     for_each_format_in_frame_interface(i, format, interface) {
-        if (format->format == drm_format && format->modifier == DRM_FORMAT_MOD_LINEAR) {
+        if (format->format == drm_format) {
             external_only = format->external_only;
             goto format_supported;
         }
diff --git a/src/plugins/gstreamer_video_player/player.c b/src/plugins/gstreamer_video_player/player.c
index 220e005..7a96cff 100644
--- a/src/plugins/gstreamer_video_player/player.c
+++ b/src/plugins/gstreamer_video_player/player.c
@@ -850,7 +850,7 @@ static int init(struct gstplayer *player, bool force_sw_decoders) {
     GError *error = NULL;
     int ok;
 
-    static const char *default_pipeline_descr = "uridecodebin name=\"src\" ! video/x-raw ! appsink sync=true name=\"sink\"";
+    static const char *default_pipeline_descr = "uridecodebin name=\"src\" ! video/x-raw ! videoconvert ! video/x-raw,format=RGBA ! appsink sync=true name=\"sink\"";
 
     const char *pipeline_descr;
     if (player->pipeline_description != NULL) {
@@ -916,7 +916,7 @@ static int init(struct gstplayer *player, bool force_sw_decoders) {
 
     // configure our caps
     // we only accept video formats that we can actually upload to EGL
-    if (frame_interface_get_n_formats(player->frame_interface) > 0) {
+    /*if (frame_interface_get_n_formats(player->frame_interface) > 0) {
         GstCaps *caps = gst_caps_new_empty();
         for_each_format_in_frame_interface(i, format, player->frame_interface) {
             GstVideoFormat gst_format = gst_video_format_from_drm_format(format->format);
@@ -928,7 +928,7 @@ static int init(struct gstplayer *player, bool force_sw_decoders) {
         }
         gst_app_sink_set_caps(GST_APP_SINK(sink), caps);
         gst_caps_unref(caps);
-    }
+    }*/
 
     gst_app_sink_set_callbacks(
         GST_APP_SINK(sink),

Do you plan to keep that in the project? I can clean it up and integrate it properly, so I can make pull request. If not, I can keep it as a patch since I am using yocto. Anyway, thank you very much for the assistance.

[ardera]

Is there a way to enable EXT_image_dma_buf_import_modifiers?

I'm afraid not, if it's not listed in the supported extensions it's just not supported. (There's actual C/C++ code that needs to be written in the driver)

As I see, it requires EGL_KHR_image_base and EGL_EXT_image_dma_buf_import which I currently have according to kmscube log.

Those are just 2 extensions with functionality that EGL_EXT_image_..._modifiers depends on. You don't just get it for free when you have EGL_KHR_image_base and EGL_EXT_image_dma_buf_import :)

If not, Here is mentioned that there is way to achieve conversion using glEGLImageTargetTexture2DOES but I am not sure how to apply that in my case.

I'm afraid that's not applicable, the FSL suffix in the egl calls & enums in that example mean they are from some vendor extension, in this case some Freescale extension called EGL_FSL_create_image. Freescale (now NXP) is the company that makes i.MX's btw. Seems like noone supports this extension, not even Freescale anymore. Probably the dmabuf import extension that I'm using is the successor of that.

glEGLImageTargetTexture2DOES doesn't do anything special, just binds an EGLImage to a OpenGL texture, and I'm actually using that already :)

flutter-pi/src/plugins/gstreamer_video_player/frame.c

Line 1076 in d62e843

interface->glEGLImageTargetTexture2DOES(target, egl_image);

The problematic part is creating an EGL image in the first place, from a YUV DMA buffer.

First I added a videoconvert to the pipeline: static const char *default_pipeline_descr = "uridecodebin name=\"src\" ! video/x-raw ! videoconvert ! video/x-raw,format=RGBA ! appsink sync=true name=\"sink\"";

Second I added DRM_FORMAT_ABGR8888 to the fallback formats.

Nice 👍 But that conversion from YUV to RGB is pretty expensive and exactly what I am trying to avoid in the first place. I mean it works, so it's good to have it as a fallback. But ideally this conversion would only happen when there's really no other way around.

I have an idea though on how to fix it. What I said previously:

Manually create an empty buffer for every pixel format / modifier combination and test if importing works? There's infinitely-many combinations though.

is not really correct, you don't need to test every pixel format / modifier combination, you just need to test every pixel format, of which there aren't thaat many. So that could work.

[kekko7072]

@ardera i'm facing what i think is the same issue with a Raspberry Pi 4 and gstreamer_video_plaier. I installed the main of flutter-pi and all the additional requirements but on time of running i have the issue

error: XDG_RUNTIME_DIR not set in the environment.

Should i use this branch for flutter-pi https://github.com/ardera/flutter-pi/tree/fix/mali ?

EDIT
I solved using fix/mali branch, why it's not on master? any additional information i should know before using in production.

[ardera]

@kekko7072 What's the concrete issue in your case? App crashing? In that case, feel free to open a new issue :) (it's probably not related to this one since you're on Raspberry Pi) If it's just this message being printed, it's probably fine. I'd guess it's coming from libinput / libudev somehow.

fix/mali contains some small fixes that are only necessary for ARM Mali GPUs. There's nothing there relevant for Raspberry Pi 4, and it's going to be merged into master soon anyway.

@bojidartonchev I just bought a Banana Pi M2 Zero myself so I can reproduce this issue on there and maybe prevent regressions like #367 in the future. It does have a different GPU than your board (Mali 400MP2 vs Mali T860), so the driver could still behave entirely differently, but I'm hoping it still works similar enough for compatibility testing

[kekko7072]

@kekko7072 What's the concrete issue in your case? App crashing? In that case, feel free to open a new issue :) (it's probably not related to this one since you're on Raspberry Pi) If it's just this message being printed, it's probably fine. I'd guess it's coming from libinput / libudev somehow.

fix/mali contains some small fixes that are only necessary for ARM Mali GPUs. There's nothing there relevant for Raspberry Pi 4, and it's going to be merged into master soon anyway.

@bojidartonchev I just bought a Banana Pi M2 Zero myself so I can reproduce this issue on there and maybe prevent regressions like #367 in the future. It does have a different GPU than your board (Mali 400MP2 vs Mali T860), so the driver could still behave entirely differently, but I'm hoping it still works similar enough for compatibility testing

Yes it was not an issue due to this but due to the --arch=arm64 missing in build phase. Now i'm facing an issue with video looping thats stops. ISSUE