Add V4L2 request support

[mark-kendall]

Current support in MythV4L2M2MContext was a haphazard guess at what might be required to get request support working.

[mark-kendall]

v4l2request.txt

[mark-kendall]

Patch attached that at least gets V4l2 request decode working:-

• requires patched FFmpeg (compilation will fail without it)
• auto detection of request API device nodes (with support for mixed 'old' style V4L2 M2M and request API usage - as is expected on the Pi 4 for HEVC (request) and H264 (current V4L2 M2M)
• tested on Tanix Tx6 (MPEG2, VP8, H264 and H265)
• plenty of issues:
• seeking is completely broken (I suspect this may be because the FFmpeg patches include a new call to flush the buffers, which we handle already)
• performance issues (I suspect too few buffers - particularly HEVC decoding)
• not clear whether direct rendering should be enabled or not (and doesn't seem to make a difference)
• consistent kernel pre-emption when exiting playback. Quite possibly the code is wrong somewhere but I'm also using a seemingly out of date version of armbian/mesa

So needs a lot of work.

[mark-kendall]

FWIW - copy back/decode only appears to work but performance is terrible and it looks like there is a conversion to YUV420P happening somewhere (it should be returning NV12)

[warpme]

Mark,

I'm wonder why using mythffmpeg at commandline with -loglevel debug i see references to v4l2_decode

mythffmpeg -loglevel debug -hwaccel drm -hwaccel_device /dev/dri/card0 -i /usr/local/share/100mb-test-1080x1920-h264.ts -pix_fmt bgra -f fbdev /dev/fb0

gives

[h264 @ 0xacd1b40] nal_unit_type: 9(AUD), nal_ref_idc: 0
[h264 @ 0xacd1b40] nal_unit_type: 6(SEI), nal_ref_idc: 0
    Last message repeated 1 times
[h264 @ 0xacd1b40] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 3
[h264 @ 0xacd1b40] ct_type:0 pic_struct:2
[h264 @ 0xacd1b40] v4l2_request_queue_decode: avctx=0xacd1b40 used=27251 controls=5 index=8 fd=23 request_fd=24 first_slice=0 last_slice=1
[NULL @ 0xac2bd90] ct_type:0 pic_struct:1
[h264 @ 0xad4ef50] nal_unit_type: 9(AUD), nal_ref_idc: 0
[h264 @ 0xad4ef50] nal_unit_type: 6(SEI), nal_ref_idc: 0
    Last message repeated 1 times

but using mythtv with log playback,libav and loglevel debug there is ffmpeg output like from sw.decode without any reference to v4l2_request:

2020-11-02 17:10:15.296371 D  [h264 @ 0000ffff8a48b1c0] nal_unit_type: 9(AUD), nal_ref_idc: 0
2020-11-02 17:10:15.296389 D  [h264 @ 0000ffff8a48b1c0] nal_unit_type: 6(SEI), nal_ref_idc: 0
2020-11-02 17:10:15.296402 D  [h264 @ 0000ffff8a48b1c0] nal_unit_type: 6(SEI), nal_ref_idc: 0
2020-11-02 17:10:15.296422 D  [h264 @ 0000ffff8a48b1c0] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 0
2020-11-02 17:10:15.296476 D  [h264 @ 0000ffff8a48b1c0] ct_type:0 pic_struct:2
2020-11-02 17:10:15.323424 D  [h264 @ 0000ffff8a48b1c0] nal_unit_type: 9(AUD), nal_ref_idc: 0
2020-11-02 17:10:15.323439 D  [h264 @ 0000ffff8a48b1c0] nal_unit_type: 6(SEI), nal_ref_idc: 0
2020-11-02 17:10:15.323450 D  [h264 @ 0000ffff8a48b1c0] nal_unit_type: 6(SEI), nal_ref_idc: 0
2020-11-02 17:10:15.323460 D  [h264 @ 0000ffff8a48b1c0] nal_unit_type: 6(SEI), nal_ref_idc: 0
2020-11-02 17:10:15.323482 D  [h264 @ 0000ffff8a48b1c0] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 0
2020-11-02 17:10:15.326766 I  AVSync: Dropping frame: Video is behind by 9730ms
2020-11-02 17:10:15.326830 I  Player(1): Waiting for video buffers...
2020-11-02 17:10:15.349839 D  [h264 @ 0000ffff8a48b1c0] nal_unit_type: 9(AUD), nal_ref_idc: 0
2020-11-02 17:10:15.349856 D  [h264 @ 0000ffff8a48b1c0] nal_unit_type: 6(SEI), nal_ref_idc: 0
2020-11-02 17:10:15.349867 D  [h264 @ 0000ffff8a48b1c0] nal_unit_type: 6(SEI), nal_ref_idc: 0
2020-11-02 17:10:15.349889 D  [h264 @ 0000ffff8a48b1c0] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 0
2020-11-02 17:10:15.349941 D  [h264 @ 0000ffff8a48b1c0] ct_type:0 pic_struct:2
2020-11-02 17:10:15.375446 D  [NULL @ 0000ffff8a48b1c0] ct_type:0 pic_struct:2
2020-11-02 17:10:15.375679 D  [NULL @ 0000ffff8a48b1c0] ct_type:0 pic_struct:1
2020-11-02 17:10:15.375890 D  [NULL @ 0000ffff8a48b1c0] ct_type:0 pic_struct:2
2020-11-02 17:10:15.375966 D  [NULL @ 0000ffff8a48b1c0] ct_type:0 pic_struct:1
2020-11-02 17:10:15.376036 D  [NULL @ 0000ffff8a48b1c0] ct_type:0 pic_struct:2
2020-11-02 17:10:15.376139 D  [NULL @ 0000ffff8a48b1c0] ct_type:0 pic_struct:1
2020-11-02 17:10:15.376918 D  [h264 @ 0000ffff8a48b1c0] nal_unit_type: 9(AUD), nal_ref_idc: 0
2020-11-02 17:10:15.376939 D  [h264 @ 0000ffff8a48b1c0] nal_unit_type: 6(SEI), nal_ref_idc: 0
2020-11-02 17:10:15.376954 D  [h264 @ 0000ffff8a48b1c0] nal_unit_type: 6(SEI), nal_ref_idc: 0
2020-11-02 17:10:15.376970 D  [h264 @ 0000ffff8a48b1c0] nal_unit_type: 6(SEI), nal_ref_idc: 0
2020-11-02 17:10:15.377017 D  [h264 @ 0000ffff8a48b1c0] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 3
2020-11-02 17:10:15.379937 I  AVSync: Dropping frame: Video is behind by 9744ms
2020-11-02 17:10:15.379993 I  Player(1): Waiting for video buffers...
2020-11-02 17:10:15.405851 D  [h264 @ 0000ffff8a48b1c0] nal_unit_type: 9(AUD), nal_ref_idc: 0
2020-11-02 17:10:15.405868 D  [h264 @ 0000ffff8a48b1c0] nal_unit_type: 6(SEI), nal_ref_idc: 0
2020-11-02 17:10:15.405883 D  [h264 @ 0000ffff8a48b1c0] nal_unit_type: 6(SEI), nal_ref_idc: 0
2020-11-02 17:10:15.405910 D  [h264 @ 0000ffff8a48b1c0] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 3
2020-11-02 17:10:15.405985 D  [h264 @ 0000ffff8a48b1c0] ct_type:0 pic_struct:2
2020-11-02 17:10:15.424627 D  [h264 @ 0000ffff8a48b1c0] nal_unit_type: 9(AUD), nal_ref_idc: 0
2020-11-02 17:10:15.424646 D  [h264 @ 0000ffff8a48b1c0] nal_unit_type: 6(SEI), nal_ref_idc: 0
2020-11-02 17:10:15.424658 D  [h264 @ 0000ffff8a48b1c0] nal_unit_type: 6(SEI), nal_ref_idc: 0
2020-11-02 17:10:15.424670 D  [h264 @ 0000ffff8a48b1c0] nal_unit_type: 6(SEI), nal_ref_idc: 0
2020-11-02 17:10:15.424702 D  [h264 @ 0000ffff8a48b1c0] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 3
2020-11-02 17:10:15.427848 I  AVSync: Dropping frame: Video is behind by 9751ms
2020-11-02 17:10:15.427924 I  Player(1): Waiting for video buffers...

above is from video playback profile "v4l2 decode only".

[mark-kendall]

@warpme don't use decode only - as noted earlier - something isn't correct there.

[warpme]

ehh silly me!.
initially i was trying full v4l2 and failed with fallback to sw.decode - i was thinking it is not ready yet....
But it turns it was due wrong rights to request fd (/dev/media0).
Thats why i switch to decode only (which newer worked ok for me btw).
Now - after fixing permissions - v4l2_request works. very nice!.
h264 is ok.
hvec works but is a bit jumpy -> probably scaling issue as i'm watching on hd display an 4k content.
will see later.

it is great work!

[warpme]

Mark,
unfortunatelly patch seems to be diverging from current master (not applying cleanly).
What is our plan with this?

[mark-kendall]

@warpme I'm swamped with issues and a little help would be nice. There were some changes last night - can you modify/update/regenerate the patch for the small changes required?

[warpme]

sure. l'll do and attach updated patch here :-)

[warpme]

Mark,
Here is updated v4l2_request code:

1. Consolidated FFmpeg patch providing support for H264/HEVC on Amlogic/Allwinner/Rockchip SoC
   Patch requires 5.10 kernel + RKVDEC patches i.e. from my repo. Patch uses copy of kernel uAPI - so even if kernel isn't correctly patched - FFmpeg code should compile ok (in case of such kernel mismatch v4l2_request just will not work correctly returning in myth log errors like "cant allocate memory for requestXX")
   https://github.com/warpme/minimyth2/blob/039204faaae737e411fe9bdfd12f353880c023f6/script/myth-master/mythtv/files/0151-ffmpeg-add-V4l2_request-support-5.10kernel.patch

2. MythTV configure patch to enable v4l2_request. By default v4l2_request is disabled - so it should not interfere on systems without v4l2support required/present
   https://github.com/warpme/minimyth2/blob/039204faaae737e411fe9bdfd12f353880c023f6/script/myth-master/mythtv/files/0152-configure-add-v4l2request-support.patch

3. Updated patch to enable working v4l2_request in MythTV v4l2 decoder (this is updated Your's v4l2request.txt to current MytTV master code)
   https://github.com/warpme/minimyth2/blob/039204faaae737e411fe9bdfd12f353880c023f6/script/myth-master/mythtv/files/0153-decoders-add_v4l2_request-support.patch

I think we should apply above code to MythTV FFmpeg - ideally as GIT submodule with patch-by-patch downstream commits applied over mainline 4.3.1 FFmpeg. Such approach IMHO will allow:

1. exactly track our downstream FFmpeg changes;
2. make visible code authors (Jonas, Jernej, etc);
3. provide good way to exploit our work by any other looking for working v4l2_request FFmpeg code.

If you decide to go this route - I can decompose patch 151-.... into series of single commits over upstream FFmpeg. Ideally will be do the same for our v4l2_m2m downstream changes (and maybe mux/dvbsubs as well)...
Above 3 patches are not interfering with systems not having v4l2_request hardware. I'm using it enabled on may x86_64 targets with no problems. Code is for 5.10 kernel uAPI (will be LTS) so we are quite safe in terms of stability of kernel uAPI exploited by above FFmpeg changes....

[mark-kendall]

@warpme I need to cleanup and commit a stash of code I have locally and will then revisit v4l2 support. I'm wary of adding v4l2 request code to master until we have fixed the regression in the existing (non-request) code.

[warpme]

Mark,
I updated v4l2_request patch to current master: https://github.com/warpme/minimyth2/blob/master/script/myth-master/mythtv/files/0153-decoders-add_v4l2_request-support.patch
Seems to work OK on my tvboxes.

Is there any chance to look on seeking issue with v4l2_request?

[mark-kendall]

@warpme It looks like v4l2 request is now available for pi4 HEVC. I'll check it out.

re seeking - I can't find the reference to it (it was on the raspberrypi.org forums somewhere) - but it is apparently a known issue with the FFmpeg code.

[warpme]

Mark,
Thx for looking on this. I gave test for new code. To get it working I updated configure to support conditonal you introduce in

mythtv/mythtv/libs/libmythtv/decoders/mythv4l2m2mcontext.cpp

Line 26 in 4598b8a

#ifdef USING_V4L2_REQUEST

Updated configure patch: https://github.com/warpme/minimyth2/blob/7e01083817b01411e2134e1855fd9d5b11f05fda/script/myth-master/mythtv/files/0152-configure-add-v4l2request-support.patch

All works perfectly (except known seek issue).

Some thoughts from my side: As You mention seek issue is ffmpeg issue. So I asked KODI friends with following Q:

when i'm switching in player from ffmpeg sw.decode to v4l2_request: are there any extra needs/changes in player calling to ffmpeg api regarding seek? (i'm asking about purely seek area). Context: we have perfectly working seek in sw. decode - but seek on v4l2_request gives us looped playback.

Answer: nope

So: as we (mythtv) have perfectly working seek on sw. decode (and on vaapi/vdpau/nvdec as well) - for non working seek on v4l2_req on mythtv i see following possible logical answers:

• a\ v4l2_request needs different/extra seek cooper. with ffmpeg when compared with sw. decode;
• b\ KODI uses ffmpeg api for seek differently than mythtv (and KODI calling way works ok for v4l2_request);

Accordingly to KODI devs a\ is false to it looks like b\ is a case.
As mythtv ffmpeg v4l2_request code is exactly KODI code - it looks like calling ffmpeg api by mythtv needs something extra for v4l2_req seek to work.

FYI: Here is relevant KODI code: https://github.com/xbmc/xbmc/blob/master/xbmc/cores/VideoPlayer/DVDCodecs/Video/DVDVideoCodecDRMPRIME.cpp

btw: When You will be playing with RPI - pls note: KODI has separate patch for RPI (https://github.com/LibreELEC/LibreELEC.tv/blob/master/packages/multimedia/ffmpeg/patches/rpi/ffmpeg-001-rpi.patch)

I was told v4l2_request code for rpi and rest of state-less decoders (allwinner/rockchip) is the same. Main addition is sand fb format, 10bits, etc.
Fact that KODI has 2 different patches for v4l2_request: RPI and rest of world say something here for me (about RPI....) :-(

Also - pls note: many discussions about seek issues are about statefull decoders (v4l2_m2m) - not about stateless (v4l2_request). IIRC key issue in stateful decoders cooperating with ffmpeg is assuring all the time (start/stop/seek) a state coherency between ffmpeg state machine and decoder firmware state machine. stateless decoders are much nicer here as all logic is usually in sw. (ffmpeg).

[mark-kendall]

@warpme I think that kodi/ffmpeg patch for pi is out of date. I was referring to the new h265 kernel driver: https://github.com/raspberrypi/linux/tree/rpi-5.4.y/drivers/staging/media/rpivid

I tried testing yesterday. Add the appropriate overlay and a new /dev/video19 device appears as expected. The usual crap when trying to do anything on the pi however; patched ffmpeg doesn't detect v4l2-request support (not exactly sure why, but one reason is probably that the pi v4l2 header (videodev2.h) is still too old, despite updating the kernel). Compiled myth anyway as it should still tell me whether it has detected any v4l2 request devices; but is linking to both broadcom and mesa opengl drivers and won't start; so threw it out of the window and am heading back to the tx6 to test.

[warpme]

Mark,
Just few thoughts from my side :-)

1. By rpi guys: 5.10 kernel would be recommended as fixes are no longer being merged to 5.4. May You try with rpi-5.10.y kernel?

2. I can play with kernel-ffmpeg part to try give you working reference - but I suspect to have all it working we need to have addressed in mythtv rpi specific drm format (called 'sand' iirc). Some examplary talk about this: https://www.raspberrypi.org/forums/viewtopic.php?t=288111

ps: Yeah - rpi is nightmare (especially with their attitude to upstream kernel & ffmpeg). This is very sad as rpi success is not in small part based on FOSS but their return to FOSS by contributing their rpi-speciffic work by upstreaming is total frustration for FOSS. I would say: it was at best "do-minimal-to-stop-foss-ranting" but with rpi4 becomes more "subtle-block-for-keeping-advantage-over-copmetition".
This is reason why i'm looking first on more cooperative embedded options (cedrus in h6 + hantro in rockchip/imx8/intel). They are supported by way more FOSS-professional ppl which are well cooperating in api changes coordination + providing clear code changes visibility instead total impossible to track code products by rpi paid contractors...

[mark-kendall]

Hardware decoding on tx6 failing. Motivation gone. Sense of humour gone. Another box thrown out of the window.

I'll come back to this when I can bear to flash another sd card...

[warpme]

Mark,
Yeah. I had such moments 4 or 5 times in past.....
For me: our subject is so technically complicated and difficult to achieve that very very few ppl are able to develop such things. In this context i think this is unique skill and already achievement
( pls see https://github.com/warpme/minimyth2#current-status ).

Regarding rpi:
imho using rpi as dev platform for v4l2_request is definitely most difficult / bumpy option. Few arguments:

1. ffmpeg required to work with rpivid decoder is hell modified 4.3.1 with many changes conflicting with changes required for rest of world (v4l2_m2m and v4l2_request). This is because development of these changes was without care about other decoders.
2. drm side of things is not better: NC12 (Y/CbCr 4:2:0 (128b cols)) and NC30 (10-bit Y/CbCr 4:2:0 (128b cols)) formats are even not yet in mainline kernel (so we need rpi version of videodev2.h)

I think worth strategy is small steep at once and starting with something well supported by good foss devs.
You mention Hardware decoding on tx6 failing.
This must be OS issue as I have current master playing perfectly on TX6 (fluent h264 and demo of HVEC).
May You pls give me some more details about context (what OS/kernel/mesa/ffmpeg patch/mythtv)?

ps: i uploaded current master build for tx6 (https://github.com/warpme/minimyth2/releases/tag/v11.12.0-v32-Pre-1976-gc9ec3de372) so you can see how well Your work already works.
ps2: Alternatively you can do on-line update of via Internet when You add in minimyth.conf MM_MINIMYTH_ONLINE_UPDATES_URL='warped.inet2.org::mm2-updates/arm64/master' and select from UI update.

[mark-kendall]

ref 9ab8e19

[mark-kendall]

ffmpeg_request_pi4.gz

Updated ffmpeg patch with small changes for raspberry pi 4 HEVC decoding (only changes are in v4l2_request.c). Decoding appears to work but presentation fails at the DMA BUF stage (i.e. DRM to EGL image). Not sure whether I've missed something from the Pi4 patches or whether the ubuntu pi kernel is too old (5.8) or there is just something else that needs to be done.

[mark-kendall]

@warpme Yup - minimyth2 working fine - seems to have improved on the tx6 - 4K HEVC @30fps is fine - 60fps not.

still not working on armbian though - kernel is 5.9 - mythtv master with your request patches (FYI - this is the armbian image you provided before - just updated)

[warpme]

Ah - this is what i was suspecting (i mean armbian thing).
If you want - I can provide you more recent armbian (with 5.10 kernel).

I feel a bit shame about this 5.9 vs. 5.10 thing - but this is because we are on bleeding edge....
Fortunately 5.10 is LTS and also cedrus (h6 vdec) is in really good shape as works on almost on vanilla 5.10 kernel - so with 5.10 i hope we will not have anymore such unpleasant creeping-api-spec issue like in 5.9->5.10.
Give me se. pls. I'll prepare Armbian and bootloader.

re 60fps@4k on tx6: i suspect it is issue of mem.bw limitation in gl driver when gpu is doing reading video buffer and writing it out in a larger form.
Isn't that out is ARGB (so 4 bytes per pixel) where the input is probably about 12 bit per pixel (nv12)? This needs data mangling and cheap tvboxes usually don't have enough mem.bw to do pixel format computation with required time regime (60fps).
I think only solution will be switch rendering to drm_prime in drm_planes mode as in this mode SoC display engine will do all format conversions + planes mixing in hw.

30fps working for you - is this on 4G tx6 variant?

btw: my friends developed support for HW deinterlaceds in h6 and rk SoCs.
DI is exposed as ffmpeg filter - so we can add this as patch for ffmpeg + mythtv code to enable ffmpeg filter. If you believe it is interesting - i can profile ffmpeg patches for this + kodi changes to exploit such di filter

re Pi4 patches: rpi ppl told me definitely we must go with rpi-5.10.y as it has huge changes / fixes in hevc and drm....
Also rpi firmware is crucial. To save your time - i would definitely build rpi-5.10.y kernel for your rpi devel. OS + update firmware to latest. I'm using kernel: https://github.com/raspberrypi/linux/commits/rpi-5.10.y firmware: https://github.com/raspberrypi/firmware/tree/master/boot

[warpme]

Mark,
Here is BullsEye 2020.11 with 5.10-rc4 kernel (most actual i was able to find): http://warped.inet2.org/armbian-2020.11.zip

Inside you have bootloader + instruction how to burn sd card
Im not able to test it with mythtv as i don't have myth builds for debian.

For 5.10 kernel You need to use mythtv ffmpeg patches for 5.10 kernel: https://github.com/warpme/minimyth2/blob/339359bbee000d40e06721d4f2aad9c85797693e/script/myth-master/mythtv/files/0151-ffmpeg-add-V4l2_request-support-5.10kernel.patch
and
https://github.com/warpme/minimyth2/blob/339359bbee000d40e06721d4f2aad9c85797693e/script/myth-master/mythtv/files/0152-configure-add-v4l2request-support.patch

This ffmpeg patches + current mythtv on 5.10 kernel should well work on h6.
Above ffmpeg + current myth + minimyth2 kernel patches are giving state like: https://github.com/warpme/minimyth2#current-status

If You will have any issues - it is possible provided Armbian 5.10 kernel needs some patching. Give me sign in such case i can provide you minimal set patches for 5.10kernel to get up and running.

ps: all this patching is save in respect upstream: my policy is to use minimal required patches from kernel ML validated by kernel maintainers....

[warpme]

Mark,
I compiled current master + https://github.com/MythTV/mythtv/files/5770000/ffmpeg_request_pi4.gz. It is on current rpi-5.10.y kernel + current rpi firmware. HVEC plays with hd.decode but screen is black. Log looks like this:

2021-01-05 18:52:12.036595 I  ScreenSaverX11: Inhibited X11 screensaver
2021-01-05 18:52:12.038378 I  PlayerVideo: Queuing callback for V4L2 request context creation
2021-01-05 18:52:12.048616 I  TV::HandleStateChange(): Main UI disabled.
2021-01-05 18:52:12.048757 I  TV::StartTV(): Entering main playback loop.
2021-01-05 18:52:12.050679 I  PlayerVideo: Executing V4L2 request context creation
2021-01-05 18:52:12.151520 N  Player(0): Waited 101ms for video buffers AAAAAA
2021-01-05 18:52:12.173348 E  EGLDMABUF: No EGLImage '12297'
2021-01-05 18:52:12.173361 I  DRMInterop: YUV composition failed. Trying separate textures.
2021-01-05 18:52:12.173439 E  EGLDMABUF: No EGLImage for plane 0 12297
2021-01-05 18:52:12.268626 E  EGLDMABUF: No EGLImage for plane 0 12297
2021-01-05 18:52:12.270916 E  EGLDMABUF: No EGLImage for plane 0 12297
2021-01-05 18:52:12.295010 E  EGLDMABUF: No EGLImage for plane 0 12297
2021-01-05 18:52:12.311731 E  EGLDMABUF: No EGLImage for plane 0 12297
2021-01-05 18:52:12.345088 E  EGLDMABUF: No EGLImage for plane 0 12297
2021-01-05 18:52:12.361736 E  EGLDMABUF: No EGLImage for plane 0 12297
2021-01-05 18:52:12.445764 E  EGLDMABUF: No EGLImage for plane 0 12297
2021-01-05 18:52:18.781872 I  PlayerFPS:   62.75 Mean: 15936 Std.Dev:  3881 CPUs: 12% 16% 16% 15% 
2021-01-05 18:52:19.766494 I  PlayerFPS:   59.94 Mean: 16683 Std.Dev:  1120 CPUs: 14% 14% 6% 17% 
2021-01-05 18:52:20.756008 I  PlayerFPS:   59.64 Mean: 16766 Std.Dev:   480 CPUs: 11% 6% 19% 11% 
2021-01-05 18:52:21.740065 I  PlayerFPS:   59.97 Mean: 16673 Std.Dev:   471 CPUs: 3% 19% 13% 9% 
2021-01-05 18:52:22.725197 I  PlayerFPS:   59.91 Mean: 16692 Std.Dev:   473 CPUs: 6% 15% 13% 19% 
2021-01-05 18:52:24.145226 I  TV::HandleStateChange(): Attempting to change from WatchingVideo to None
2021-01-05 18:52:24.145266 I  ScreenSaverX11: Uninhibited screensaver
2021-01-05 18:52:24.191427 I  OpenGLInterop: Deleted 0 textures in 8 groups
2021-01-05 18:52:24.283648 I  TV::HandleStateChange(): Changing from WatchingVideo to None
2021-01-05 18:52:24.283781 I  TV::StartTV(): Exiting main playback loop.

for me it looks like decoded frames from rpi video decoder are with format not recognised by egl. as result no picture.
arę we sure that gl (mesa) recognises and convert rpi pixel format ?

update: I tested https://github.com/MythTV/mythtv/files/5770000/ffmpeg_request_pi4.gz for any regressions. happy to report all 10 platforms playing ok :-)

[mark-kendall]

I compiled current master + https://github.com/MythTV/mythtv/files/5770000/ffmpeg_request_pi4.gz. It is on current rpi-5.10.y kernel + current rpi firmware. HVEC plays with hd.decode but screen is black. Log looks like this:
2021-01-05 18:52:12.173439 E EGLDMABUF: No EGLImage for plane 0 12297
for me it looks like decoded frames from rpi video decoder are with format not recognised by egl. as result no picture.
arę we sure that gl (mesa) recognises and convert rpi pixel format ?

Those are the same errors I'm seeing. I've taken code from popcornmix's github branch (forget which one - but latest update was only hours old at the time). I'm not going to worry about it just yet - see below.

update: I tested https://github.com/MythTV/mythtv/files/5770000/ffmpeg_request_pi4.gz for any regressions. happy to report all 10 platforms playing ok :-)

Great - thanks.

I've made a bit of a breakthrough on DRM rendering (I didn't realise Qt allowed us to access something critical) - so now focused on DRM rendering - which should be the last piece of the puzzle.

[warpme]

hmmmm..... i'm starting to feel soon we will be most advanced player in embedded world. Already kernel drm guys citing/suggesting us as most prospective playback pipeline!

[warpme]

Mark,

I compiled current master, enabled env. variables and give try.
Tested on h6/rk3399/rpi3b/i5-nuc

Unfortunately on all platforms at playback start I’m: getting „Please wait” black screen, 1sec of sound in background and then player hang.

Looking on fe logs I always see: Qt: Failed to commit atomic request (code=-22)

Pls see attached log from H6 (file has also some extra logs about kernel dmesg, drmmodes, etc)

As I have the same error on 4 different hardwares - it think issue might be:
1\ my qt build
2\ mythtv calling atomic page flip

I started to look into my qt build.
Looking in https://code.qt.io/cgit/qt/qtbase.git/tree/src/platformsupport/kmsconvenience/qkmsdevice.cpp?id=56149c0fbb19946050a3249acef4e86e511d3cd4#n823 i see my Qt has compiled ok support for atomic_drm - as to reach this warning atomic checks must be passed.
Verifying this in Qt configure log:

executing config test drm_atomic
+ cd /home/piotro/minimyth-dev/script/qt/qt5/work/main.d/qt-everywhere-src-5.15.2/config.tests/drm_atomic &&
+ cd /home/piotro/minimyth-dev/script/qt/qt5/work/main.d/qt-everywhere-src-5.15.2/config.tests/drm_atomic &&
> make[1]: Entering directory '/home/piotro/minimyth-dev/script/qt/qt5/work/main.d/qt-everywhere-src-5.15.2/c
> aarch64-minimyth-linux-gnu-g++ -c -pipe -pipe -pipe -march=armv8-a+fp+simd -O3 -flto -fPIC -w  -I. -I/home/
> aarch64-minimyth-linux-gnu-g++ -pipe -pipe -march=armv8-a+fp+simd -O3 -flto -Wl,-O1 -o drm_atomic main.o
> make[1]: Leaving directory '/home/piotro/minimyth-dev/script/qt/qt5/work/main.d/qt-everywhere-src-5.15.2/co
test config.qtbase_gui.tests.drm_atomic succeeded

Looking on Internet I found a bit similar issue: ardera/flutter-pi#86

fix seems to be like this: ardera/flutter-pi@1a49227

Looking into fix I see:

• make sure DRM properties are supported before requesting them
• verbose atomic commits and pageflips

Maybe issue is: DRM properties provided for atomic flip are unsupported on my case(hw + Kernel/DRM)?
Is it possible to add extra logging for atomic operations for future debugging cases like this one?

log from my TanixH6

update:
I get DRM log via:

echo 0xFF > /sys/module/drm/parameters/debug

(see attached file) and is there is interesting line at 3439.154434

[ 3439.154434] [drm:drm_atomic_set_mode_prop_for_crtc] [CRTC:41:crtc-0] invalid mode (ret=-22, status=H_ILLEGAL):

So maybe this something related to video mode change at just starting videoplayback?

drm.log

update2:
i disable "use separated modes" and now i have playback but screen is black. OSD works and shows v4l2 decoder and low cpu usage. Frontend log is like this:

frontend-no-video-modes.log

[mark-kendall]

@warpme - I've confirmed video mode switching isn't working on tx6.

The no video problem appears to be a file system permissions issue. I think you are only getting a partial Qt setup as the following check is failing:-

https://github.com/MythTV/mythtv/blob/master/mythtv/libs/libmythui/platforms/mythdrmdevice.cpp#L229

Does that make sense?

To be clear, this is what I get at the start of my logs - which is identical to your logs apart from the json config file - and without the format set the GUI plane completely obscures the video:-

`2021-01-19 22:50:13.458837 I [2953/2953] thread_unknown platforms/mythdrmdevice.cpp:147:SetupDRM Exporting 'QT_QPA_EGLFS_KMS_ATOMIC=1'
2021-01-19 22:50:13.458887 I [2953/2953] thread_unknown platforms/mythdrmdevice.cpp:151:SetupDRM Exporting 'QT_QPA_EGLFS_ALWAYS_SET_MODE=1'
2021-01-19 22:50:13.460074 I [2953/2953] thread_unknown platforms/mythdrmdevice.cpp:648:Authenticate /dev/dri/card0: Authenticated
2021-01-19 22:50:13.574081 I [2953/2953] thread_unknown platforms/mythdrmdevice.cpp:914:AnalysePlanes /dev/dri/card0: Found 4 planes; 4 for this CRTC
2021-01-19 22:50:13.574129 I [2953/2953] thread_unknown platforms/mythdrmdevice.cpp:988:AnalysePlanes /dev/dri/card0: Selected Plane #31 Overlay for video
2021-01-19 22:50:13.574239 I [2953/2953] thread_unknown platforms/mythdrmdevice.cpp:990:AnalysePlanes /dev/dri/card0: Supported DRM video formats: NV16,NV12,NV21,NV61,P010,P210,UYVY,VYUY,YUYV,YVYU,YU11,YU12,YU16,YV11,YV12,YV16
2021-01-19 22:50:13.574248 I [2953/2953] thread_unknown platforms/mythdrmdevice.cpp:1000:AnalysePlanes /dev/dri/card0: Selected Plane #37 Overlay for GUI
2021-01-19 22:50:13.574931 I [2953/2953] thread_unknown platforms/mythdrmdevice.cpp:235:SetupDRM Wrote eglfs_kms_config.json:
{
"device": "/dev/dri/card0",
"outputs": [ { "name": "HDMI1", "format": "argb8888" } ]
}

2021-01-19 22:50:13.574941 I [2953/2953] thread_unknown platforms/mythdrmdevice.cpp:236:SetupDRM Exporting 'QT_QPA_EGLFS_KMS_CONFIG=eglfs_kms_config.json'
2021-01-19 22:50:13.575332 I [2953/2953] thread_unknown platforms/mythdrmdevice.cpp:245:SetupDRM Exporting 'QT_QPA_EGLFS_KMS_PLANE_INDEX=2'
2021-01-19 22:50:13.575342 I [2953/2953] thread_unknown platforms/mythdrmdevice.cpp:246:SetupDRM Exporting 'QT_QPA_EGLFS_KMS_PLANES_FOR_CRTCS=41,37'
2021-01-19 22:50:13.575371 I [2953/2953] thread_unknown platforms/mythdrmdevice.cpp:256:SetupDRM Exporting 'QT_QPA_EGLFS_KMS_ZPOS=1'
2021-01-19 22:50:13.575415 I [2953/2953] thread_unknown platforms/mythdrmdevice.cpp:500:~MythDRMDevice /dev/dri/card0: Closed
`

[warpme]

Mark,

I must say: this is IMPRESSIVE!.

tested: rk3328/rk3399/h6/rpi3b/rpi4
All offering smooth HD playback with CPU < 5..7%

DRM_PRIME with DRM planes shines here: I done test on lowest tvbox i have which was practically unusable at all due v.slow memory (rk3328). Now i have really smooth HD playback on it. This makes huge amount of SoCs with poor GPU (like i.e. mali450) and slow memory nicely working frontends.

Another nice surprise: on H6 I have smooth playback of HD/4k@30 - but now also 4k@60 works smooth. With EGL DRM_PRIME - 4k30 was ok - but 4k60 was jumpy.

On rpi3 all works nicely even on mainline kernel.
RPI4 with downstream rpi kernel works perfectly fine - albeit hevc gives black screen.

Interesting is that on rpi i have quite well working seek on H264 - but rest of platforms have non-working seek.
What surprised me: while h264 seek not works on H6, I have working seek with hvec on h6...
As this is v.initial support - i'll not list issues now ;-)

Once again: oh man - this is fantastic work!

[warpme]

Mark,

Just update about v4l2 decode testing on various boxes: mythv code as of ccec025 gives me following status:
https://github.com/warpme/minimyth2#current-status

Summarizing:

• stateful v4l2_m2m: works fine on all platforms i have
• stateless v4l2_request: works fine on all platforms i have except rpi4
• DRM_PRIME via EGLDMAbuf works fine on all platforms i have
• DRM_PRIME via DRMplanes works fine on all platforms i have

So far a bit showstopers (just FYI):

• UseVIdeoModes=1 breaks DRM_PRIME via DRMplanes (error -22 at modeset atomic commit); all platforms
• seek breaks h264 playback (looped picture): this is on Allwinner/Rockchip; RPi somehow recovers

great work!

[mark-kendall]

@warpme Unfortunately I'm not seeing the same success. For example no HEVC decoding on tanix tx6 - so I can't really get a feel for how well it handles 4k. I'm still using the armbian image you supplied me - which I think was only 5.10rc4 - do you have any idea when HEVC was finalised? (the FFmpeg error complains about failing to set HEVC controls). My gut feeling is that we need some sort of header/kernel/version check if the request stuff is to go into master - otherwise I just see too many issues.

[warpme]

Mark,
Sorry for late replay!
I was adding wayland support (btw works well; but hell can't figure how to auto-start weston on system without systemd...).

re Your Q about hvec finalisation: at current mainline kernel it is in flux as v4l2_request controls are not finalised so:

1. not yet exposed as kernel uAPI;
2. there is discrepancy between various kernel's hvec decoder drivers.
3. plan to finalize is next months (at leat this date was expressed in Dec2020)

With above, to move our work forward, I managed to get ffmpeg & kernel code with common v4l2_request hvec controls for aml/aw/rk (rpi4 crap is beyond my patience atm).
ffmpeg patches we are using currently already offers working hvec decode on h6/rk3328/rk3399/s905/s912/sm1 (all hw i have to test).
But to get it working you must sync v4l2_request hvec controls also on kernel side (in cedrus/rkvdec/vdec kernel video decoders).

Im quite sure balbes150 (creator of Armbian image i provided to You) not included some of kernel v4l2_request hvec controls changes - so this explains why hvec not works for you on armbian.
This means, to move forward, we need to patch+rebuild kernel in Armbian you are using.
I have all required patches for 5.10 mainline - so effectively task is just to rebuild armbian kernel you are using.

Im not debian guy, so armbian kernel rebuild needs some learning curve for me - but if required i can play with this. (so pls advice i You need me here).
Alternatively - If you will find energy/time to play with armbian kernel recompile - I have all required patches handy and tested...

re "My gut feeling is that we need some sort of header/kernel/version check if the request stuff is to go into master - otherwise I just see too many issues."
My experience/observation from past: finalisation of h264 took 1.5y.
For hevc i'm suspecting it will be faster so probably with 3q2021 or 4q2021 we may expect kernel offers uAPI with stabilised controls, so ffmpeg guys will start consume it. This will take next let say 6mo. Effectively out-of-box hvec will be probably 1..1.5y from now.

By this I think we may consider 2 steep strategy here:

1. intermediate steep (next 12..18mo): 5.10 lts kernel + patches + ffmpeg + patches
2. target: mainline kenrel + mainline ffmpeg (12..18mo from today)

As far as i know this exact strategy is of all incarnations of OS'es used for KODI (OpenELEC, CoreELEC, LibereELEC, etc)

let me know what you think about above!

[bowlstim]

Please remove me from this list….I have no idea why I am on it

…

On 3 Feb 2021, at 21:20, Piotr Oniszczuk ***@***.***> wrote: Mark, Sorry for late replay! I was adding wayland support (btw works well; but hell can't figure how to auto-start weston on system without systemd...). re Your Q about hvec finalisation: at current mainline kernel it is in flux as v4l2_request controls are not finalised so: not yet exposed as kernel uAPI; there is discrepancy between various kernel's hvec decoder drivers. plan to finalize is next months (at leat this date was expressed in Dec2020) With above, to move our work forward, I managed to get ffmpeg & kernel code with common v4l2_request hvec controls for aml/aw/rk (rpi4 crap is beyond my patience atm). ffmpeg patches we are using currently already offers working hvec decode on h6/rk3328/rk3399/s905/s912/sm1 (all hw i have to test). But to get it working you must sync v4l2_request hvec controls also on kernel side (in cedrus/rkvdec/vdec kernel video decoders). Im quite sure balbes150 (creator of Armbian image i provided to You) not included some of kernel v4l2_request hvec controls changes - so this explains why hvec not works for you on armbian. This means, to move forward, we need to patch+rebuild kernel in Armbian you are using. I have all required patches for 5.10 mainline - so effectively task is just to rebuild armbian kernel you are using. Im not debian guy, so armbian kernel rebuild needs some learning curve for me - but if required i can play with this. (so pls advice i You need me here). Alternatively - If you will find energy/time to play with armbian kernel recompile - I have all required patches handy and tested... re "My gut feeling is that we need some sort of header/kernel/version check if the request stuff is to go into master - otherwise I just see too many issues." My experience/observation from past: finalisation of h264 took 1.5y. For hevc i'm suspecting it will be faster so probably with 3q2021 or 4q2021 we may expect kernel offers uAPI with stabilised controls, so ffmpeg guys will start consume it. This will take next let say 6mo. Effectively out-of-box hvec will be probably 1..1.5y from now. By this I think we may consider 2 steep strategy here: intermediate steep (next 12..18mo): 5.10 lts kernel + patches + ffmpeg + patches target: mainline kenrel + mainline ffmpeg (12..18mo from today) As far as i know this exact strategy is of all incarnations of OS'es used for KODI (OpenELEC, CoreELEC, LibereELEC, etc) let me know what you think about above! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#278 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC7YMRCIASTYVJ5D5PNHXUTS5G4ZXANCNFSM4THHCA7Q>.

[stuarta]

Please remove me from this list….I have no idea why I am on it
…

Use the unsubscribe link at the bottom of the email, and if you don't want more of these, edit your notification settings on github

[warpme]

Mark,

By excercise I build 5.10.12 armbian kernel + all patches required for working hevc on aw/aml/rk:

http://warped.inet2.org/armbian-kernel-5.10.12.tar.xz

For upgrade:

1. install .deb under running Armbian
2. copy boot/dtb/* to SD card boot part
3. copy boot/zImage to boot part

I done simple test with upgrading and Armbian (http://warped.inet2.org/armbian-2020.11.zip) seems to boot ok and reports 5.10.12 kernel.
I don't have myth armbian packages so can't test is anything broken/improved.
(if you have - pls provide - i can test on 5.10.12 to see isn't anything broken)

ps: this is my first debian .pkg - so i hope nothing is broken.
before upgrade - pls backup exiting armbian image as there is non-zero probability system will to boot after upgrade....

pls let me know how it goes for you :-)

edit 10.02.2021:
I prepared 5.10.14 build with H6 HW deinterlacer module enabled + some H6 4k@60 HDMI fixes:
http://warped.inet2.org/armbian-kernel-5.10.14.tar.xz

[warpme]

Mark,
I started to look (with my v.limited skills) on v4l2_request seek issue and here are some findings:
-seek issue is only on h264 content on v4l2_request.
-on mpeg2/hevc/vp8 v4l2_request seek works ok.

update:
comparing playback from bookmark (offset in content by bookmark; works ok) & seek (offset in content by seek; not works) i see following difference:

works

2021-03-09 10:23:14.918832 I AFD: SeekReset(40040, 10, do flush, do discard)
2021-03-09 10:23:14.918962 I VidOutGPU: (1): AAAAAA
2021-03-09 10:23:14.918981 I VideoBuffers::DiscardFrames(1): AAAAAA
2021-03-09 10:23:14.919012 I VideoBuffers::DiscardFrames(1): AAAAAA -- done
2021-03-09 10:23:14.919032 I AFD: SeekReset() flushing
2021-03-09 10:23:14.999497 I PlayerVideo: Queuing callback for V4L2 request context creation
2021-03-09 10:23:15.023072 I PlayerVideo: Executing V4L2 request context creation
2021-03-09 10:23:15.023085 I MythCodecContext: Create decoder callback
2021-03-09 10:23:15.023112 I MythCodecContext: Created hardware device \'drm\'
2021-03-09 10:23:15.324551 I Player(5): ClearAfterSeek(0)

failed

2021-03-09 09:45:45.430159 I AFD: SeekReset(768, 0, do flush, do discard)
2021-03-09 09:45:45.430212 I VidOutGPU: (1): APPUUU
2021-03-09 09:45:45.430256 I VideoBuffers::DiscardFrames(1): AAAUUU
2021-03-09 09:45:45.430281 I VideoBuffers::DiscardFrames(1): AAAAAA -- done
2021-03-09 09:45:45.430362 I AFD: SeekReset() flushing
2021-03-09 09:45:45.435926 I Player(0): ClearAfterSeek(0)
2021-03-09 09:45:45.435984 I Player(0): Waiting for video buffers...
2021-03-09 09:45:45.491582 I PlayerVideo: Queuing callback for V4L2 request context creation
2021-03-09 09:45:45.493956 I PlayerVideo: Executing V4L2 request context creation
2021-03-09 09:45:45.493976 I MythCodecContext: Create decoder callback
2021-03-09 09:45:45.494002 I MythCodecContext: Created hardware device \'drm\'

difference i see is:
works ok: Player(0): ClearAfterSeek(0) is AFTER V4L2 request context creation
not works: Player(0): ClearAfterSeek(0) is BEFORE V4L2 request context creation

[warpme]

I implemented change in ffmpeg code giving me nicely working seek on v4l2_request on h264 content: warpme/minimyth2@d32dd07

With this i can say:
current master has nicely working v4l2_request for: vp8/vp9/h264/hevc.
vc1 support will be added when ffmpeg devs will implement v4l2_request api for vc1.

We can close this ticket with status:
initial implementation works well.