kmsgrab does not download the plane with hardware when encoding #7
Comments
|
Is libyuv really necessary? |
|
Yeah, i found out libyuv can actually meet the performance of rga on 3588. |
|
Regarding memory leaks: |
|
For the sake of comparison, screen-cast with ffmpeg (encoder branch) and gstreamer (JeffyCN): ffmpeg (CPU 259.1%):
gstreamer (CPU 49.9%):
|
|
@avafinger how do you measure the CPU consumption (exactly which command)? I have something prototype and i want to compare with your results. Are you also on a 3588? |
|
Here is the latest with commit 84466c4cb94627a4b1acd7a601e2180e63d716a1 Currently onyl libyuv conversion for RGB is working so there is some room for improvement when i make this stupid RGA work. There seems to be some alignment issue somewhere and it does not work. 2nd: i am not sure how hwdownload filter works, if it creates a copy then there are some performance left on the table there as well due to copy. If this is the case next step would be implement DRMPrime pixfmt in the encoder and directly load the kmsgrab framebuffers to Mpp without any hwdownload. But thats the second step. First i ahve make this stupid RGA work. |
|
ah nevermind, please ignore all, i found a way to bypass any type of conversion for RGB. So the best rga is the one which is not used. I will ping when things gets matured, i think there is enough for me to deal with atm :) thanks for your help. |
|
@avafinger I have utilized 0-copy when encoding and 1920x1080 @60 fps realtime encoding gives around %10~%15 Cpu load. The code is all over the place at the moment, but i want to checkpoint to see where i am. Measurement command while ffmpeg is running Output: |
|
@hbiyik , I use overall %cpu usage (peak) for the process from htop and top which is a little bit more. Using your command i get 250% CPU usage for the ffmpeg process. I can't run your command on my setup, i think i have RGB and not NV12 on my monitor 1920x1080. Looks like you are on 4K display. So i have to convert RGB to NV12, which seems to be using sw. If you have a spare 1080p display around, try with that. I don't have a 4K where i am with the board, but i can test it on the weekend. Note: I am not using your latest code! |
|
which makes sense, my output was on the local git commits where RGB + DRMPRIME is implemented in the encoder.
so +2 additional copies and 1 SW conversion really upsets the resources. In my case, there is not hwdownload because kmsgrab already outputs DRMPrime frames, and i can directly import them to MPP encoder input buffer, so no copies or conversion neither hiw hw rga or sw. I have not yet pushed those changes yet because they are too immature, but thanks for the tests, the direction seems to be correct. %12 load vs %250 No need for further testing with this i guess, i will refactor this thing several times more anyways. But i would wonder how much cpu gstreamer or ffmpeg-rk would consume for comparison. I think gstreamer would be high but ffmpeg-rk was doing something similar to what i have done with 1 addional copy if havent misunderstood its code. |
|
Here is the first preview: All i have left is to work on transcoding. Otherthan that below should be fine check out the cpu load :) %13 with 1080p 60fps casting screencast_60fps.mp4 |
|
@hbiyik I can replay it on Intel Box, but not on rk3588. Can you tell me what is your command? %CPU usage 6.9% Update: Update 2 Update 3
|
I think it is this issue: #8, i need to sped some time on it but my guess is there is something wrong with time timestamps when encoding. I also need to work on transcodin parts. It sould not work good enoguh on transcoding as of now. But caapturing with KMs and decoder id done. btw
this should be bgr0 not nv12 |
|
yes, for the mkv that's the issue. for the mp4 maybe this can help: |
|
any idea how to open hdmin to capture? getting some funny errors with ffmpeg |
|
I guess ffmpeg can not handle Multiplanar V4L2 Capture devices. Thats kindof odd... below works for my usb webcam. but does not work for video0 which is multi planar capture device. https://github.com/FFmpeg/FFmpeg/blob/be91c5c80d8c66a5a39d52c21f9094c9c455ec84/libavdevice/v4l2.c#L184 This should be V4L2_CAP_VIDEO_CAPTURE_MPLANE. See "Video Capture" capability for usb webcam See "Video Capture Multiplanar" capability for hdmiin |
|
And i found out a work that supports multiplanar v4l2 devices https://github.com/rigaya/FFmpeg/commits/v4l2-multiplanar @rigaya, do you have any plan to mainline those changes to ffmpeg? |
|
@hbiyik |
|
i only have a 4gb model, but isnt this problem already solved if you change the allocator in mpp to specifically use "/dev/dma_heap/system-dma32", then the allocated space will never exceed 4GB address range. rockchip-linux/mpp#356 (comment) Isnt thsi the same issue? |
I have all those workarounds + the patch to make RGA3 use dma32, but for this specific situation (i think the conversion) the limit is back. I have another kernel to test and check. Does your implementation make use of RGA3 or libyuv? I don't get the 4GB limit error. |
|
i think my implementation also should use RGA3 when:
But in any case RGA3 fails, i fallback to libyuv so it is quite transparent and effiicient almost not noticable. i think you can test with "FFMPEG_RKMPP_PIXFMT=NV12 ffplay Poltergeist_DOM_TrailerD-1080p-HDTN.mp4". I know this sepecific file is NV16. If it fails to RGA it, then it will spit a message like "RGA failed falling back to soft conversion" |
|
Ahh, i get it now. With @amazingfate workaround it uses RGA2 and not RGA3: With the workaround + patch it uses RGA3 and hit the 4GB limit. Maybe with official mpp + patch may work. |
No, as this branch is focused to support RK3588 HDMI In for rkmppenc, which makes it support v4l2 RGB, YUV420, YUV444 (all 8bit) input in ROCK 5B HDMI In, and hw resize by librga, then hw encode by mpp. Rock 5B HDMI In (tested with Ubuntu 20.04) v4l2 has some problems other than multi-planar format, it does not return properly to some v4l2 commands which ffmpeg uses (such as VIDIOC_S_INPUT, VIDIOC_G_INPUT, VIDIOC_G_PARM), causing further more errors in ffmpeg. The branch also includes work around for these, which I think is required to handle RK3588 HDMI In, but might not be appropriate for generic use. |
The main change of your branch is at file |
Actually, it's not forgotten, already there for my branch in rigaya@a91b413, I simply set "channel" and "ignore_input_error" via command line. Of course, it's effectively same with your commit. |
|
when i try to change in the input format with either -input_format or -pixel_format, kernel always spits out below error when i look at the source of kernel driver at below line i see that code can never change the pixel format, because it returns Error if it is not same with the actual format (wtf code) Were you ever able to change the input format of hdmirx device? May be instead of changing the format, there is a way to directly initialize with this specific format. edit: my kernel is at: Linux alarm 5.10.110-37-rockchip-g74457be0716d #rockchip SMP Mon Feb 6 09:18:21 UTC 2023 aarch64 GNU/Linux |
|
No, I also were not able to change input format from ffmpeg or even with v4l2-ctl. I’m actually converting each format to YUV420 in rkmppenc to be able to handle all 4 formats(RGB, YUV420, 422, 444). |
|
I see, it also look like a bug to me in the driver. BTW, at least for vepu580, it can directly accept nv24 aka YUV444SP, i am thinking directly using NV24 instead of using RGA. This will cause issues with vepu54x. I dont know which SOCs use vepu54x, but RK3588 at least uses vepu580. in my case v4l hdmmin driver always defaults to NV24 and not possible to change it afterwards, i am not sure it is related to the hdmi device attached or not. I have support for nv12, nv16, and bgr0 as input format for encoder expect one format that v4l driver is stuck with which is NV24 :). Thanks rockchip for making things needlessly harder .. |
|
Doe this work for you? I was able to switch from NV24 to BGR and back when i could get edid. v4l2-ctl -d /dev/video20 --get-edid > edid.txt |
|
this seems to be working, i am no expert but this is something like telling the transmitter on te other end of the HDMI end use this specific format i guess. (totally guess working). The moment i ask ffmpeg to use NV16 (hex,ycbcr422) it fallbacks to NV24 again (hex,ycbcr444) Is there a way to force rgb888? |
|
Do you mean this format? |
|
I know the basics... :) In my case i have only this: v4l2-ctl -d /dev/video20 --list-formats |
|
i get error if i try: HDMI in Support FMT: RGB888/YUV420/YUV422/YUV444 8bit I believe it means 24bit rgb888. |
|
@avafinger h264 encoder problem is solved at 20985fa, now the output encoded h264 files can be payed back with mpp. Hevc still has issues though, i think this might be an mpp issue. |
|
do you know if hevc encoder works in gstreamer? or somewhere else.. |
|
hevc encoder works on gstreamer |
|
I can confirm it's working for h264. |
|
i did not notice such a thing, here is the ss for a transcode and screencast in my case. Am i not seeing something? left is screen, right is recoding left is original, rigth is transcoded |
|
You can see it on "Big Buck Bunny, Sun Flower......" title. The Left (original) is white, transcoded is grey. Maybe the problem is only with the white color. Seems to be color saturation. Original: |
I think thats because the title of the player was not active, but i saw what you mean in your example
and i think thats because encoding parameters. You can change variables. I think if you inject |
|
I think specially the rc_mode could be the culprit, should default to AVBR which is not the best mode |
|
Here are the mpp parms from gstreamer without passing any parms , h264, matrox. The encoded file is worst than ffmpeg. video_kms_1920x1080_30fps_h264.mkv.zip Higher bitrate: |
|
to be honest i also dont know. But i agree for some reasons regardless of interface you use (even default mpp_enc_test) the queality is not the best. And i think thats because the QP values are in full range of 0-51 by default, which means when in dire situations, encoder can lower the quality as much as it requires. I need to play with those little bit to find a sweet spot. |
|
I have created a new issue for the coloring, at least the kmsgrab and recording stuff has been fixed, im closing this issue before it becomes another chat window :) Here is the new issue: |
test with:
./FFmpeg-encoder/ffmpeg -loglevel debug -device /dev/dri/card0 -framerate 30 -f kmsgrab -i - -vf 'hwdownload,format=bgr0' -c:v hevc_rkmpp_encoder -b:v 2000K screencast_30fps_4.mkv
The text was updated successfully, but these errors were encountered: