rga3: fix rga3 uncompact detection

[hbiyik]

RGA3 has uncompact mode to convert/scale to plane format P010 which is very well accepted HDR format.

There is a bug in vendor kernel to pass this (uncompact 10bit) configuration to the rga3 hardware registers. This patch fixes the problem.

Tested by me and @nyanmisaka.

Also it is better to merge this to rkr3.4 branch where there is an auto build system of debos kernel, do you need another PR for it?

Details are here: airockchip/librga#45

[RadxaYuntian]

Do you have a simple test case for us to check the change?

We will move rest of the product to rkr4.1 soon (currently only rk356x products use it) so no need for PR on rkr3.4.

[nyanmisaka]

# Build my jellyfin-ffmpeg branch
sudo apt install -y librockchip-mpp-dev librga-dev libdrm-dev pkg-config build-essential
cd ~
git clone -b next-rockchip --depth=1 https://github.com/nyanmisaka/jellyfin-ffmpeg ffmpeg
cd ffmpeg
./configure --disable-doc --disable-ffplay --enable-static --enable-gpl --enable-version3 --enable-libdrm --enable-rkmpp --enable-rkrga
make -j $(nproc)

# You may need to run as root user to access MPP devices
# Run ffmpeg to use RGA to convert 10-bit YUV to 8-bit YUV and write one frame to PNG file.
./ffmpeg -init_hw_device rkmpp=hw:/dev/dri/card0 -filter_hw_device hw \
-f lavfi -i testsrc=s=1920x1080,format=p010 \
-vf hwupload,scale_rkrga=format=nv12,hwmap=mode=read,format=nv12 \
-vframes 1 -f image2 -y /tmp/img.png

# Check the color of the `/tmp/img.png` image. It should looks awful.

# Apply the patch to librga and make install to somewhere.
LD_LIBRARY_PATH=/path/to/the/patched/librga/lib/dir

# Run the ffmpeg cmd again and check the image.

edit: note that rga3 is rk3588 only.

For reference this is how is the correct image looks like.

[hbiyik]

@RadxaYuntian is there anything we can do to merge this? It is important to have this fix to get 10bit support from the vpu.

[RadxaYuntian]

@peterwang2050 please take a look at this.

[great9]

@RadxaStephen

[RadxaStephen]

Added it to branch linux-5.10-gen-rkr4.1.

[RadxaYuntian]

rkr3.4 was affected as well since we did not upgrade the kernel version.