[RFC] add video backend for Exynos based boards

[tobiasjakobi]

Hi,

at the moment it's difficult to get well performing video playback on Exynos based development boards, without the help of the GPU. Since most (or all?) available SoCs have some Mali GPU, it's impossible to get decent video playback when using an upstream kernel.

Recently some rewrite of the IPP kernel API was published:
https://www.spinics.net/lists/linux-samsung-soc/msg60657.html

This API exposes blocks on the SoC to userspace, which are capable of video format CSC and scaling. On Exynos4 SoCs this would be the FIMC, on Exynos5 some other, but similar, block is available.

This RFC introduces a new vo which uses this IPPv2 API, together with the atomic KMS API.

The necessary libdrm bits can be found here:
https://github.com/tobiasjakobi/libdrm/tree/ippv2

The vo is still very much incomplete:

• basically just one color format working atm
• OSD rendering is missing atm
• no aspect ratio handling
• IPP modules have limits with respect to buffer dimensions, these need to be checked as well
• the current code allocates a source GEM buffer object for the IPP, and then copies the content of the mp_image to the BO. I would like a solution where the decoder directly writes into the GEM bo, so that additional memory bandwidth (which is scarce on these boards) can be saved. Not sure if this is feasible, but it could also be added in the future.
• more misc stuff

I have, inspired by drm_common, tried to put all Exynos agnostic code into a new atomic KMS module.

Looking forward to some feedback. Again, this is an RFC. The aforementioned kernel patches haven't landed yet, and the userspace API in libdrm is preliminary.

I have tested the backend with my ODROID-X2, which is Exynos4412 based, with the Sintel and the Big Buck Bunny movie so far (H264, 1080p, 30fps).

A lot of Hardkernel boards should be able to benefit from this vo, in particular the X, X2, U2 and U3 from the ODROID series. But as I said, the IPPv2 API is also designed to work on Exynos5, so some of the Chromebooks as well.

[ghost]

Shouldn't modern DRM support color conversion etc. by itself? Why is there still vendor-specific code (all those exynos_ functions) in this?

There are some plans of adding atomic KMS support with DRM YUV overlay support, which would benefit RockChip.

[tobiasjakobi]

No, core DRM doesn't handle CSC at all. You can query which hardware plane supports which color formats, but you're limited to using these or performing CSC in software to get an adequate format for presentation.

In fact, the IPP rewrite was first intended to be much broader, in the sense that it should've served as a general API in DRM space to allow for abstraction of CSC, scaling and maybe bitblit operation. That was quickly aborted by the DRM maintainers.

In short: There is never going to be a core DRM API that does CSC. It's always going to be vendor-specific.

[tobiasjakobi]

For reference, here is the original approach by Marek:
https://www.spinics.net/lists/linux-samsung-soc/msg54810.html

As you can see, this was quickly NAKed by most DRM folks.

[ghost]

This doesn't fit into our architecture. We're not going to have N vendor specific VOs. Enough is enough.

The hope was that DRM would get rid of those vendor-specific Linux video API pieces of shit. (Including mmal.)

[tobiasjakobi]

OK, so what would need to happen to have a chance of getting this upstream?

[ghost]

As you can see, this was quickly NAKed by most DRM folks.

That really sounds like they're shooting themselves into their own feet. (Or whatever that idiom is.) On the other hand, there's stuff like v4l2 M2M, which is rather inappropriate in the kernel (according to some people's opinion), and all kinds of shoehorned into it.

Were other proposals into that direction shot down too, or is it just an issue with details about concrete API proposals?

I don't know what's the problem with having such an API. Video filters aren't exactly a new concept, and they've roughly stayed the same in concept for decades.

In any case, DRM does support color space conversion, by nv12 surface scanout. Rockchip and apparently vaapi support this. It requires making all the messy video conversion details (colorspaces etc.) part of the DRM API.

OK, so what would need to happen to have a chance of getting this upstream?

I definitely don't want more special VOs. But if it could be modulerized in a nice way, it would be acceptable. For example look at existing desktop API stuff in mpv. I can list the following examples of somewhat successful modularization:

• we have common UI (windowing) code for X11/win32/etc., which is separate from actual rendering
• vo_gpu's video rendering code supports both GL and Vulkan via a relatively small common API (and maybe soon D3D11)
• vo_gpu supports multiple hardware acceleration methods (which unfortunately all require special code) under a small interface (hwdec_* stuff)
• ffmpeg/mpv support multiple hwaccel methods under a relatively uniform code
• ffmpeg/mpv support video post processing for vaapi, vdpau, d3d11 as normal video filters

Now I'm not saying that exynos(whatever fit into any of this. But I could imagine that there could be an "overlay" VO, or maybe vo_drm (or vo_drm_atomic) with multiple backends for specific hardware. That kind of stuff. The idea is to reduce vendor specific "garbage" to as little code as possible, and keeping the glue code (and what can be shared) as common code.

[tobiasjakobi]

Closing this for now. I got enough feedback on IRC. Going to be back once I have come up with a better design.