Support new VA-API video postprocessing API [was: Hardware deinterlace with VA-API]

[aktau]

This is actually more of a feature request, more something which might be handy in the future after other things have been worked out:

On sandy bridge/ivy bridge/modern intel systems, the hardware can do video post processing, such as deinterlacing, freeing up the CPU to do more meaningful work. It's also a nice way of avoiding the GPU to CPU memory transfer that would otherwise be needed and just blit directly to the screen. In the OpenELEC linux distro, they've integrated this in the XMBC they use for the distro. The code is referenced here:

http://openelec.tv/forum/116-vaapi-intel/64006-vaapi-deinterlacing-testing

Just for future reference.

[ghost]

On sandy bridge/ivy bridge/modern intel systems, the hardware can do video post processing, such as deinterlacing,

We support some hardware deinterlacing, but it's pretty bad, especially compared to vdpau. I don't know if the vaapi API allows something better, but if so, I don't know about it.

It's also a nice way of avoiding the GPU to CPU memory transfer that would otherwise be needed and just blit directly to the screen.

mpv doesn't even support reading back from GPU to RAM (except for screenshots), because that would be quite inefficient, and negate any advantage of hardware decoding (so why should it).

http://openelec.tv/forum/116-vaapi-intel/64006-vaapi-deinterlacing-testing

Is there anything interesting there?

[ghost]

PS: deinterlacing is generally enabled with shift+d. Should work with -vo vaapi.

[aktau]

I don't know if the vaapi API allows something better, but if so, I don't know about it.

I believe that support for new hardware deinterlace modes was added in the newer intel-vaapi-driver (1.1) / libva (1.2) releases. Since there are very few distros that already ship with these versions, I believe it's unlikely that mpv makes use of these facilities.

PS: deinterlacing is generally enabled with shift+d. Should work with -vo vaapi.

Good to know.

If you want to see some of the code (not sure if that's the final one):

OpenELEC/OpenELEC.tv@20f7603

[ghost]

I see. Looks pretty ugly. I wonder why Intel keeps polluting the world with their bullshit instead of using vdpau. Retards. Not sure if I can be bothered to work on this. Fuck Intel.

[aktau]

I see. Looks pretty ugly. I wonder why Intel keeps polluting the world with their bullshit instead of using vdpau. Retards. Not sure if I can be bothered to work on this. Fuck Intel.

Of course not, I was just bringing it to your attention, just in case, because I know you recently did the VA-API backend for mpv (and did it quite well imho).

Having had experience with the intel mailing list, I wouldn't count on them supporting VDPAU anytime soon. What's worse, the projects that attempt to do automatic translation between these backends (kind of like Wine on Linux) are not really all that stable neither fast. (for example: https://github.com/i-rinat/libvdpau-va-gl). That said, when VA-API works, it works really well, at the very least VA-API fulfills its promise, smooth display of HD content without taxing the CPU.

[ghost]

That's fine, just my motivation to work on this is pretty low. (Not sure if I even have the hw for that.)

It enrages me to see Intel continue to extend their API (and in a pretty bad way too; the vdpau API for deinterlacing is much, much simpler, just look at the size of that xbmc patch), instead of moving towards standard interfaces. By now, VA-API is a completely vendor and hardware specific API just for Intel. VDPAU is becoming the standard, as Mesa is adding support for it, and you'll be able to use VDPAU with AMD GPUs (just not Intel GPUs, sigh...). All non-Intel VA-API backends are dead and broken, even the vdpau one.

I heard Intel "needs" VA-API in order to support features that aren't part of the VDPAU API. So why don't they just add vendor specific extensions to vdpau? Or why don't they add a VDPAU implementation to lower the load on software developers and to ease interoperability? As you know, the Linux desktop graphics stack is already suffering enough, and Intel's NIH sure doesn't help.

In any case, everyone should know that Intel and VA-API are full of crap.

for example: https://github.com/i-rinat/libvdpau-va-gl

This was my original motivation to add VA-API at all.

smooth display of HD content without taxing the CPU.

That makes sense on laptops only. Everywhere else, hardware decoding is either pointless, or uses different APIs (think of smartphones etc.).

[aktau]

Trust me, as a fellow dev, I feel your pain when you're talking about ugly, bloated, duplicated API's which don't even work exactly the same on different hardware sometimes. So yea, you're right! Having one API for hardware acceleration would be... some sort of nirvana.

That makes sense on laptops only. Everywhere else, hardware decoding is either pointless, or uses different APIs (think of smartphones etc.).

Yea, laptops and desktops, where most people run linux (that know they're running linux, i.e.: not android). Though I must say, for my purposes, even a rather recent intel celeron runs too hot and bothered trying to software decode, and let's not even speak about the atoms. So I'm glad to get the load off my systems.

[ghost]

By the way, if someone wants to write a patch, that would be accepted of course. Because it might have sounded like we'd flat out reject supporting this feature.

[bylee20]

I have implemented deinterlacer ported from CMPlayer: bylee20@f860f2c
And, I have some questions.

1. How can I activate deinterlacer?
   I have run mpv with --vo=vaapi:deint=bob, but it does not change p->deint value. This is the reason why I initialized the value at bylee20@f860f2c#L0R1093.
   I tried to press 'D' which is explained in man page, but it just activated frame-drop. What am I missing?
2. vo_vaapi does actually handle doubling framerate?
   I deleted codes at bylee20@f860f2c#L0L460, because it seems that this code just puts the same picture twice. Even if vaPutSurface handles deinterlacing, there's no pts handling like vo_vdpau which means two pictures will be displayed wihtout proper time interval.
3. Can you export VADisplay variable (p->display) to video filter chain?
   I think this functions should be impmented as a video filter such as vf_vaapi including denosise and color blanacing although I don't know how they work well. However, same display should be shared through hwdec, vf, vo. Current framework of mpv can share display between hwdec and vo only which makes implementing vf_vaapi impossible.

[ghost]

I have implemented deinterlacer ported from CMPlayer

Neat!

How can I activate deinterlacer? I have run mpv with --vo=vaapi:deint=bob, but it does not change p->deint value.

With --deinterlace, or shift+d at runtime (maps to deinterlace property).

vo_vaapi does actually handle doubling framerate?

No. It would require similar code to vdpau, which will probably get messy and complicated.

Can you export VADisplay variable (p->display) to video filter chain?

That's an interesting idea, and would probably simplify the messy logic in vdpau and simplify the implementation of vaapi deinterlacing. I think I considered moving this code to the video filter chain once, but had reservations. I can't remember what was the problem, though. If you want to try, feel free to do so. It shouldn't be hard to give the context to the filter chain somehow. (Maybe the simplest way would be making it a public member of vf_instance, or maybe as VFCTRL.)

[ghost]

Oh right, I remember why putting deinterlacing into a filter wouldn't really work with vdpau: it has a single function which does all the filtering including scaling in one go. There are no intermediate surfaces. At this point, you'd have to play tricks to coerce the vdpau filtering into a video filter chain, such as tagging vdpau mp_images with additional metadata, that tells the final renderer to apply certain filtering. I think VLC does that. (Consider sharpening: a vf_vdpau_sharpening would have to mark an mp_image as "to be sharpened", without actually changing the image data. When vo_vdpau receives this mp_image, it would inspect the metadata, and request sharpening from the vdpau video mixer.)

But maybe putting these things into the video chain is actually reasonable with vaapi.

[bylee20]

OK, maybe vf_vdpau is not so desirable(or, could be done simply by adding antoher flags or fields to mp_image?). Anyway, if I do some works, they are only for vaapi not vdpau because I have only intel GPUs and I cannot test vdpau codes. Actually, I've tried libvdpau-va-gl thing but it did not accelerate decoding at all(CPU usage's over 100% with 4 cores), so I couldn't get meaningful results about vdpau unless I buy new machine with nvidia GPUs.
I'll give it a shot to implement vf_vaapi later although I'm not sure if I'm capable of it because I'm not good at coding with C.

[bylee20]

By the way, is it allowed to change the PTS of mp_image in video filter? For instance, in order to make fps doubled in vf_vaapi, two surfaces(of course, two mp_images) should be created and their PTSes will be set by vf_vaapi. Can I expect that PTS modified by filter is valid and respected in mpv' playloop?
And, how can I know the actual rendering fps of vo to check if the fps-doubling works well?

[ghost]

Yep, that's what the pts field is for. The playloop will use the pts as reported by the VO for video timing. (For most VOs, vo.c takes care of this, but for vo_vdpau.c it's more complicated.) Note that mp_image_copy_attributes() copies the pts and some other properties, and is the standard way of making sure frame attributes are correctly handled to new frames created in filters.

And, how can I know the actual rendering fps of vo to check if the fps-doubling works well?

I don't think there's a way, short of adding debug printfs. But you could use --osd-level=3 --osd-fractions to display the video time, and use frame stepping to check whether the timestamps are produced correctly.

[i-rinat]

for example: https://github.com/i-rinat/libvdpau-va-gl

This was my original motivation to add VA-API at all.

Heh, Hate Driven Development in action. :-)

[ghost]

OK, we support the new API now. It doesn't actually support anything more than the old code right now (just field and bob deinterlacing), but at least bob deinterlacing should work more correctly with the new code. (Basically, the original mplayer-vaapi patch sucked a lot and handled this incorrectly, and I considered it too much effort to fix it, but with deinterlacing as video filter it becomes simpler again.)