Is there a plan to implement amf decoder in ffmpeg?

[nyanmisaka]

Only amf encoders(h264_amf/hevc_amf) are available in ffmpeg now while there is no amf decoder to use.
AFAIK on Window VCN/UVD is accessable via dxva/d3d11. Can we gain much more trancoding speed from amf decoders instead of dxva/d3d11 hardware acceleration?

[MikhailAMD]

AMF decoders can work through DXVA2 or D3D11VA on Windows so there is no gain in this code path. But option to do encoding via Vulkan was integrated into AMF encoder in FFmpeg. This is useful for Vulkan applications and for Linux. We tried to submit AMF decoder to FFmpeg (especially Vulkan option) but nobody in FFmpeg is willing to commit the proposed patches. Don't really know why. We submitted such patch to Handbrake and it was accepted there.

[nyanmisaka]

AMF decoders can work through DXVA2 or D3D11VA on Windows so there is no gain in this code path. But option to do encoding via Vulkan was integrated into AMF encoder in FFmpeg. This is useful for Vulkan applications and for Linux. We tried to submit AMF decoder to FFmpeg (especially Vulkan option) but nobody in FFmpeg is willing to commit the proposed patches. Don't really know why. We submitted such patch to Handbrake and it was accepted there.

I also tried to submit patches to ffmpeg but it took a long time to commit them. Some players don't know the details, so they think that AMD's work in this area is not actively leading to the lack of available AMF decoders.

On the one hand, I use the D3D11 decoder and h264_amf encoder on Windows. On the other hand, I use the VAAPI decoder on linux and the h264_amf encoder via vulkan. However, the encoding and decoding are not completely solved by AMF. I hope your team can promote the implementation of codec software in the new year : )

[nyanmisaka]

AMF decoders can work through DXVA2 or D3D11VA on Windows so there is no gain in this code path. But option to do encoding via Vulkan was integrated into AMF encoder in FFmpeg. This is useful for Vulkan applications and for Linux. We tried to submit AMF decoder to FFmpeg (especially Vulkan option) but nobody in FFmpeg is willing to commit the proposed patches. Don't really know why. We submitted such patch to Handbrake and it was accepted there.

BTW could you provide the AMF decoder patch submitted to ffmpeg if possible? I want to compile them for use in ffmpeg myself.

[RarogCmex]

You should send its to ffmpeg branch again. Sometimes there is no crew or just forgotten, and by two or three commits its being added

[MikhailAMD]

We tried several times. You can find the patches and discussion in FFmpeg archives if you search for "Ovchinnikov".

[nyanmisaka]

We tried several times. You can find the patches and discussion in FFmpeg archives if you search for "Ovchinnikov".

I can find some patches but they seems irrelevant to AMF decoder? Perhaps the patch you submitted was obscured by other new patches. And what is the new AMF decoder called in ffmpeg?

https://patchwork.ffmpeg.org/project/ffmpeg/list/?submitter=824
https://patchwork.ffmpeg.org/project/ffmpeg/list/?submitter=559
https://patchwork.ffmpeg.org/patch/13813/

[MikhailAMD]

I will notify the developer with link to this issue. He should be able to send you the latest.

[OvchinnikovDmitrii]

@nyanmisaka Hi! a patches with a decoder dependencies (hw_context_amf) were sent earlier but not applied.They are a bit outdated and now I am updating them and plan to re-send the entire series of patches soon, including the patch with the decoder. I'll be back when there's an update.

[nyanmisaka]

@nyanmisaka Hi! a patches with a decoder dependencies (hw_context_amf) were sent earlier but not applied.They are a bit outdated and now I am updating them and plan to re-send the entire series of patches soon, including the patch with the decoder. I'll be back when there's an update.

I can't wait to apply the new AMF decoder to my project. Thank you for helping me submit a patch about fixing the HEVC profile option not long ago, although it does not seem to have been merged yet.

[RarogCmex]

Ok, I'll port AMF to Gentoo Linux distribution if all become successfull.

[RarogCmex]

Any news?

[OvchinnikovDmitrii]

Any news?

Now we are preparing and testing patches, will be published soon

[nyanmisaka]

Now we are preparing and testing patches, will be published soon

Thank you for your works! Don't forget to send the scale_amf filter as a patch too, this will be very useful ; )

[RarogCmex]

Any news?

Now we are preparing and testing patches, will be published soon

That's very good news.

[nyanmisaka]

FFmpeg 4.3 seems to be coming soon, the AMF decoder may not be able to catch up 😂

[RarogCmex]

Ffmpeg 4.3 is out.
What is current status of AMF?

[barolo]

@RarogCmex which closed components are required for this to work after this gets sorted out? I'm also on Gentoo

[nyanmisaka]

@OvchinnikovDmitrii
Hi! I spent some time these days to merge old patches containing hwcontext_amf and amfdec into ffmpeg master to test.

I found that when using the decoder, AMF_MEMORY_DX11/DX9 does not work properly, and the encoding will proceed normally only when the AMFSurface is converted to HOST. Transcoding is my main purpose, so this will greatly affect the throughput.
amfdev/FFmpeg@9668284#diff-0a2f87051af6e67fdb6752e0fdd778d3R189

In addition, the encoder only creates SurfaceFromNative when the format is D3D11/DXVA2_VLD, but these formats are not available in the FormatMap, which makes encoding impossible. Then I used some ugly hacks to solve the above problem, but it seems that hw_frames_ctx(pointer to HW accelerator (frame allocator)) has not been implemented.

At this point, I can't let decoding and encoding be done in GPU memory at the same time.

I am not an expert in ffmpeg. Could you please guide me which parts need to be changed? Perhaps it was caused by API changes in ffmpeg and AMF as these patches were last submitted two years ago.

I sincerely hope AMD can restart ffmpeg integration that has been stuck for two years. Over the years, the decoder/encoder/filter of QSV and NVENC have gone through many iterations, and their functions and types have become more and more complete, and they have even become a selling point. However, the lack of AMF decoder and scaler in ffmpeg results in the transcoding pipeline not being able to be performed completely in GPU memory. This is neither efficient nor power-saving.

Start by adding you to the list of ffmpeg maintainers. This is very helpful to speed up the merge patch.

@MikhailAMD
Hi! I see that some of your folks tried to add himself to the ffmpeg maintainer list, but he did not receive a reply. He has been idle for a year in the patchwork list, can I know if he is still doing ffmpeg related work?
[FFmpeg-devel] MAINTAINERS: add myself to the AMF section 2019-07-03 Alexander Kravchenko

[OvchinnikovDmitrii]

Hi! Decoder implementation needs some fixes, but I had a lot of high-priority tasks at this time.

but these formats are not available in the FormatMap
hw_frames_ctx(pointer to HW accelerator (frame allocator)) has not been implemented.

Accepted formats defined in .pix_fmts = ff_amf_pix_fmts, not in format_map.
In addition, we do not use custom hw_frames_ctx, only native d3d11 or dxva

During this time, I updated the patches several times, but they quickly became outdated. I can send the patch version for 19.06.
For end-to-end work on the GPU, the pipeline should look like this: dxva decoder -> amf_scaler -> amf_encoder.
Сurrent plans:
1)update patches for the new API
2)publish them
3) fix the decoder
4) add ffmpeg Vulcan context support

Start by adding you to the list of ffmpeg maintainers.

I asked about this in ffmpeg- dev , but they rejected, because of the small amount of patches published by me. I will try again after accepting the new patch pack.

[nyanmisaka]

Thank you for your reply!

Here are some of my questions:

• Should we add -hwaccel amf or -init_hw_device amf to create a context when using this fully GPU accelerated pipeline just like qsv or cuda?

• Seems that the hevc decoder only support NV12 surface now while P010 surface is supported since RX 480. Can these be easily impelemented? 10-bit is very popular in media storage.

• For software media systems, there is always a need to transcode hevc 10-bit to h264 8-bit on the fly. Can amf_scaler do that? I have verified it on vaapi_scaler(open source implementation based on uvd/vce/vcn). But the sample TranscodeHW failed to do so.

My tiny suggestions on plans:

1. update and publish patches for hwcontex_amf
2. fix h264_amf/hevc_amf decoder base on hwcontex_amf
3. fix scale_amf scaler base on hwcontex_amf (I guess. As I didn't succeed on it.)
4. add ffmpeg Vulkan context support (It is low priority in my opinion before a complete pipeline is useable.)

Please let me know once you have a new patch.

[barolo]

where are we at currently? @OvchinnikovDmitrii

[boardlord1]

Just another respectful ask for an update on this @OvchinnikovDmitrii :)

[OvchinnikovDmitrii]

Hi! Last time, the preparatory patches were rejected, so I need to reorganize the code and submit it again. As soon as I finish my current task(hopefully this month), I will prepare new patches and submit again.

[nyanmisaka]

Thanks for your work again! I have seen that patch series last year.
Would be great If we can derive an AMF device from Vulkan and OpenCL, so as we can run some computing on video surfaces.

[LuisB79]

hi!, what is the current status of your patches? @OvchinnikovDmitrii

[nyanmisaka]

FFmpeg/FFmpeg@9b13078

@OvchinnikovDmitrii Hi! It seems AMD has finally been added into the ffmpeg maintainer for AMF codes.

Any plans on the upcoming AV1 encoder and the AMF hwcontext?

[OvchinnikovDmitrii]

FFmpeg/FFmpeg@9b13078

@OvchinnikovDmitrii Hi! It seems AMD has finally been added into the ffmpeg maintainer for AMF codes.

Any plans on the upcoming AV1 encoder and the AMF hwcontext?

@nyanmisaka
Hi, AV1 encoder is planned to be added to ffmpeg immediately after the release of the new driver version.
AMF hwcontext is also in the plans, I would be glad to have your review when the corresponding patches are ready)

[Zesko]

Would two AMF H264 and H265 decodes be built in ffmpeg for many media applications. e.g Firefox, mpv etc.... now?