[Feature Request] Hardware encoders (hevc_qsv, nvenc_hevc, hevc_amf, vaapi)

[msaintauret]

will it be added ?

[cdgriffith]

Editing my top comment for newer overview:

FastFlix now supports AMD and Nvidia hardware encoding through both ffmpeg and Rigaya's encoders (which support HDR10)

Supported:
* ffmpeg nvenc_hevc, nvenc_h264 (Windows and Linux)
* VCEncC HEVC (Windows), AVC (Windows and Linux)
* NVEncC HEVC, AVC (Windows and Linux)

Not supported, but possible / accepting PRs:
* ffmpeg hevc_qsv
* Rigaya's QSV encoder
* ffmpeg amf encoders
* ffmpeg vaapi

Original response

That is not currently high priority for me to add any of the hardware accelerated options as they:

1. Are OS specific (hevc_ on windows and vaapi on linux)
2. Don't support HDR10 metadata (vaapi might support standard HDR10 sei headers? But I think that's it)
3. Default builds, even the static ones, don't come with support for it on linux

[cdgriffith]

Just another addition to this: Hardware encoding is great for real-time streams. However it is ill-advisable to use it for files, like FastFlix is designed for, due to their much lower quality vs bitrate.

Not saying it's a total waste, as some people have situations where their GPU is really only thing powerful enough to bother encoding with, just not the normal or recommended path.

[msaintauret]

yes you're probably right
i don't know very well how hevc_qsv works

[MarcoRavich]

...what about hwaccel auto ?

Check out similar discussion @ Axiom: MattMcManis/Axiom#42

[cdgriffith]

I have been getting more inquiries into this lately, and after looking at figures with the more recent nvidia options https://unrealaussies.com/tech/nvenc-x264-quicksync-qsv-vp9-av1/6/#Final-thoughts-on-Maxwell-and-Pascal-H264-AVC it seems they are not so bad if using the newest generation. I will probably start off with adding the nvenc_hevc encoder first after the 4.0 release.

[cdgriffith]

First thing that is being added currently is HEVC NVENC included in FFMpeg builds.

However I am looking to add https://github.com/rigaya/NVEnc/tree/5.25 as part of the standard include, as it provides support for HDR10 and HDR10+!

[tyeeman]

Yes, I've been using Rigaya's nvenc for many months in a batch file so a gui is very welcome. I won't use staxrip, it's just too big just to create a command line.

[balanadi]

I manage to use Fastflix to create the command line and to use it in ffmpeg with hevc_qsv using 90 % of my GPU on an INTEL(R) CORE(TM) I3-8109U CPU @ 3.00GHZ
-map 0 -c:v hevc_qsv -preset:v 1 -profile:v 0 -load_plugin hevc_hw -pix_fmt nv12 -global_quality 18 -b_strategy 1 -compression_level 7 -qcomp 0.80 -rdo 1 -g:v 250 -bf:v 3 -refs:v 5 -bitrate_limit 1 -vf "scale=1920:1080:flags=lanczos,format=nv12,yadif=2" -aspect 1.7777 -color_primaries bt709 -color_trc bt709 -colorspace bt709 -c:a aac -b:a 128k -ac 2 -ar 48000
The process takes between 30 to 1 hour , and the result is way better than using x265 slow preset which takes almost 12-to 20 hours ..
I know that not many of us are having a good intel processor to be able to use hevc_qsv , but so far i am happy with fastflix as long as i can use it .

[MarcoRavich]

1st of all: encoding with hevc_qsv is at least twice as fast on my i7-7700 CPU, so it would be really great if FF will exploit it...

I manage to use Fastflix to create the command line and to use it in ffmpeg with hevc_qsv

I'm testing too, but for 2-pass (need specific target size) AVC to HEVC 10bit...

So how to modify this FF-generated commandline (1st pass):
-y -i <inputfile> -metadata title=<title> -max_muxing_queue_size 1024 -filter_complex <filters> -map "[v]" -c:v libx265 -pix_fmt yuv420p10le -profile:v main10 -x265-params "aq-mode=2:repeat-headers=0:strong-intra-smoothing=1:bframes=4:b-adapt=2:frame-threads=0:hdr10_opt=0:hdr10=0:pass=1:no-slow-firstpass=1" -passlogfile <logfile>-b:v <bitrate> -preset medium -an -sn -dn -f mp4 NUL

in order to correctly integrate these (and maybe other) parameters ?
-hwaccel auto -i <inputfile> -c:v hevc_qsv -profile:v main10 <outputfile>

I also found some interesting resources that may help:

• https://nico-lab.net/installing_qsv_with_ffmpeg/ (Japanese, use translator)
• HEVCQSV : GPU load decreasing (due to a specific option) mdhiggins/sickbeard_mp4_automator#932

Thanks in advance to anyone that can/would help !

[cdgriffith]

@forart I don't have a intel chip to test with, but should just need to replace the existing options there and remove all the x265-params as those are encoder specific.

ffmpeg -y -hwaccel auto -i <inputfile> -metadata title=<title> -max_muxing_queue_size 1024 -filter_complex <filters> -map "[v]" -c:v hevc_qsv -pix_fmt yuv420p10le -profile:v main10 -passlogfile <logfile>-b:v <bitrate> -preset medium -an -sn -dn -f mp4 NUL

[MarcoRavich]

ffmpeg -y -hwaccel auto -i <inputfile> -metadata title=<title> -max_muxing_queue_size 1024 -filter_complex <filters> -map "[v]" -c:v hevc_qsv -pix_fmt yuv420p10le -profile:v main10 -passlogfile <logfile>-b:v <bitrate> -preset medium -an -sn -dn -f mp4 NUL

Thanks a lot @cdgriffith, but hevc_qsv works with "p010le" (instead of "yuv420p10le") as pix_fmt...

...anyway, if you check those links above, many puts "load_plugin hevc_hw" parameter also in the command line, but I haven't understand exactly what it does.

Last but not least, it would be great to use qsv filters too:

• scale_qsv
• deinterlace_qsv
• overlay_qsv
• ...

[MarcoRavich]

Just found this interesting insight @ StackExchange:

Notes on encoding with the QSV encoders:
Typically, you will need to initialize a hardware device that will be used by both the encoder(s) in use and filtering, as shown in the example below:

ffmpeg -y -loglevel debug -init_hw_device qsv=hw -filter_hw_device hw -hwaccel qsv -hwaccel_output_format qsv \
-i simpsons.mp4 -vf 'format=qsv,hwupload=extra_hw_frames=64'  \
-c:v hevc_qsv \
-bf 3 -b:v 3.75M -maxrate:v 3.75M -bufsize:v 0.5M -r:v 30 -c:a copy -f mp4 trolled.mp4

See how the hardware device for use with MFX sessions has been initialized (-init_hw_device qsv=hw) and mapped as available to filters such as hwupload (-filter_hw_device hw).

Recommended for further reading: The Advanced Video options section on FFmpeg wiki.

The example above demonstrates the use of the hevc_qsv encoders with some private options passed to it, for reference.

Another example, showing the use of the h264_qsv encoder:

ffmpeg -y -loglevel debug -init_hw_device qsv=hw -filter_hw_device hw -hwaccel qsv -hwaccel_output_format qsv \
-i simpsons.mp4 -vf 'format=qsv,hwupload=extra_hw_frames=64'  \
-c:v h264_qsv \
-bf 3 -b:v 15M -maxrate:v 15M -bufsize:v 2M -r:v 30 -c:a copy -f mp4 trolled.mp4

Take note that both examples above will use the constant bitrate control (CBR) method in MFX, as shown in the console log:
[hevc_qsv @ 0x55faf21eedc0] Using the constant bitrate (CBR) ratecontrol method

Rate control, similar to how the VAAPI implementation governs it, is driven by the parameters -b:v (target video bitrate) and the -maxrate:v (maximum video bitrate) passed to the encoder. If they are equal, CBR (constant bitrate control) is used. If maxrate is greater than target bitrate, then VBR, and in effect, look-ahead based control (if so desired) are enabled.

Observe how we call up the hwupload filter, chained with the format=qsv filter to ensure that the MFX runtime receives a supported pixel format. Failure to pass this video filter chain will result in initialization failure, with output similar to this:

[h264_qsv @ 0x560e1bda7280] Selected ratecontrol mode is unsupported
[h264_qsv @ 0x560e1bda7280] Low power mode is unsupported
[h264_qsv @ 0x560e1bda7280] Current frame rate is unsupported
[h264_qsv @ 0x560e1bda7280] Current picture structure is unsupported
[h264_qsv @ 0x560e1bda7280] Current resolution is unsupported
[h264_qsv @ 0x560e1bda7280] Current pixel format is unsupported
[h264_qsv @ 0x560e1bda7280] some encoding parameters are not supported by the QSV runtime. Please double check the input parameters.
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height

The error message may seem ambiguous at first, but it stems primarily from mapping invalid options to the underlying MFX library, such as an unsupported pixel format.

The extra argument extra_hw_frames=64 passed to hwupload has to do with the MFX runtime requiring a fixed frame pool size to be pre-allocated. Use a number suitable to your requirements. Generally, you'll need a larger number (64, or thereabouts) if using features such as look-ahead (LA-ICQ).

Hope that helps/inspires !

[cdgriffith]

First wave of hardware encoding done for NVENC with AVC and HEVC if you have https://github.com/rigaya/NVEnc/releases

Added in 4.2.0 https://github.com/cdgriffith/FastFlix/releases/tag/4.2.0

[cdgriffith]

With this announcement, I will probably just wait a few months and see if this takes off fast for both AMD and Intel that could simplify the whole ecosystem: https://www.phoronix.com/scan.php?page=news_item&px=Vulkan-Video-Provisional-Spec

[centuryx476]

With this announcement, I will probably just wait a few months and see if this takes off fast for both AMD and Intel that could simplify the whole ecosystem: https://www.phoronix.com/scan.php?page=news_item&px=Vulkan-Video-Provisional-Spec

This appears to be a high demand for this type of encoding and im pretty shock that its not already a standard.

[MarcoRavich]

OK, by metioning FF I just asked in the Intel community if is possible to take advantage of QuickSynch for the 1st of 2 passes.

Let's see if something comes out.

[FFMPEG] Hybrid HEVC encoding (QuickSynch 1st pass)

[sourpatched]

Any update on VAAPI support?

[cdgriffith]

@sourpatched I have opened a separate issue #109 for vaapi as it's a lot more re-work than other encoders. The filters will have be done specifically for that encoder which will be...fun.

However it's a great multipurpose option for linux so I think it's worth it long term.

I will be adding support for QSV encoding with rigaya's encoder and Apples Videotoolbox in the next 4.9.0 release and close this ticket then.

I am not planning in near term to add ffmpeg's amf or qsv encoders, as rigaya's have more options and closer similarities to each other so they are easier to support. (I will of course welcome PRs or more comments on those.)

[MarcoRavich]

I will be adding support for QSV encoding with rigaya's encoder and Apples Videotoolbox in the next 4.9.0 release and close this ticket then.

I am not planning in near term to add ffmpeg's amf or qsv encoders, as rigaya's have more options and closer similarities to each other so they are easier to support. (I will of course welcome PRs or more comments on those.)

Nice, thanks for the valuable work you are donating your time on.

It will be interesting to side-to-side benchmarking FF <> HB in QSV (multipass ?) encoding approach...

...but does @rigaya 's QSVEnc relays on Intel oneAPI/VPL too ?

[cdgriffith]

...but does @rigaya 's QSVEnc relays on #284 too ?

Haven't dug into the code but do see on the readme page: "This software depends on oneVPL, ffmpeg, libass, tinyxml2, dtl, clRNG, ttmath & Caption2Ass."

[rigaya]

The backend of QSVEnc has moved from MediaSDK to oneVPL from QSVEnc 6.00.

QSVEnc utilize DirectX(d3d11) on Windows and VAAPI on Linux for QSV encode (and decode) via oneVPL interface.

[MarcoRavich]

The backend of QSVEnc has moved from MediaSDK to oneVPL from QSVEnc 6.00.

QSVEnc utilize DirectX(d3d11) on Windows and VAAPI on Linux for QSV encode (and decode) via oneVPL interface.

Cool, so this means that - in terms of encoding performances - should be equal to the HandBrake implementation ?

Another question: is multipass achiveable with QSVE ?

Thanks for your work and time you're spending.

Haven't dug into the code but do see on the readme page: "This software depends on oneVPL, ffmpeg, libass, tinyxml2, dtl, clRNG, ttmath & Caption2Ass."

@cdgriffith: seems that QSVE does support quicksynch deinterlacing/filtering/etc too...

...it would be great if FF could exploit them !

[rigaya]

in terms of encoding performances - should be equal to the HandBrake implementation ?

Should be equivalent as the base is the same (= oneVPL), but it might differ, by the difference in the actual implementation, and the handling of encoding parameters might also differ between HandBrake and QSVEnc.

is multipass achiveable with QSVE ?

If you mean multipass like the one x264 or x265 does, I think QSV (via oneVPL) encoding does not have that feature.

[MarcoRavich]

If you mean multipass like the one x264 or x265 does, I think QSV (via oneVPL) encoding does not have that feature.

Yeah, (unfortunally) we know, but - if the video codec respects its own (HEVC, but AVC too) standard - it should be possible to generate compliant 1st pass .log file "analisis" to be used for x265 (or x264) second pass.

As previously reported, we 3aded a [FFMPEG] Hybrid HEVC encoding (QuickSynch 1st pass) @ Intel Communities Product Support Forums Visual Computing Media (Intel® oneAPI Video Processing Library, Intel Media SDK) and "they" claims We are working on this internally and we will get back to you soon with an update in Jan '22...

Dunno if is technically realizable, but hw-accellerated 1st pass can improve 2-pass encodings at least for 40% and even more - depending on usable hw computing power too, of course - for the 1st pass.

Thanks again for your constructive efforts.

[cdgriffith]

Finally added QSV and Mac's hardware encoding in https://github.com/cdgriffith/FastFlix/releases/tag/4.9.0

Closing this issue just as the major effort is over (with #109 tracking VAAPI). Please feel free to keep commenting here or anything thoughts on Hardware encoding things. Also please feel free to open separate issues for any individual features wanted! It's a lot easier to track a lot of single feature requests than a lot in a single issue 😄

[MarcoRavich]

@cdgriffith ...trying 4k encode but: where to select QSV encoder ? (portable version)

[cdgriffith]

@forart Need to download https://github.com/rigaya/QSVEnc/releases/tag/7.01 , extract it, and link the executable in the Setting page.

I need to also update Readme for adding this, as well as better instructions for that, thanks for reminder.

[MarcoRavich]

@cdgriffith ...why not FFMPEG-like automatic download ?

[cdgriffith]

No technical reason, just would have to detect if you have that capability somehow to bother downloading it, or just having it hidden away in Help or something to click manually.

Basically very low on the list of improvements, but would love to add simplicity and clarity wherever possible!

[MarcoRavich]

OK, found another problem: cannot set custom bitrate in GUI-QSVenc settings.

[cdgriffith]

@forart seems to work for me on HEVC, I see there is an issue with the build command for AVC that I need to fix

Edit: In the drop down make sure to select "Custom" for bitrate

[MarcoRavich]

OK, 1st FF-commandline-generated 4K - SDR - QSVenc (both audio and video) @6000kbs/1pass-VBR done.
AFR ~16fps
Acceptable quality.

Found custom setting, sorry (but what about to put in on top of list ?).

@rigaya
Now encoding the same source with same parameters in HB: ~19fps AFR...

[MarcoRavich]

Here are the two encodings (note that, in the end, both performed at the same speed):

Handbrake:

Video
ID : 1
Format : HEVC
Format/Info : High Efficiency Video Coding
Format profile : Main 10@L6@Main
Codec ID : V_MPEGH/ISO/HEVC
Width : 3.840 pixel
Height : 2.160 pixel
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 24,000 FPS
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 10 bit
Color range : Limited
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709
File size : 5.847.406.486

FastFlix:

Video
ID : 1
Format : HEVC
Format/Info : High Efficiency Video Coding
Format profile : Main 10@L5@High
Codec ID : V_MPEGH/ISO/HEVC
Width : 3.840 pixel
Height : 2.160 pixel
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 24,000 FPS
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 10 bit
Color range : Limited
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709
File size : 5.448.903.256

Quality: to my eyes FF encode have a bit more fidelity, due to better grain preservation (and it's - lightly - smaller !).

Let me know if I have to post side-by-side framerips for a visual compatison.