VP9 Encoding is completely broken in Handbrake 1.7

[alikalhor77]

Problem Description

HandBrake 1.7 VP9 problem.zip

1- the result is an unusable footage
2- version 1.6.1 was fine
3- It also can't hit bitrate target in ver1.7
4- in addition to producing garbage footage it runs 2X slower than version 1.6.1
5-Shutter Encoder uses the same libvpx version of Handbrake 1.7 and it's normal and fine.
you can see VMAF scores and I upload the final clips so you can watch it for yourself.

Activity Log, Crash Log or any other details

I uploaded files instead.

What Operating System are you running?

Windows 11

What version of HandBrake are you running?

1.7

Where did you download HandBrake from?

No response

[alikalhor77]

this is frame#100 of the footages for comparison
also I should add that both VP9 and VP9_10 bit are affected.

[galad87]

A two-passes ABR uses the right bitrate, one-pass ABR seems to use a random value, on my computer the output file was 20000 Kbit/s 🧐. Nothing was changed in encavcodec.c, I wonder which of libvpx 234234234 options is the cause.
I remember a similar issue in my libaom PR.

[galad87]

Please post the activity log too.

[alikalhor77]

A two-passes ABR uses the right bitrate, one-pass ABR seems to use a random value, on my computer the output file was 20000 Kbit/s 🧐. Nothing was changed in encavcodec.c, I wonder which of libvpx 234234234 options is the cause. I remember a similar issue in my libaom PR.

I don't have the logs right now but maybe I produce one for tomorrow. clips that I shared are 2-pass encode and not 1-pass.

[alikalhor77]

A two-passes ABR uses the right bitrate, one-pass ABR seems to use a random value, on my computer the output file was 20000 Kbit/s 🧐. Nothing was changed in encavcodec.c, I wonder which of libvpx 234234234 options is the cause. I remember a similar issue in my libaom PR.

OK, I tested more and now I will share the results.
1- this time I decided to use WebM container to see if the container has an impact on this bug. the answer: it does not.
2- I tested 1-Pass and 2-Pass this time and the results are interesting. Both of them have bugs and are not functioning correctly but they behave differently.
2-Pass encoding produces garbage and unusable footage, can't reach the targeted bitrate, and is extremely slow to run.
1-Pass encoding is the exact opposite! the quality is nearly identical to the source BUT the bitrate of the final file is EXTREMELY HIGH. you talked about random numbers in your comment. I think it's not random but that is the bitrate that the encoder can produce identical results to the source. for the source I use (Extremely complicated and short test sample) the final bitrate is 127000Kbps!(25.4 times more than what it should be). This number is different for more or less complicated sources

I will upload the logs and file info files below.
logs.zip
FileInfos.zip

[alikalhor77]

I also should note that inability to reach bitrate target is not the reason why final footages of 2-Pass are garbage.
below I created 2100 Kbps (the bitrate that HB1.7 2-Pass mode reaches when you set 5000Kbps) VP9 files using Shutter Encoder. It's obvious that the problem is more complicated.

[galad87]

Another victim of my quest to avoid memory copies 😵‍💫
Can you try the version on https://github.com/galad87/HandBrake/actions/runs/6983890698 ?

[alikalhor77]

Another victim of my quest to avoid memory copies 😵‍💫 Can you try the version on https://github.com/galad87/HandBrake/actions/runs/6983890698 ?

In terms of quality it's fixed but because I installed it on Windows Sandbox I cannot comment on the speed of encoding. maybe the performance of this version needs more improvement.

[marcosfrm]

About speed, libvpx enabled more features on speed 0-2 recently:

webmproject/libvpx@03ddac4
webmproject/libvpx@660031c