Artefacts when using low quantizer

[SmilingWolf]

While running some encodes at CQ 30, speed 5, Psy tune using rav1e commit 46a64c4 I noticed heavy blocking on one clip and general metrics degradation, with PSNR-HVS-M being particularly affected.

For example:

Closer inspection revealed a variety of more or less subtle artefacts, such as intensity shifting in the Y channel, in most clips encoded using the configuration from above.

Reduced clip for testing: https://mega.nz/#!FloAHajK!_OHewBCUU1D7AJvGYG3mxtWzFZNne17aC5j5bcJnRx0
Exact cmdline: rav1e -o dist.rav1e.cq30.sp5.ivf --quantizer 30 -s 5 --tune Psychovisual rav1e.sp5.cq30.psy.bugRepro.y4m

I have been unable to reproduce the issue using any other configuration so far, such as lower or higher CQ, using speed 6, or tune Psnr, at least using the clip above for testing.

Attached are two encodes, a faulty one at CQ 30 and a good one at CQ 31.
dist.encodes.zip

[shssoichiro]

Interestingly, the same bug appears at q29 s5, but on the sleeve of the man to the right instead of on the sign. It does not seem to happen at q30 s4.

[quietvoid]

I've also had some blocking artifacts appear with 2 pass 10 bit, at around 8000 kbps.
Happened at speed 6 and 2, not sure about other speeds.

In some places it was darker, large blocks. Other places it was lighter blocks. Sometimes it was blocking at the edges of in-motion objects.
It only seemed to start appearing after 10s (240 frames at 24 fps, --keyint 24) for me.

Could only recreate with one clip though, not sure why it didn't happen with the exact same scenario but another clip.

CLIs:
rav1e.exe sample.y4m --keyint 24 --threads 24 --tile-rows 8 --tile-cols 8 --bitrate 8000 --range limited --first-pass out.stats --output out.ivf
rav1e.exe sample.y4m --keyint 24 --threads 24 --tile-rows 8 --tile-cols 8 --bitrate 8000 --range limited --second-pass out.stats --output out.ivf

If it's of any use, it didn't seem to happen in my tests with a build from commit cd1c5a6

I don't think I can share the source clip, but this is part of the encode clip: https://mega.nz/#!0dU33AjS!4kpEh2eGDaJywBYD3ke3PNM0vnEXtoKITY1-F0uBbyQ

[shssoichiro]

I'll do a bisect. See you in 8 hours. 😄

[rzumer]

@shssoichiro Keep in mind the affected speed level is among those that were bumped up recently, so during a bisect it might be closer to 4 depending on the commit. Some speed levels were touched up individually as well, so it might be harder to get accurate bisect results if the cause happens to be a feature that was flipped on or off for balance.

[shssoichiro]

Indeed, a simple bisect shows 1ecbae4 as the first bad commit. It's possible this was just never triggered before, since rdo_tx_decision and reduced_tx_set couldn't be enabled at the same time prior. Going to bed now, anyone else can feel free to pick up on this.

[SmilingWolf]

Applied 1ecbae4 on top of various "interesting" commits. Turned out the bug starts manifesting right after ea71393.

As countercheck I reverted ea71393 from the top of current master, commit
0282b20, using git apply -R ../ea71393.diff and the blocking artefacts are gone and metrics are looking regular again.

I'll stress test this build with encodes at even lower Qs and report back.

@KyleSiefring is it possible this commit is the source of the bug, or could it be bringing out another that has gone undetected for longer?

EDIT: testing with CQ 25 and 29 gives artefacts using the supposedly "good" build (0282b20 without ea71393), and a PSNR inspection shows that CQ 30 is lower quality than CQ 31. In fact, I get this same results with builds going all the way to to at least 2db00b4 (after applying 1ecbae4).
Further testing will proceed using CQ 29, which seems to makes the problem more obvious at visual inspection in a broader range of cases.

EDIT2: reproduced with CQ 29 using commit 082f8e8. Testing further is going to be trickier because of possible bitstream corruptions.

[SmilingWolf]

0fa5464 + PR #1694 gives an encode with artifacts.
e0d9e78 with no other patches, tested at speed 3, 4 and 5 at a few different CQs between 25 and 35 doesn't seem affected.

EDIT: bisected to cc7c5f3. Reproduced using both CQ 25 and 29, speed 5, Psy tune. Using the same configuration and a build from 1ee1e81 yelds a good encode.

[barrbrain]

@SmilingWolf Can you confirm whether PR #1824 eliminates the encoded artifacts?