VMAF and AV1's film grain synthesis

[veikk0]

When it comes to the film grain synthesis feature of AV1, I'm under the impression that this synthetic grain should be disabled at decode-time (-export_side_data film_grain in FFmpeg) when calculating objective metrics, because the grain synthesis tool only creates an approximation of the film grain and doesn't reproduce it in an exact manner. And therefore objective metrics (or at least current ones) don't capture the subjective quality effect of synthesized grain. In the paper An Overview of Coding Tools in AV1: the First Video Codec from the Alliance for Open Media
, the authors state:

For grainy content, film grain synthesis significantly reduces the bitrate necessary to reconstruct the grain (up to 50% bitrate savings can be found on sequences with heavy grain or noise). This tool is not used in the comparison in section 3 since it does not generally improve objective quality metrics because of the mismatch in positions of individual grains.

The authors of this paper used PSNR for their evaluation. VMAF correlates much better with subjective quality, but in my (non-exhaustive) testing, synthesized grain does still interfere with VMAF scores. And the stronger the grain, the more significant the effect.

I also recall that disabling grain synth has been mentioned in at least one of the video presentations that have been made about AV1 & VMAF over the years, but I don't know which one it was. So I thought I'd bring it up here, since Netflix is probably the biggest user of AV1's film grain synthesis.

[nilfm]

Hi, sorry for the delayed response. You are right that synthesized grain interferes with VMAF scores, and we recommend to disable it when computing VMAF.

Regarding the 50% figure, section 6 in this paper states that the comparison was done through an informal subjective test since objective metrics don't work well in this case.

Not sure I addressed all your points, please let me know if you have other questions on this topic.

[veikk0]

Thank you very much, that answers my question.