Quality Assessment

Introduction

FFEncoder provides an interface for Netflix's Video Multi-Method Assessment Fusion (VMAF), which can be used to assess the quality of your encode. Also included are options for the more traditional comparison methods, Structural Similarity Index (SSIM) and Peak Signal to Noise Ratio (PSNR). All assessment options are provided via libvmaf.

Resizing, cropping, and frame rate settings are handled for you automatically; the library requires that these be identical before comparison.

The machine learning model files are also provided, and the script locates them dynamically at runtime so you don't have to worry about configuring file paths.

Each assessment method comes with pros and cons, and results should be used as a guideline rather than an absolute metric. This article gives a nice high-level overview on the strengths and weaknesses of each assessment.

Parameter Set Options

The VMAF feature of FFEncoder operates within its own parameter set, meaning specific options are required to enable it, and most script parameters have no effect on its outcome. To activate, you must pass the -CompareVMAF (or its alias, -VMAF) switch parameter to the script.

Parameters

Name	Alias	Description
EnableVMAF	VMAF	Switch to enable VMAF, SSIM, and PSNR evaluations. Required
Reference	Source, InputPath, I	Reference (source) video to use during comparison
Distorted	Encode, OutputPath, O	Distorted (encoded) video to use during comparison
EnableSSIM	SSIM	Add a SSIM evaluation in addition to VMAF
EnablePSNR	PSNR	Add a PSNR evaluation in addition to VMAF
VMAFResizeKernel	VMAFKernel	Set the resizing kernel (when applicable). Default is bicubic
LogFormat	LogType, VMAFLog	Output format of the VMAF log. Default is json

-Reference, -Source, -Distorted, and -Encode are all aliases for -InputPath and -OutputPath, respectively, and were added specifically for the VMAF feature to make them more relevant and meaningful; however, you can use any combination you wish.

Enabling SSIM or PSNR

libvmaf provides an interface for calculating SSIM and PSNR, too, so you can perform every assessment in one run!

Scores are calculated using the harmonic mean rather than the default arithmetic mean; basically, the former gives a more accurate overall result by utilizing different weights rather than weighting everything the same (arithmetic mean).

To enable these, simply pass their associated switch parameters, -SSIM and -PSNR, to the script.

Log Format Options

The default log format is json, but the following options can be used as well:

• csv
• xml
• sub

Examples

Using this feature is very straightforward:

# Run a basic VMAF comparison
PS > .\FFEncoder.ps1 -Reference $HOME\Videos\source.mkv -Distorted $HOME\Videos\encode.mkv -EnableVMAF

# Run VMAF comparison with SSIM (aliases used here)
PS > .\FFEncoder.ps1 -Reference $HOME\Videos\source.mkv -Distorted $HOME\Videos\encode.mkv -VMAF -SSIM

# Change the log format to xml
PS > ./FFEncoder.ps1 -Reference ~/Videos/source.mkv -Distorted ~/Videos/encode.mkv -VMAF -LogFormat xml