-
Notifications
You must be signed in to change notification settings - Fork 1
About
The purpose of this library is faster perceptual image & video fidelity metrics. Traditional metrics like PSNR & SSIM that aren't as perceptually valuable are already very fast, due in large part to insurmountable algorithmic realities. However, better metrics don't have to be so much slower; VMAF v1: Good Is Not Good Enough from Netflix shows us that existing subjectively-oriented metrics can become simultaneously more perceptually valuable as well as faster.
Our first foray into this domain was with fcvvdp, and due to that project's success we decided to bring gains to other metrics through similar efforts. Now, metrics like IW-SSIM (which topped the charts on the JPEG AIC-3 dataset, second only to CVVDP) are up to 12x faster than they used to be, unlocking completely new use cases without any dedicated hardware necessary.
We test against the reference implementations of the metrics we implement to verify correctness and speed gains. Reference metric implementations tested include:
- Butteraugli: libjxl's
butteraugli_main - CVVDP: Our fcvvdp
- IW-SSIM: A fork of Python IW-SSIM
- MS-SSIM: libvmaf's MS-SSIM filter via
ffmpeg. - SSIMULACRA2: Cloudinary's
ssimulacra2
To verify correctness, we test MOS correlation, which shows how closely a metric correlates with subjective human ratings. This allows us to verify that even if our metrics aren't identical to the originals, they still provide similar perceptual value. Our tests use mos.py via mos-correlation, on CID22. Below, we report the Spearman Rank Correlation Coefficient (SRCC) for each metric, where higher is better.
Speed testing was done on a stock Core i7-13700K with 3840x2160 source & distorted PAM images (Drive link, lossless JPEG-XL sources; run djxl <*.pam.jxl> <*.pam> to decompress).
| Metric | SRCC (reference) | SRCC (fmetrics) | Difference (%) |
|---|---|---|---|
| butteraugli (p3 i203)* | 0.7929 | 0.7863 | -0.83% |
| fcvvdp** | 0.8274 | 0.8286 | +0.15% |
| iw_ssim | n/a | 0.7925 | +0.00% |
| ms_ssim | 0.7845 | 0.8048 | +2.59% |
| ssimulacra2 | 0.8916 | 0.8910 | -0.07% |
*Note: Because Butteraugli is a smaller-is-better metric, the signs are flipped for the SRCCs reported above.
**Note: fmetrics uses the fcvvdp library (as a Zig module) with different I/O, so the underlying metric implementation is the same.
| Metric | ms (reference) | ms (fmetrics) | Difference (%) |
|---|---|---|---|
| butteraugli (p3 i203) | 4110 | 2010 | 104.5% faster |
| fcvvdp* | 1060 | 1060 | 0.00% |
| iw_ssim | 3020 | 228 | 1224.6% faster |
| ms_ssim** | 1110 | 106 | 947.2% faster |
| ssimulacra2 | 722 | 232 | 211.2% faster |
*Note: fmetrics uses the fcvvdp library (as a Zig module) with different I/O, so the underlying metric implementation is the same.
**Note: MS-SSIM comparison isn't fair, as libvmaf has to compute other metrics in the filterchain alongside MS-SSIM.
| Metric | MB (reference) | MB (fmetrics) | Difference (%) |
|---|---|---|---|
| butteraugli (p3 i203) | 2440 | 1670 | -31.56% |
| fcvvdp* | 1600 | 1600 | 0.00% |
| iw_ssim | 2660 | 551 | -79.29% |
| ms_ssim** | 841 | 376 | -55.29% |
| ssimulacra2 | 1370 | 741 | -45.91% |
*Note: fmetrics uses the fcvvdp library (as a Zig module) with different I/O, so the underlying metric implementation is the same.
**Note: MS-SSIM comparison isn't fair, as libvmaf has to compute other metrics in the filterchain alongside MS-SSIM.
Numbers last updated: 5f1ef3c