-
Notifications
You must be signed in to change notification settings - Fork 1
About
We test against the reference implementations of the metrics we implement to verify correctness and speed gains. Reference metric implementations tested include:
- Butteraugli: libjxl's
butteraugli_main - CVVDP: Our fcvvdp
- IW-SSIM: A fork of Python IW-SSIM
- MS-SSIM: libvmaf's MS-SSIM filter via
ffmpeg. - SSIMULACRA2: Cloudinary's
ssimulacra2
To verify correctness, we test MOS correlation, which shows how closely a metric correlates with subjective human ratings. This allows us to verify that even if our metrics aren't identical to the originals, they still provide similar perceptual value. Our tests use mos.py via mos-correlation, on CID22. Below, we report the Spearman Rank Correlation Coefficient (SRCC) for each metric, where higher is better.
Speed testing was done on a stock Core i7-13700K with 3840x2160 source & distorted PAM images (Drive link, lossless JPEG-XL sources; run djxl <*.pam.jxl> <*.pam> to decompress).
| Metric | SRCC (reference) | SRCC (fmetrics) | Difference (%) |
|---|---|---|---|
| butteraugli (p3 i203)* | 0.7929 | 0.7863 | -0.83% |
| fcvvdp** | 0.8274 | 0.8286 | +0.15% |
| iw_ssim | n/a | 0.7925 | +0.00% |
| ms_ssim | 0.7845 | 0.8048 | +2.59% |
| ssimulacra2 | 0.8916 | 0.8910 | -0.07% |
*Note: Because Butteraugli is a smaller-is-better metric, the signs are flipped for the SRCCs reported above.
**Note: fmetrics uses the fcvvdp library (as a Zig module) with different I/O, so the underlying metric implementation is the same.
| metric | ms (reference) | ms (fmetrics) | difference (%) |
|---|---|---|---|
| butteraugli (p3 i203) | 4110 | 2010 | 104.5% faster |
| fcvvdp* | 1060 | 1060 | 0.00% |
| iw_ssim | 3020 | 228 | 1224.6% faster |
| ms_ssim** | 1110 | 106 | 947.2% faster |
| ssimulacra2 | 722 | 232 | 211.2% faster |
*Note: fmetrics uses the fcvvdp library (as a Zig module) with different I/O, so the underlying metric implementation is the same.
**Note: MS-SSIM comparison isn't fair, as libvmaf has to compute other metrics in the filterchain alongside MS-SSIM.
| metric | MB (reference) | MB (fmetrics) | difference (%) |
|---|---|---|---|
| butteraugli (p3 i203) | 2440 | 1670 | -31.56% |
| fcvvdp* | 1600 | 1600 | 0.00% |
| iw_ssim | 2660 | 551 | -79.29% |
| ms_ssim** | 841 | 376 | -55.29% |
| ssimulacra2 | 1370 | 741 | -45.91% |
*Note: fmetrics uses the fcvvdp library (as a Zig module) with different I/O, so the underlying metric implementation is the same.
**Note: MS-SSIM comparison isn't fair, as libvmaf has to compute other metrics in the filterchain alongside MS-SSIM.
Numbers last updated: 5f1ef3c