Gradually I intend to add new rankings, but my priority is to keep the existing ones up to date. Below is a list of 3 upcoming updates that I intend to add to keep the existing rankings up to date:
I will also gradually change the layout of tables, so the old and new layouts of the tables will appear simultaneously for some time. In the near future I will write a little more information about what inspired me to add my new repository: Monocular Depth Estimation Rankings and 2D to 3D Video Conversion Rankings
Researchers! Please train at least one of your models on perceptual loss. I have made a special column in my rankings dedicated specifically to such models. Why models trained on perceptual loss? This is best summarised by the following quote 1:
"the model trained using color loss 𝓛Lap performs best in terms of PSNR and SSIM whereas the one trained using perceptual loss 𝓛F performs best in terms of LPIPS. We further note that the 𝓛F-trained model better recovers fine details in challenging cases, making it preferable in practice."
It can be seen from the results of two video frame interpolation models from the quote above on the Vimeo-90K triplet test set 1:
Model | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
---|---|---|---|
SoftSplat - 𝓛Lap | 36.10dB | 0.970 | 0.021 |
SoftSplat - 𝓛F | 35.48dB | 0.964 | 0.013 |
Sometimes even almost 3dB better PSNR result does not guarantee better LPIPS result, as shown by the results of two different video frame interpolation methods on the Vimeo-90K septuplet test set 2:
Model | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
---|---|---|---|
VFIT-B | 36.963dB | 0.9649 | 0.0304 |
RIFE | 34.048dB | 0.9449 | 0.0233 |
LPIPS 3 is a metric that reflects human perception much better than PSNR or SSIM, which is also evident from the results presented in the paper of the competitive perceptual metric 4:
IQA Model |
BAPPS database Frame interpolation 2AFC score ↑ |
BAPPS database Video deblurring 2AFC score ↑ |
Ding20 database Deblurring 2AFC score ↑ |
---|---|---|---|
Human | 0.686 | 0.671 | 0.843 |
LPIPS | 0.630 | 0.605 | 0.788 |
PSNR | 0.543 | 0.590 | 0.518 |
SSIM | 0.548 | 0.583 | 0.575 |
Each ranking includes only the best model for one method.
The rankings exclude all event-based models.
- 👑 RBI with real motion blur✔️: LPIPS😍 (no data)
This will be the King of all rankings. We look forward to ambitious researchers. - RBI with real motion blur✔️: PSNR😞>=28.5dB
- Adobe240 (640×352) with synthetic motion blur✖️: LPIPS😍 (no data)
- Adobe240 (640×352) with synthetic motion blur✖️: PSNR😞>=33.3dB
- Adobe240 (5:8) with synthetic motion blur✖️: LPIPS😍 (no data)
- Adobe240 (5:8) with synthetic motion blur✖️: PSNR😞>=25dB
- (to do)
- Vimeo-90K triplet: LPIPS😍(SqueezeNet)<=0.014
- Vimeo-90K triplet: LPIPS😍<=0.018
- Vimeo-90K triplet: PSNR😞>=36dB
- Vimeo-90K septuplet: LPIPS😍<=0.032
- Vimeo-90K septuplet: PSNR😞>=36dB
- Appendix 1: Rules for qualifying models for the rankings (to do)
- Appendix 2: Metrics selection for the rankings
RK | Model | PSNR ↑ {Input fr.} |
Originally announced or Training dataset |
Official repository |
Practical model |
VapourSynth |
---|---|---|---|---|---|---|
1 | BVFI |
35.43 {4} |
Adobe240 | - | - | - |
2 | BiT++ |
34.97 {3} |
Adobe240 | - | ||
3 | DeMFI-Netrb(5,3) |
34.34 {4} |
Adobe240 | - | - | |
4 | ALANET | 33.34dB 5 | August 2020 5 | - | - | |
5 | PRF4 -Large | 33.32dB 6 | February 2020 7 | - | - |
RK | Model | PSNR ↑ | Originally announced |
Official repository |
Practical model |
VapourSynth |
---|---|---|---|---|---|---|
1 | VIDUE | 28.74dB 8 | March 2023 8 | - | - | |
2 | FLAVR | 27.23dB 8 | December 2020 9 | - | - | |
3 | UTI-VFI | 26.69dB 8 | December 2020 10 | - | - | |
4 | DeMFI | 25.71dB 8 | November 2021 11 | - | - |
RK | Model | LPIPS ↓ | Originally announced |
Official repository |
Practical model |
VapourSynth |
---|---|---|---|---|---|---|
1 | CDFI w/ adaP/U | 0.008 12 | March 2021 13 | - | - | |
2 | EDSC_s-𝓛F | 0.010 13 | June 2020 14 | EDSC_s-𝓛F | - | |
3 | DRVI | 0.013 15 | August 2021 15 | - | - | - |
RK | Model | LPIPS ↓ {Input fr.} |
Training dataset |
Official repository |
Practical model |
VapourSynth |
---|---|---|---|---|---|---|
1 | EAFI-𝓛ecp |
0.012 {2} |
Vimeo-90K triplet | - | EAFI-𝓛ecp | - |
2 | UGFI 𝓛S |
0.0126 {2} |
Vimeo-90K triplet | - | UGFI 𝓛S | - |
3 | SoftSplat - 𝓛F |
0.013 {2} |
Vimeo-90K triplet | SoftSplat - 𝓛F | - | |
4 | FILM-𝓛S |
0.0132 {2} |
Vimeo-90K triplet | FILM-𝓛S | - | |
5 | EDSC_s-𝓛F |
0.016 {2} |
Vimeo-90K triplet | EDSC_s-𝓛F | - | |
6 | CtxSyn - 𝓛F |
0.017 {2} |
proprietary | - | CtxSyn - 𝓛F | - |
7 | PerVFI |
0.018 {2} |
Vimeo-90K triplet | - | PerVFI | - |
RK | Model | PSNR ↑ {Input fr.} |
Originally announced or Training dataset |
Official repository |
Practical model |
VapourSynth |
---|---|---|---|---|---|---|
1 | MA-GCSPA triplet-trained | 36.76dB 16 | March 2022 16 | - | - | |
2 | VFIformer + HRFFM ENH: |
36.69 {2} |
Vimeo-90K triplet | ENH: - |
- | - |
3 | LADDER-L |
36.65 {2} |
Vimeo-90K triplet | - | - | - |
4 | EMA-VFI | 36.64dB 17 | March 2023 17 | - | - | |
5 | DQBC-Aug | 36.57dB 18 | April 2023 18 | - | - | |
6 | TTVFI | 36.54dB 19 | July 2022 19 | - | - | |
7 | AMT-G | 36.53dB 20 | April 2023 20 | - | - | |
8 | AdaFNIO | 36.50dB 21 | November 2022 21 | - | - | |
9 | FGDCN-L | 36.46dB 22 | November 2022 22 | - | - | |
10 | UPR-Net LARGE | 36.42dB 23 | November 2022 23 | - | - | |
11 | EAFI-𝓛ecc | 36.38dB 24 | July 2022 24 | - | EAFI-𝓛ecp | - |
12 | H-VFI-Large | 36.37dB 25 | November 2022 25 | - | - | - |
13 | UGFI 𝓛1 |
36.34 {2} |
Vimeo-90K triplet | - | UGFI 𝓛S | - |
14 | SoftSplat - 𝓛Lap with ensemble | 36.28dB 26 | March 2020 1 | SoftSplat - 𝓛F | - | |
15 | NCM-Large | 36.22dB 27 | July 2022 27 | - | - | - |
16-17 | IFRNet large | 36.20dB 28 | May 2022 28 | - | - | |
16-17 | RAFT-M2M++ ENH: |
36.20 {2} |
Vimeo-90K triplet | - | - | |
18-19 | EBME-H* | 36.19dB 29 | June 2022 29 | - | - | |
18-19 | RIFE-Large |
36.19 {2} |
Vimeo-90K triplet | RIFE v4.15 | ||
20-21 | ABME | 36.18dB 30 | August 2021 30 | - | - | |
20-21 | ProBoost-Net |
36.18 {2} |
? | - | - | - |
22 | TDPNetnv w/o MRTM |
36.069 {2} |
Vimeo-90K triplet | - | TDPNet | - |
23 | FILM-𝓛1 |
36.06 {2} |
Vimeo-90K triplet | FILM-𝓛S | - |
RK | Model | LPIPS ↓ | Originally announced |
Official repository |
Practical model |
VapourSynth |
---|---|---|---|---|---|---|
1 | RIFE | 0.0233 2 | November 2020 31 | RIFE v4.15 | ||
2 | IFRNet | 0.0274 2 | May 2022 28 | - | - | |
3 | VFIT-B | 0.0304 2 | November 2021 32 | - | - | |
4 | ABME | 0.0309 2 | August 2021 30 | - | - |
RK | Model | PSNR ↑ {Input fr.} |
Originally announced or Training dataset |
Official repository |
Practical model |
VapourSynth |
---|---|---|---|---|---|---|
1 | JNMR | 37.19dB 33 | June 2022 33 | - | - | |
2 | VFIT-B | 36.96dB 32 | November 2021 32 | - | - | |
3 | VRT | 36.53dB 34 | June 2022 (VFI) 34 | - | - | |
4 | ST-MFNet | 36.507dB 35 | November 2021 36 | - | - | |
5 | MA-GCSPA septuplet-trained | 36.50dB 16 | March 2022 16 | - | - | |
6 | EDENVFI PVT(15,15) | 36.387dB 35 | July 2023 35 | - | - | - |
7 | IFRNet |
36.37 {2} |
Vimeo-90K septuplet | - | - | |
8 | RN-VFI |
36.33 {4} |
Vimeo-90K septuplet | - | - | - |
9 | FLAVR | 36.25dB 9 | December 2020 9 | - | - | |
10 | DBVI | 36.17dB 37 | October 2022 37 | - | - | |
11 | EDC | 36.14dB 33 | February 2022 38 | - | - |
Currently, the most commonly used metrics in the existing works on video frame interpolation and video deblurring are: PSNR, SSIM and LPIPS. Exactly in that order.
The main purpose of creating my rankings is to look for the best perceptually-oriented model for practical applications - hence the primary metric in my rankings will be the most common perceptual image quality metric in scientific papers: LPIPS.
At the time of writing these words, in October 2023, in relation to VFI, I have only found another perceptual image quality metric - DISTS in one paper: and also in one paper I found a bespoke VFI metric - FloLPIPS [arXiv]. Unfortunately, both of these papers omit to evaluate the best performing models based on the LPIPS metric. If, in the future, some researcher will evaluate LPIPS top-performing models using alternative, better perceptual metrics, I would of course be happy to add rankings based on those metrics.
I would like to use only one metric - LPIPS. Unfortunately still many of the best VFI and video deblurring methods are only evaluated using PSNR or PSNR and SSIM. For this reason, I will additionally present rankings based on PSNR, which will show the models that can, after perceptually-oriented training, be the best for practical applications, as well as providing a source of knowledge for building even better practical models in the future.
I have decided to completely abandon rankings based on the SSIM metric. Below are the main reasons for this decision, ranked from the most important to the less important.
-
The main reason is the following quote, which I found in a paper by researchers at Adobe Research: 26. In the quote they refer to a paper by researchers at NVIDIA: [arXiv].
We limit the evaluation herein to the PSNR metric since SSIM [57] is subject to unexpected and unintuitive results [39].
-
The second reason is, more and more papers are appearing where PSNR scores are given, but without SSIM: 35 and A model from such a paper appearing only in the PSNR-based ranking and at the same time not appearing in the SSIM-based ranking may give the misleading impression that the SSIM score is so poor that it does not exceed the ranking eligibility threshold, while there is simply no SSIM score in a paper.
-
The third reason is, that often the SSIM scores of individual models are very close to each other or identical. This is the case in the SNU-FILM Easy test, as shown in Table 3: [CVPR 2023], where as many as 6 models achieve the same score of 0.991 and as many as 5 models achieve the same score of 0.990. In the same test, PSNR makes it easier to determine the order of the ranking, with the same number of significant digits.
-
The fourth reason is that PSNR-based rankings are only ancillary when a model does not have an LPIPS score. For this reason, SSIM rankings do not add value to my repository and only reduce its readability.
-
The fifth reason is that I want to encourage researchers who want to use only two metrics in their paper to use LPIPS and PSNR instead of PSNR and SSIM.
-
The sixth reason is that the time saved by dropping the SSIM-based rankings will allow me to add new rankings based on other test data, which will be more useful and valuable.
Footnotes
-
Softmax Splatting for Video Frame Interpolation [CVPR 2020] [arXiv] ↩ ↩2 ↩3
-
Exploring Discontinuity for Video Frame Interpolation [CVPR 2023] [arXiv] ↩ ↩2 ↩3 ↩4 ↩5
-
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric [CVPR 2018] [arXiv] ↩
-
Locally Adaptive Structure and Texture Similarity for Image Quality Assessment [MM 2021] [arXiv] ↩
-
ALANET: Adaptive Latent Attention Network for Joint Video Deblurring and Interpolation [MM 2020] [arXiv] ↩ ↩2
-
Video Frame Interpolation and Enhancement via Pyramid Recurrent Framework [TIP 2020] ↩
-
Blurry Video Frame Interpolation [CVPR 2020] [arXiv] ↩
-
Joint Video Multi-Frame Interpolation and Deblurring under Unknown Exposure Time [CVPR 2023] [arXiv] ↩ ↩2 ↩3 ↩4 ↩5
-
FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation [WACV 2023] [arXiv] ↩ ↩2 ↩3
-
Video Frame Interpolation without Temporal Priors [NeurIPS 2020] [arXiv] ↩
-
DeMFI: Deep Joint Deblurring and Multi-Frame Interpolation with Flow-Guided Attentive Correlation and Recursive Boosting [ECCV 2022] [arXiv] ↩
-
AdaPool: Exponential Adaptive Pooling for Information-Retaining Downsampling [TIP 2022] [arXiv] ↩
-
CDFI: Compression-Driven Network Design for Frame Interpolation [CVPR 2021] [arXiv] ↩ ↩2
-
Multiple Video Frame Interpolation via Enhanced Deformable Separable Convolution [TPAMI 2021] [arXiv] ↩
-
DRVI: Dual Refinement for Video Interpolation [Access 2021] ↩ ↩2
-
Exploring Motion Ambiguity and Alignment for High-Quality Video Frame Interpolation [CVPR 2023] [arXiv] ↩ ↩2 ↩3 ↩4
-
Extracting Motion and Appearance via Inter-Frame Attention for Efficient Video Frame Interpolation [CVPR 2023] [arXiv] ↩ ↩2
-
Video Frame Interpolation with Densely Queried Bilateral Correlation [IJCAI 2023] [arXiv] ↩ ↩2
-
TTVFI: Learning Trajectory-Aware Transformer for Video Frame Interpolation [TIP 2023] [arXiv] ↩ ↩2
-
AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolation [CVPR 2023] [arXiv] ↩ ↩2
-
AdaFNIO: Adaptive Fourier Neural Interpolation Operator for video frame interpolation [arXiv] ↩ ↩2
-
Flow Guidance Deformable Compensation Network for Video Frame Interpolation [TMM 2023] [arXiv] ↩ ↩2
-
A Unified Pyramid Recurrent Network for Video Frame Interpolation [CVPR 2023] [arXiv] ↩ ↩2
-
Error-Aware Spatial Ensembles for Video Frame Interpolation [arXiv] ↩ ↩2
-
H-VFI: Hierarchical Frame Interpolation for Videos with Large Motions [arXiv] ↩ ↩2
-
Revisiting Adaptive Convolutions for Video Frame Interpolation [WACV 2021] [arXiv] ↩ ↩2
-
Neighbor Correspondence Matching for Flow-based Video Frame Synthesis [MM 2022] [arXiv] ↩ ↩2
-
IFRNet: Intermediate Feature Refine Network for Efficient Frame Interpolation [CVPR 2022] [arXiv] ↩ ↩2 ↩3
-
Enhanced Bi-directional Motion Estimation for Video Frame Interpolation [WACV 2023] [arXiv] ↩ ↩2
-
Asymmetric Bilateral Motion Estimation for Video Frame Interpolation [ICCV 2021] [arXiv] ↩ ↩2 ↩3
-
Real-Time Intermediate Flow Estimation for Video Frame Interpolation [ECCV 2022] [arXiv] ↩
-
Video Frame Interpolation Transformer [CVPR 2022] [arXiv] ↩ ↩2 ↩3
-
JNMR: Joint Non-linear Motion Regression for Video Frame Interpolation [TIP 2023] [arXiv] ↩ ↩2 ↩3
-
Efficient Convolution and Transformer-Based Network for Video Frame Interpolation [ICIP 2023] [arXiv] ↩ ↩2 ↩3 ↩4
-
ST-MFNet: A Spatio-Temporal Multi-Flow Network for Frame Interpolation [CVPR 2022] [arXiv] ↩
-
Deep Bayesian Video Frame Interpolation [ECCV 2022] ↩ ↩2
-
Enhancing Deformable Convolution based Video Frame Interpolation with Coarse-to-fine 3D CNN [ICIP 2022] [arXiv] ↩