conda create -n LC_Mamba
conda activate LC_Mambaconda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidiapip install -r requirements.txt
cd kernels/selective_scan && pip install .
Checkpoints can be downloaded from the following link:
Download Checkpoints
Please place the checkpoints in ./ckpt/.
| Method | Vimeo90K (PSNR/SSIM) | UCF101 (PSNR/SSIM) | Xiph 2K (PSNR/SSIM) | Xiph 4K (PSNR/SSIM) | M.B. (IE) | SNU-FILM Easy (PSNR/SSIM) | SNU-FILM Medium (PSNR/SSIM) | SNU-FILM Hard (PSNR/SSIM) | SNU-FILM Extreme (PSNR/SSIM) | Params (M) | FLOPS (T) |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Ours-C | 36.10/0.9801 | 35.38/0.9700 | 37.12/0.946 | 34.81/0.908 | 1.94 | 40.10/0.9915 | 36.11/0.9809 | 30.81/0.9405 | 25.69/0.8710 | 4.3 | 0.27 |
| Ours-E | 36.20/0.9802 | 35.42/0.9699 | 37.17/0.946 | 34.99/0.910 | 1.96 | 40.15/0.9912 | 36.18/0.9809 | 30.89/0.9416 | 25.81/0.8725 | 6.7 | 0.29 |
| Ours-B | 36.52/0.9810 | 35.47/0.9703 | 37.33/0.947 | 35.14/0.911 | 1.90 | 40.20/0.9909 | 36.30/0.9810 | 31.00/0.9417 | 25.83/0.8722 | 16.2 | 1.07 |
To comprehensively evaluate the proposed model's performance under various conditions and resolutions, experiments were conducted using multiple datasets.
/data/datasets/
├── middlebury
├── snufilm
├── ucf101
├── vimeo_triplet
├── Xiph
The following datasets were used:
-
Vimeo90K dataset:
Consists of frame triplets with a resolution of 448×256. The test set includes 3,782 triplets. -
UCF101 dataset:
The test set contains 379 frame triplets selected from DVF, with a resolution of 256×256. -
Xiph dataset:
The original images were downsampled to 2K resolution as "Xiph-2K" and cropped centrally to form "Xiph-4K" for testing. -
Middlebury OTHER dataset:
The OTHER set, with a resolution of approximately 640×480, was used for testing. -
SNU-FILM dataset:
This dataset consists of 1,240 frame triplets with a resolution of approximately 1280×720. It is categorized into four difficulty levels—Easy, Medium, Hard, and Extreme—based on motion magnitude, enabling detailed performance comparisons.
Run the benchmark using the following command:
Make bench_Ours-C
Make bench_Ours-E
Make bench_Ours-BThis project is distributed under the Apache 2.0 license. It incorporates concepts and code from RIFE, EMA-VFI, and VFIMamba, and users are advised to adhere to the licensing terms of these respective projects.
We extend our gratitude to the authors of these works for their exceptional contributions.