Evaluation, Training, Demo, and Inference of SI-DDPM-FMO

Single-Image Deblurring, Trajectory and Shape Recovery of Fast Moving Objects with Denoising Diffusion Probabilistic Models (WACV 2024)

Radim Spetlik, Denys Rozumnyi, Jiri Matas

[paper] [cvf]

Setup

Repository clone & environment installation

git clone https://github.com/radimspetlik/SI-DDPM-FMO
cd SI-DDPM-FMO
pipenv install

Note that we only support pipenv installation. If you do not have pipenv installed, please install it first using pip install pipenv.

Evaluation

The download process of pretrained models, and benchmark datasets is somewhat cumbersome, so we will eventually put everything in one script (use issues to show your interest in doing so, please).

1. Download the pre-trained models

The pre-trained SI-DDPM-FMO models as reported in the paper are available here.

Download them and place them in the models dir.

For the baseline method, visit DeFMO, download the pre-trained models and place them it in the models dir.

2. Download the FMO benchmark dataset

The FMO benchmark datasets are available here. After downloading, place the data in the datasets dir.

FMO benchmark evaluation

To evaluate the SI-DDPM-FMO model on the FMO benchmark dataset, run:

./benchmark_siddmpfmo.sh

To evaluate the baseline model on the FMO benchmark dataset, run:

./benchmark_baseline.sh

Training

For training, you first need to generate the training dataset as described bellow, and place it in the datasets dir. Then, simply run

./trn_siddpmfmo.sh

Inspecting the script, you find the training parameters. The training is expected to be run on multiple GPUs, with the DDP library of pytorch. Modify according to your needs.

The baseline training is run in the same way, but using the trn_baseline.sh script.

Synthetic dataset generation

For the dataset generation, please download:

• ShapeNetCore.v2 dataset.

• Textures from the DTD dataset. The exact split used in DeFMO is from the "Neural Voxel Renderer: Learning an Accurate and Controllable Rendering Tool" model and can be downloaded here.

• Backgrounds for the training dataset from the VOT dataset.

• Backgrounds for the testing dataset from the Sports1M dataset.

• Blender 2.79b with Python enabled.

Then, insert your paths in renderer/settings.py file. To generate the dataset, run in renderer sub-folder:

python run_render.py

Note that the full training dataset with 50 object categories, 1000 objects per category, and 24 timestamps takes up to 1 TB of storage memory. Due to this and also the ShapeNet licence, we cannot make the pre-generated dataset public - please generate it by yourself using the steps above.

Reference

If you use this repository, please cite the following publication:

@inproceedings{siddpmfmo2024,
  author = {Radim Spetlik and Denys Rozumnyi and Jiri Matas},
  title = {Single-Image Deblurring, Trajectory and Shape Recovery of Fast Moving Objects with Denoising Diffusion Probabilistic Models},
  booktitle = {WACV},
  address = {Waikoloa, Hawaii, USA},
  month = jan,
  year = {2024}
}

Code Sources

We use a wild mix of the following repositories in our implementation:

• RePaint
• GuidedDiffusion
• ImprovedDiffusion
• DeFMO
• FMO-Deblurring-Benchmark

Therefore a big thanks to the authors of these repositories!