Skip to content

Deep Video Deblurring: The Devil is in the Details (ICCV Workshop 2019)

License

Notifications You must be signed in to change notification settings

visinf/deblur-devil

Repository files navigation

Deep Video Deblurring: The Devil is in the Details


This PyTorch implementation accompanies the paper:

Jochen Gast and Stefan Roth, Deep Video Deblurring: The Devil is in the Details,
ICCV Workshop on Learning for Computational Imaging (ICCVW), Seoul, Korea, November, 2019

Preprint: https://arxiv.org/abs/1909.12196 DeepVisinf Logo
Contact: Jochen Gast (jochen.gast@visinf.tu-darmstadt.de)

Visual Inference



Components for Deep Video Deblurring

Ignoring the framework code, the components for Deep Video Deblurring are contained in the modules

Modules Description
datasets.gopro_nah implementation (+ augmentations) for reading Nah et al's GoPro dataset
datasets.gopro_su implementation (+ augmentations) for reading Su et al's GoPro dataset
losses.deblurring_error loss implementation (just plain MSE)
models.deblurring.dbn our baseline implementation for Su et al's DBN network
models.deblurring.flowdbn our FlowDBN implementation
models.flow.flownet1s FlowNet1S optical flow network used in FlowDBN
models.flow.pwcnet PWC optical flow network used in FlowDBN
visualizers.gopro_inference TensorBoard Visualizer for inference (on a given dataset)
visualizers.gopro_visualizer TensorBoard Visualizer for monitoring training progress

Datasets

The mirrors of the GoPro datasets can be found here:

GoPro Dataset Paper
Su et al
mirror
S. Su, M. Delbracio, J. Wang, G. Sapiro, W. Heidrich, and O. Wang.
Deep video deblurring for hand-held cameras, CVPR, 2017.
Nah et al
mirror
S. Nah, T. H. Kim, and K. M. Lee.
Deep multi-scale convolutional neural network for dynamic scene deblurring, CVPR, 2017.

Installation, External Packages & Telegram Bot

This code has been tested with Python 3.6, PyTorch 1.2, Cuda 10.0 but is likely to run with newer versions of PyTorch and Cuda. For installing PyTorch, follow the steps here.

Additional dependencies can be installed by running the script

pip install -r requirements.txt

The code for the PWC network relies on a correlation layer implementation provided by Clément Pinard.
To install the correlation layer, run

cd external_packages && . install_spatial_correlation_sampler.sh

The framework supports status updates sent via Telegram, i.e. using

logging.telegram(msg, *args)

Per default, any finished epoch will send an update with the current losses. Note that the dependencies for the Telegram bot, i.e. filetype jsonpickle telegrambotapiwrapper, are included in requirements.txt, but they are not essential to the framework.

In order to use the Telegram bot, you need to copy dot_machines.json to ~/.machines.json and fill in the required chat_id and tokens. Here you can find out how to obtain these credentials.

Scripts and Models

The scripts make use of the environment variables DATASETS_HOME pointing to the root folder of the respective dataset, and EXPERIMENTS_HOME to create output artifacts.

Scripts Description
scripts.deblurring.gopro_nah Scripts for GoPro by Nah et al.
scripts.deblurring.gopro_su Scripts for GoPro by Su et al.

To train the models you need the prewarping networks. Place them in $EXPERIMENTS_HOME/prewarping_networks.

Models Description
Prewarping Networks FlowNet1S and PWCNet

More models will follow soon!

Deep Visual Inference Framework

Adding Classes

For adding new classes/functions following packages are relevant:

Modules Description
augmentations Register (gpu) augmentations
contrib Supposed to be used for PyTorch utilities
datasets Register datasets
losses Register losses
models Register models
optim Register optimizers
utils Any non-PyTorch related utilities
visualizers Register Tensorboard visualizers

Everything is a Dictionary

Most things are automated for convenience. To accomodate this automation, there are three essential Python dictionaries being passed around which hold any artifact produced along the way

Dictionary Name Description
example_dict The example dictionary. Gets created by a dataset and is supposed to hold all data relevant to a dataset example. Note that an augmentation is supposed to receive and return an example_dict. The example dictionary is received by augmentations, losses, models, and visualizers.
model_dict The model dictionary. Gets created by a model. Supposed to hold any data, i.e. predictions, generated by a model. The example dictionary is received by losses and visualizers.
loss_dict The loss dictionary. Gets created by a loss. Supposed to hold any data, i.e. metrics, generated by a loss. The loss dictionary is received by visualizers.

Keep this in mind, when adding new functionality.

Magic Keys

Again, the framework passes around dictionaries between augmentations, datasets, losses, models, and visualizers. You may want a subset of these tensors to be transfered to Cuda along the way. The are two magic prefixes for dictionary keys to assure that, i.e. input* and target*. Any input or target tensor that should be transfered to Cuda along the way, has to start with one of these prefixes.

E.g. a dataset may return a dictionary with the keys input1 and target1 (both cpu tensors); any augmentation, loss, or model will fetch these as gpu tensors.

Data Loader Collation

This is in particular interesting for the GoPro datasets. The framework allows dataloaders to return multiple -- typically photometric -- augmentations per single example, in order to increase throughput when loading large images. In such a case a tensor returned from a single dataset example should have three dimensions. The pipeline will stack the first dimension into a batch.

Let's say, a dataset returns four examples per load, i.e. input1 and target1 tensors have the dimension 4xHxW. For, e.g. batch_size=8, the resulting tensors (received by the augmentation, loss, model, and visualizers) will be 32xHxW.

Logging Facilities

There are some automated logging facilities which log any output into the given save directory.

  • A args.txt and args.json file remembering the passed arguments.
  • A logbook.txt gets created remembering all calls to logging.info(msg, *args).
  • CSV/MAT file generation of any loss (of any epoch) found in the loss_dict.
  • For every run, the complete source code gets zipped into the log directory.
  • There is always a tensorboard created at the $SAVE/tb directory.

Custom Tensorboard Calls

When calling tensorboard summaries in your models for debugging, I suggest to replace any call to torch.utils.tensorboard.SummaryWriter.add_* by utils.summary.*

For instance, instead of

from torch.utils.tensorboard import SummaryWriter
writer = SummaryWriter(..)
writer.add_scalar(tensor1)
writer.add_images(tensor2)

use

from utils import summary
summary.scalar(tensor1)
summary.images(tensor2)

Advantages:

  • This will automatically use the created tensorboard in the save directory.
  • The current epoch gets automatically set by the framework.
  • Otherwise, they are identical.

More Settings

A limited amount of customization is available in constants.py where you can set

  • Logging indents, time zones, and colors
  • Filename defaults
  • Flush intervals
  • etc

License

This code is licensed under the Apache License, Version 2.0, as found in the LICENSE file.

Citation

# brief
@inproceedings{Gast:2019:DVD,
    Author = {Jochen Gast and Stefan Roth},
    Booktitle = {ICCV Workshops},
    Title = {Deep Video Deblurring: {T}he Devil is in the Details},
    Year = {2019}
}
# detailed
@inproceedings{Gast:2019:DVD,
    Address = {Seoul, Korea},
    Author = {Jochen Gast and Stefan Roth},
    Booktitle = {ICCV Workshop on Learning for Computational Imaging (ICCVW)},
    Month = nov,
    Title = {Deep Video Deblurring: {T}he Devil is in the Details},
    Year = {2019}
}

About

Deep Video Deblurring: The Devil is in the Details (ICCV Workshop 2019)

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published