OpticalFlow

A collection of Optical Flow papers and code and some numbers.

Network	Paper	Code	Year	Best for	Supervised/Unsupervised	Complexity	Sintel-clean EPE
FlowNet	https://arxiv.org/abs/1504.06852	https://github.com/ClementPinard/FlowNetTorch , https://github.com/ClementPinard/FlowNetPytorch, https://github.com/aasharma90/FlowNet	2015	Basic Optical Flow	Supervised	32,070,472 parameters (FlowNetS), 32,561,032 parameters (FlowNetC)	---
FlowNet 2.0	https://arxiv.org/abs/1612.01925	https://github.com/lmb-freiburg/flownet2, https://github.com/NVIDIA/flownet2-pytorch, https://github.com/sampepose/flownet2-tf, https://github.com/vt-vl-lab/pytorch_flownet2	2016	Segmentation	Supervised	More than FlowNet	---
SPyNet	https://arxiv.org/abs/1611.00850	https://github.com/anuragranj/spynet, https://github.com/sniklaus/pytorch-spynet, https://github.com/anuragranj/spynet	Year	Fast Learning	Supervised	1,200,250 parameters	6.64
DSTFlow	https://pdfs.semanticscholar.org/47bc/34ae6f5dc104bc289ae3bb4fa75ef75fbc21.pdf	---	2017	Huge amount of Unlabeled Data	Unsupervised	---	---
PWC-Net	https://arxiv.org/abs/1709.02371	https://github.com/NVlabs/PWC-Net/tree/master/PyTorch, https://github.com/NVlabs/PWC-Net, https://github.com/sniklaus/pytorch-pwc	2018	Good Trade-Off Accuracy and Running Time	Supervised	Small Complexity	3.45
SelFlow	https://arxiv.org/pdf/1904.09117.pdf	https://github.com/ppliuboy/SelFlow,	2019	No Labeled Data, Good Accuracy also for Data with Occlusions	Unsupervised	Quite big but based on PWC-Net	3.74
PCLNet	https://arxiv.org/pdf/1907.11628.pdf	https://github.com/Kwanss/PCLNet	2019	With Unlabeled Data	Unsupervised	Big	---

Datasets for Optical Flow training and testing

Dataset	Link	Description
MPI Sintel Flow Dataset	http://sintel.is.tue.mpg.de/	The MPI Sintel Dataset addresses limitations of existing optical flow benchmarks. It provides naturalistic video sequences that are challenging for current methods. It is designed to encourage research on long-range motion, motion blur, multi-frame analysis, non-rigid motion. The dataset contains flow fields, motion boundaries, unmatched regions, and image sequences. The image sequences are rendered with different levels of difficulty. Sintel is an open source animated short film produced by Ton Roosendaal and the Blender Foundation. Here we have modified the film in many ways to make it useful for optical flow evaluation.
KITTI 2012	http://www.cvlibs.net/datasets/kitti/eval_stereo_flow.php?benchmark=flow	The stereo / flow benchmark consists of 194 training image pairs and 195 test image pairs, saved in loss less png format. Our evaluation server computes the average number of bad pixels for all non-occluded or occluded (=all groundtruth) pixels. We require that all methods use the same parameter set for all test pairs. Our development kit provides details about the data format as well as MATLAB / C++ utility functions for reading and writing disparity maps and flow fields.
KITTI 2015	http://www.cvlibs.net/datasets/kitti/eval_scene_flow.php?benchmark=flow	The stereo 2015 / flow 2015 / scene flow 2015 benchmark consists of 200 training scenes and 200 test scenes (4 color images per scene, saved in loss less png format). Compared to the stereo 2012 and flow 2012 benchmarks, it comprises dynamic scenes for which the ground truth has been established in a semi-automatic process. Our evaluation server computes the percentage of bad pixels averaged over all ground truth pixels of all 200 test images. For this benchmark, we consider a pixel to be correctly estimated if the disparity or flow end-point error is <3px or <5% (for scene flow this criterion needs to be fulfilled for both disparity maps and the flow map). We require that all methods use the same parameter set for all test pairs.
FlyingChairs	https://lmb.informatik.uni-freiburg.de/resources/datasets/FlyingChairs.en.html#flyingchairs	The "Flying Chairs" are a synthetic dataset with optical flow ground truth. It consists of 22872 image pairs and corresponding flow fields. Images show renderings of 3D chair models moving in front of random backgrounds from Flickr. Motions of both the chairs and the background are purely planar.
FlyingChairs2	https://lmb.informatik.uni-freiburg.de/resources/datasets/FlyingChairs.en.html#flyingchairs2	The Flying Chairs 2 dataset is generated using the same settings as the Flying Chairs dataset, but contains additional modalities that were used to train the networks in the paper Occlusions, Motion and Depth Boundaries with a Generic Network for Disparity, Optical Flow or Scene Flow Estimation.
ChairsSDHom	https://lmb.informatik.uni-freiburg.de/resources/datasets/FlyingChairs.en.html#chairssdhom	"ChairsSDHom" is a synthetic dataset with optical flow ground truth. Designed to be robust to untextured regions and to produce flow magnitude histograms close to those of the UCF101 dataset, ChairsSDHom is a good candidate for training if you want your optical flow method to work well on real-world data and generally rather small displacements. The dataset is abstract enough to not overfit to any realistic scenario, so even if you have specialized data, ChairsSDHom may still serve as good additional or pretraining data.
ChairsSDHom extended	https://lmb.informatik.uni-freiburg.de/resources/datasets/FlyingChairs.en.html#chairssdhom2	The extended version contains the same flows and images, but also additional modalities that were used to train the networks in the paper Occlusions, Motion and Depth Boundaries with a Generic Network for Disparity, Optical Flow or Scene Flow Estimation.
FlyingThings3D	https://lmb.informatik.uni-freiburg.de/resources/datasets/SceneFlowDatasets.en.html	The collection contains more than 39000 stereo frames in 960x540 pixel resolution, rendered from various synthetic sequences. For details on the characteristics and differences of the three subsets, we refer the reader to our paper.