SCOOP: Self-Supervised Correspondence and Optimization-Based Scene Flow

[Project Page] [Paper] [Video] [Slides] [Poster]

Created by Itai Lang^1,2, Dror Aiger², Forrester Cole², Shai Avidan¹, and Michael Rubinstein².
¹Tel Aviv University   ²Google Research

Abstract

Scene flow estimation is a long-standing problem in computer vision, where the goal is to find the 3D motion of a scene from its consecutive observations. Recently, there have been efforts to compute the scene flow from 3D point clouds. A common approach is to train a regression model that consumes source and target point clouds and outputs the per-point translation vector. An alternative is to learn point matches between the point clouds concurrently with regressing a refinement of the initial correspondence flow. In both cases, the learning task is very challenging since the flow regression is done in the free 3D space, and a typical solution is to resort to a large annotated synthetic dataset.

We introduce SCOOP, a new method for scene flow estimation that can be learned on a small amount of data without employing ground-truth flow supervision. In contrast to previous work, we train a pure correspondence model focused on learning point feature representation and initialize the flow as the difference between a source point and its softly corresponding target point. Then, in the run-time phase, we directly optimize a flow refinement component with a self-supervised objective, which leads to a coherent and accurate flow field between the point clouds. Experiments on widespread datasets demonstrate the performance gains achieved by our method compared to existing leading techniques while using a fraction of the training data.

Citation

If you find our work useful in your research, please consider citing:

@InProceedings{lang2023scoop,
  author = {Lang, Itai and Aiger, Dror and Cole, Forrester and Avidan, Shai and Rubinstein, Michael},
  title = {{SCOOP: Self-Supervised Correspondence and Optimization-Based Scene Flow}},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  pages = {5281--5290},
  year = {2023}
}

Installation

The code has been tested with Python 3.6.13, PyTorch 1.6.0, CUDA 10.1, and cuDNN 7.6.5 on Ubuntu 16.04.

Clone this repository:

git clone https://github.com/itailang/SCOOP.git
cd SCOOP/

Create a conda environment:

# create and activate a conda environment
conda create -n scoop python=3.6.13 --yes
conda activate scoop

Install required packages:

sh install_environment.sh

Compile the Chamfer Distance op, implemented by Groueix et al. The op is located under auxiliary/ChamferDistancePytorch/chamfer3D folder. The following compilation script uses a CUDA 10.1 path. If needed, modify script to point to your CUDA path. Then, use:

sh compile_chamfer_distance_op.sh

The compilation results should be created under auxiliary/ChamferDistancePytorch/chamfer3D/build folder.

Usage

Data

Create folders for the data:

mkdir ./data/
mkdir ./data/FlowNet3D/
mkdir ./data/HPLFlowNet/

We use the point cloud data version prepared Liu et al. from their work FlowNet3D. Please follow their code to acquire the data.

• Put the preprocessed FlyingThings3D dataset at ./data/FlowNet3D/data_processed_maxcut_35_20k_2k_8192/. This data is denoted as FT3D_o.
• Put the preprocessed KITTI dataset at ./data/flownet3d/kitti_rm_ground/. This dataset is denoted as KITTI_o and its subsets are denoted as KITTI_v and KITTI_t.

We also use the point cloud data version prepared Gu et al. from their work HPLFlowNet. Please follow their code to acquire the data.

• Put the preprocessed FlyingThings3D dataset at ./data/HPLFlowNet/FlyingThings3D_subset_processed_35m/. This dataset is denoted as FT3D_s.
• Put the preprocessed KITTI dataset at ./data/flownet3d/KITTI_processed_occ_final/. This dataset is denoted as KITTI_s.

Note that you may put the data elsewhere and create a symbolic link to the actual location. For example:

ln -s /path/to/the/actual/dataset/location ./data/FlowNet3D/data_processed_maxcut_35_20k_2k_8192  

Training and Evaluation

Switch to the scripts folder:

cd ./scripts

FT3D_o / KITTI_o

To train a model on 1,800 examples from the train set of FT3D_o, run the following command:

sh train_on_ft3d_o.sh

Evaluate this model on KITTI_o with 2,048 point per point cloud using the following command:

sh evaluate_on_kitti_o.sh

The results will be saved to the file ./experiments/ft3d_o_1800_examples/log_evaluation_kitti_o.txt.

Evaluate this model on KITTI_o with all the points in the point clouds using the following command:

sh evaluate_on_kitti_o_all_point.sh

The results will be saved to the file ./experiments/ft3d_o_1800_examples/log_evaluation_kitti_o_all_points.txt.

KITTI_v / KITTI_t

To train a model on the 100 examples of KITTI_v, run the following command:

sh train_on_kitti_v.sh

Evaluate this model on KITTI_t with 2,048 point per point cloud using the following command:

sh evaluate_on_kitti_t.sh

The results will be saved to the file ./experiments/kitti_v_100_examples/log_evaluation_kitti_t.txt.

Evaluate this model on KITTI_t with all the points in the point clouds using the following command:

sh evaluate_on_kitti_t_all_points.sh

The results will be saved to the file ./experiments/kitti_v_100_examples/log_evaluation_kitti_t_all_points.txt.

FT3D_s / KITTI_s, FT3D_s / FT3D_s

To train a model on 1,800 examples from the train set FT3D_s, run the following command:

sh train_on_ft3d_s.sh

Evaluate this model on KITTI_s with 8,192 point per point cloud using the following command:

sh evaluate_on_kitti_s.sh

The results will be saved to the file ./experiments/ft3d_s_1800_examples/log_evaluation_kitti_s.txt.

Evaluate this model on the test set of FT3D_s with 8,192 point per point cloud using the following command:

sh evaluate_on_ft3d_s.sh

The results will be saved to the file ./experiments/ft3d_s_1800_examples/log_evaluation_ft3d_s.txt.

Visualization

First, save results for visualization by adding the flag --save_pc_res 1 when running the evaluation script. For example, the script for evaluating on KITTI_t. The results will be saved to the folder ./experiments/kitti_v_100_examples/pc_res/.

Then, select the scene index that you would like to visualize and run the visualization script. For example, visualizing scene index #1 from KITTI_t:

python visualize_scoop.py --res_dir ./../experiments/kitti_v_100_examples/pc_res --res_idx 1 

The visualizations will be saved to the folder ./experiments/kitti_v_100_examples/pc_res/vis/.

Evaluation with Pretrained Models

First, download our pretrained models with the following command:

bash download_pretrained_models.sh

The models (about 2MB) will be saved under pretrained_models folder in the following structure:

pretrained_models/
├── ft3d_o_1800_examples/model_e100.tar
├── ft3d_s_1800_examples/model_e060.tar
├── kitti_v_100_examples/model_e400.tar

Then, use the evaluation commands mentioned in section Training and Evaluation, after changing the experiments folder in the evaluation scripts to the pretrained_models folder.

License

This project is licensed under the terms of the MIT license (see the LICENSE file for more details).

Acknowledgment

Our code builds upon the code provided by Puy et al., Groueix et al., Liu et al., and Gu et al. We thank the authors for sharing their code.