Residual 3D Scene Flow Learning with Context-Aware Feature Extraction (IEEE Transactions on Instrumentation and Measurement)

This is the official implementations of our TIM 2022 paper, "Residual 3D Scene Flow Learning with Context-Aware Feature Extraction" created by Guangming Wang, Yunzhe Hu, Xinrui Wu, and Hesheng Wang.

Citation

If you find our work useful in your research, please cite:

@article{wang2022residual,
  title={Residual 3-D Scene Flow Learning With Context-Aware Feature Extraction},
  author={Wang, Guangming and Hu, Yunzhe and Wu, Xinrui and Wang, Hesheng},
  journal={IEEE Transactions on Instrumentation and Measurement},
  volume={71},
  pages={1--9},
  year={2022},
  publisher={IEEE}
}

Prerequisites

• Python 3.6.9
• PyTorch 1.5.0
• CUDA 10.2
• numba
• tqdm

Data preprocess

For fair comparison with previous methods, we adopt the preprocessing steps in HPLFlowNet. Please refer to repo. We also copy the preprocessing instructions here for your reference.

• FlyingThings3D: Download and unzip the "Disparity", "Disparity Occlusions", "Disparity change", "Optical flow", "Flow Occlusions" for DispNet/FlowNet2.0 dataset subsets from the FlyingThings3D website (we used the paths from this file, now they added torrent downloads) . They will be upzipped into the same directory, RAW_DATA_PATH. Then run the following script for 3D reconstruction:

python3 data_preprocess/process_flyingthings3d_subset.py --raw_data_path RAW_DATA_PATH --save_path SAVE_PATH/FlyingThings3D_subset_processed_35m --only_save_near_pts

• KITTI Scene Flow 2015 Download and unzip KITTI Scene Flow Evaluation 2015 to directory RAW_DATA_PATH. Run the following script for 3D reconstruction:

python3 data_preprocess/process_kitti.py RAW_DATA_PATH SAVE_PATH/KITTI_processed_occ_final

Usage

Install pointnet2 library

Compile the furthest point sampling, grouping and gathering operation for PyTorch. We use operations from this repo.

cd pointnet2
python setup.py install
cd ../

Train

Set data_root in config_train.yaml to SAVE_PATH in the data preprocess section. Then run

python train.py config_train.yaml

After training the model with a quarter dataset, you can finetune the model with the full dataset and achieve a better results by running the following command. Remember to set pretrain in config_train_finetune.yaml as the path to the pretrained weights.

python train.py config_train_finetune.yaml

Evaluate

We provide pretrained weights in pretrain_weights.

Set data_root and in config_evaluate.yaml to SAVE_PATH in the data preprocess section, and specify dataset in the script . Then run

python evaluate.py config_evaluate.yaml

Quantitative results

Acknowledgements

We thank the following open-source projects for the help of the implementations.

• PointNet++

• HALFlow

• PointPWC-Net