Deep Material-aware Cross-spectral Stereo Matching

Tiancheng Zhi, Bernardo R. Pires, Martial Hebert, Srinivasa G. Narasimhan

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.

[Project] [Paper] [Supp]

Disclaimer

This is an improved and simplified version of the CVPR code. Compared with the original CVPR version, this code achieves a better performance (see pretrained model below). Main changes include:

• No white balancing in STN
• Use normal convolution instead of symmetric convolution in STN
• Randomly flip the input and output of STN
• Use learning rate scheduler
• Hyperparameter changes

To compare with the original CVPR result, please refer to the project page (first download link to the dataset).

Requirements

• TITAN Xp GPU * 2
• Ubuntu 16.04
• Python 3
• PyTorch 1.0
• OpenCV
• Visdom (for visualization)

Data

Download rgbnir_stereo, and move "data" and "lists" into the "cs-stereo" folder.

Download precomputed_material, and put it under the "cs-stereo" folder.

Then run:

sh cp_material.sh precomputed_material data

See project page for more information and downlad links of PittsStereo Dataset.

Training

CUDA_VISIBLE_DEVICES=1,0 python3 train.py

Testing

CUDA_VISIBLE_DEVICES=1,0 python3 test.py --ckpt-path ckpt/47.pth

Pretrained Model

Download pretrained.pth

Performance (RMSE, lower is better):

Model	Common	Light	Glass	Glossy	Vegetation	Skin	Clothing	Bag	Mean
CVPR'18	0.53	0.69	0.65	0.70	0.72	1.15	1.15	0.80	0.80
Pretrained	0.47	0.56	0.56	0.61	0.72	0.93	0.91	0.86	0.70