FP-Stereo

Hardware-Efficient Stereo Vision on FPGAs

This is a repository for FP-Stereo, a framework for building high-performance stereo matching pipelines on FPGAs automatically.

FP-Stereo covers both traditional and deep-learning based stereo matching methods. We have released the source code for buidling stereo vision pipelines on FPGAs using semi-global matching (SGM) which is widely adopted in real-world embeddded systems. The technical details and results can be found in our paper. FP-Stereo achieves the state-of-the-art performance on the KITTI dataset: 161 frames per second (FPS) for 1242*374 image resolution on an ZCU102 FPGA. In the near future, we will provide more design choices for SGM and cover DNN-based stereo matching methods.

Please give us a star if you find FP-Stereo useful and cite our paper if FP-Stereo is helpful for your research:

@inproceedings{zhao2020fp,
  title={FP-Stereo: Hardware-Efficient Stereo Vision for Embedded Applications},
  author={Zhao, Jieru and Liang, Tingyuan and Feng, Liang and Ding, Wenchao and Sinha, Sharad and Zhang, Wei and Shen, Shaojie},
  booktitle={2020 30th International Conference on Field Programmable Logic and Applications (FPL)},
  year={2020},
  organization={IEEE}
}

Download the repository

To get a local copy of the repository, clone this repository to the local system with the following command:

git clone https://github.com/zjru/FP-Stereo.git

Software and hardware requirements

Software: SDx 2018.3 Development Environment and Xilinx ZCU102 reVISION stack.

Hardware: Xilinx ZCU102 FPGA

Configuration and Setup

1. Modify Makefile

cd ./SGM/src
vim Makefile
# Line 4: set the platform in Makefile to the path of the downloaded ZCU102 reVISION stack
# Line 11: select the suitable compiler option 

• Notes: SDSoC provide HLS estimation for latency and resource usage. To get HLS estimations, set CC with -perf-est flag (line 11). To generate bitstream and run the pipeline on FPGA board, set CC with "-impl-strategy" (line 12).

2. Set the environmental variable and paths to the source code and results.

cd ./SGM/scripts
vim run_sdx.py
# Modify line 12-16 and set the variables correspondingly.

Build an SDSoC project with FP-Stereo

• Build a project with default parameters:

cd ./SGM/scripts
python run_sdx.py 374 1242 0 5 64 16 4 0 0 5 3 7 86

• Build a project with user-specified parameters:

cd ./SGM/scripts
python run_sdx.py [argument list]

• List of arguments:
  • height: the height of input image
  • width: the width of input image
  • cost_function: the type of cost function (0: census transform, 1: rank transform, 2: SAD, 3: ZSAD)
  • window_size: length of the window for cost computation (typical choices: 3, 5, 7)
  • max_disp: the disparity range (64 and 128 are typical disparity ranges)
  • parallel_disp: unrolling factor in the disparity dimensition
  • num_dir: the number of directions to aggregate the costs (4 is supported. 5 and 8 will be supported in the future)
  • uniqueness: whether to adopt uniqueness check for refinement
  • lr_check: the method for L-R consistency check (0: NLR, 1: LR1, 2: LR2)
  • filter_win: the window length of median filter for refinement (5 is a good choice)
  • shd_window: the window length for the SHD cost computation method
  • p1: penalty p1 for cost aggregation
  • p2: penalty p2 for cost aggregation

Design space exploration with FP-Stereo

• Perform DSE with default parameter setting:

cd ./SGM/scripts
python run_dse_hls.py

• Perform DSE in a customized design space:
  • Modify parameter options to customize design space in "run_dse_hls.py".
  • Set the number of CPU cores for multi-thread processing.
  • Open terminal and run:

cd ./SGM/scripts
python run_dse_hls.py

License:

The source code is released under GPLv3 license. If you have any questions about FP-Stereo, please contact Jieru ZHAO (jzhaoao@connect.ust.hk).

For commercial inqueries, please contact Prof. Wei ZHANG (wei.zhang@ust.hk).