3D Object Detection using ZED and Tensorflow
Based on Tensorflow's object detection API tutorial, this Python 3 program will show how to detect, identify and locate objects using the ZED camera and Tensorflow's ssd_mobilenet inference model.
The project has the following dependencies:
- Python 3
- ZED SDK
- pyzed-python (ZED SDK python wrapper)
- Tensorflow models
- Install the ZED SDK
- Install the python wrapper for the ZED SDK
- Install cuDNN, be careful of using the correct CUDA version
Install Tensorflow, ideally tensorflow-gpu by following the instruction for your platform https://www.tensorflow.org/install/
It should be along the line of:
pip3 install --upgrade tensorflow-gpu
Install Tensorflow Object Detection API
First download the models repository
git clone https://github.com/tensorflow/models
Then setup the Tensorflow Object Detection API by following the instructions here :
Make sure the object detection API is in the Python path.
Run the sample
Make sure the virtualenv is active
Run the code with python3
A DockerFile is provided in the docker folder
CUDA / cuDNN version
Please refer to this link to know the cuDNN and CUDA versions needed for a given Tensorflow version. At the moment of writing the current versions support CUDA 9.0 and cuDNN 7.
This sample uses 2 threads, one for the ZED images capture and one for the Tensorflow detection. While it may seems more complex at first, it actually solves 2 issues at a time :
The performances are increase, the depth computation and the ZED pipeline in general is done in parallel to the inference.
Tensorflow and the ZED SDK uses CUDA GPU computation and therefore requires the use of CUDA Context. Since we currently can't share the CUDA Context between the ZED and TF, we have to separates the GPU computation. Each CUDA Context must therefore have his own thread. Another solution (not tested) could be to use something similar to this based on pop/push before each operations.