Frame Rate

[sparro12]

What is the speed of this implementation? We are mostly interested in finding a way to stitch camera images and had thoughts about transferring to birds eye view for depth information. Our purpose is to implement this is on our autonmous race car, and we will be running at high speeds so traditional homography seems to have too many inaccuracies.

Your methods pose interest to us. However, given that this is on a racecar, speed is of high importance. We are currently running our cameras anywhere from 25 to 40 Hz. In the paper, you mention 2Hz. Is this a consequence from the speed of the network?

In short, what's the latency or maximum speed of the network?

[lreiher]

The 2Hz in the paper only relates to the sampling rate of the training sample generation process, i.e., training samples in our dataset are 0.5s apart.

We have a C++ ROS Node to run trained models in inference mode. I quickly measured its runtimes for the following configuration:

• NVIDIA GeForce RTX 3090
• uNetXST model
• 512x256 image size, 10 predicted semantic classes
• computation of the segmented images is not included in runtime
• single image: ~40ms
• 4 images: ~100ms

Possible areas of runtime improvement:

• The DeepLab models will likely run faster, especially DeepLab MobileNet. Their runtime also doesn't depend on the number of input images, since they work on a single homography image instead. In that case though one would also need to consider the runtime of computing the homography image.
• uNetXST can also be simplified by reducing its Encoder-Decoder-Depth (default: 5).
• Reducing image size will speed-up processing.
• Reducing the number of predicted semantic classes will speed-up processing.

Please also note that we have not focused on achieving minimal runtime. Typical model optimization techniques like pruning or quantization could further decrease latency.

[sparro12]

Thank you. This was helpful. We will probably try to replicate this as first steps and then move to see if there are improvements. For our high speeds, a transformation to birds eye may be necessary as image stitching through homography alone seems to be unreliable.

[sparro12]

Would you be willing to share the C++ ROS node with me? I'd like to do some experimenting and it would speed up the process to have your working node

[lreiher]

Please understand that at the moment we have no capacity to publicly release the inference node.

We can however give you some hints on what such an inference node would need to do:

1. Load parameters
   • path to exported TensorFlow model
   • paths to one-hot-conversion files for input/label
2. Setup node
   • load TensorFlow model
   • parse one-hot-conversion palettes
   • create as many image subscribers as you have cameras
   • synchronize image subscribers using message_filters and register callback function
   • create publisher for output image
3. Callback/Inference Function
   • create input tensors from image messages
     • convert ROS image message to CV images using cv_bridge
     • resize images, if needed
     • one-hot-encode images using the parsed palette
   • run inference using TensorFlow C++ Library
   • create output image message from output tensor
     • one-hot-decode output tensor using the parsed palette
     • convert tensor to ROS image
   • publish output image message