Custom model integration

[jp3spinelli]

• rpi-deep-pantilt version: Latest version
• Python version: 3.7.3
• TensorFlow version: 2.2.0
• Operating System: Raspbian

I am trying to use this to get my pan/tilt hat to track an opposing Raspberry Pi camera. I have trained my custom model and converted it to TFlite, I just need to know how to integrate my model into this repository to be able to track. I trained my model on ssd_mobilenet_v3_small_coco_2019_08_14.tar.gz, which you use, so I figure there is some way to tweak the files to make this work.
Thanks!

[jp3spinelli]

I found three scripts that include the word 'Tracking': camera.py, ssd_mobilenet_v3_coco.py, visualization.py
I also figured I may need to specify the model being referred to in ssd_mobilenet_v3.py
Lastly, I know I need to fix the label maps, but I need direction on all this.

[leigh-johnson]

Hey @jp3spinelli! You'll need to create a class similar to SSDMobileNet_V3_Coco_EdgeTPU_Quant or SSDMobileNet_V3_Small_Coco_PostProcessed

https://github.com/leigh-johnson/rpi-deep-pantilt/blob/master/rpi_deep_pantilt/detect/ssd_mobilenet_v3_coco.py

For visualize_boxes_and_labels_on_image_array to work correctly, you'll need to create a .pbtxt file with labels that correspond to the classes your network is trained to recognize.
https://github.com/leigh-johnson/rpi-deep-pantilt/blob/master/rpi_deep_pantilt/data/mscoco_label_map.pbtxt

I'm also using the TFLite_Detection_PostProcess op as an output layer, which contains a non-max suppression implementation specifically for models in TensorFlow's object detection zoo.
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/kernels/detection_postprocess.cc

Here's how I add this op with an older TFlite toco converter. Word of warning, this might not work with the newer MLIR-based TFlite converter.
https://github.com/leigh-johnson/rpi-deep-pantilt/blob/master/tools/tflite-postprocess-ops-float.sh

After you have a working model class setup, you can call pantilt_process_manager with that class and the detection/track loop will run.
https://github.com/leigh-johnson/rpi-deep-pantilt/blob/master/rpi_deep_pantilt/cli.py#L146

You'll probably have to adjust the default PID gains, since these are tuned to track a human-sized object from a camera at roughly desk height. My notes on PID controllers are below as well.
https://github.com/leigh-johnson/rpi-deep-pantilt/blob/master/rpi_deep_pantilt/control/manager.py#L104
https://miro.medium.com/max/1400/1*9sQpP7SqMHrhAwySJBJffQ.png

Good luck!

[leigh-johnson]

Closing for now, so it's easier for me to see issues in need of triage at a glance. I'd love to see your final setup when you get this working!