Inference failure after converting to .blob

[aiManny]

I've performed some custom training of the yolov5 network with 8 object classes by following this tutorial: yolov5_training.ipynb

Inference works perfectly using the .pt file on my host laptop, and http://tools.luxonis.com/ generates the .blob file with no issues.

From there I run the main_api.py found at gen2-yolo to perform OAK-D inference, but the preview window is immediately cluttered by erroneous bounding boxes:

I've already tried adjusting the iou_threshold value in my configuration .json with no effect. Is there a pre-processing step I should've done before generating the .blob?

[tersekmatija]

Hey @aiManny ,

This seems like a config issue. Which DepthAI version are you using? Can you share the command you use to call the main_api.py? And can you share the config so I can take a quick look and see if there might be something obviously wrong going on?

[aiManny]

I'm using DepthAI version 2.20.2.0. To run main_api.py I entered the command:

python main_api.py --model ./saved_models/cube/yolov5/yolov5n_300.blob --config ./saved_models/cube/yolov5/yolov5n_300.json

Since Github won't allow me to upload a JSON, here's the config file as a .txt:
config.txt

[tersekmatija]

DepthAI version should be fine. Just in case please try updating to the latest version.

The command also looks OK and I can't spot any issues from the config itself. Is it possible to share the whole output ZIP from the tools?

[aiManny]

I did go ahead and upgrade to 2.21.2.0 but the issue persists.
Here's the ZIP output, minus the .onnx file because it was too large
results_condensed.zip

[HonzaCuhel]

Hi @aiManny,

I apologize for the delay in my response. I just tried our YoloV5 Colab training notebook and trained YoloV5s for 1 epoch. The performance wasn't impressive, but I didn't notice any issues similar to yours. Could you maybe elaborate more about how you trained the model, please?

Thanks.

Best,
Jan

[aiManny]

Sure thing, but first a quick question. Were you able by any chance to test the yolov5n_300.json and yolov5n_300_openvino_2022.1_6shave.blob from the zip I posted in my previous comment?

I generated it by following the luxonis YoloV5_training.ipynb found here. The main difference was that I ran Ultralytics yolov5/segment/train.py script instead of yolov5/train.py because I'm interested in instance segmentation as opposed to just object detection.

I trained for 300 epochs on my custom dataset. For evaluation, I run the yolov5/segment/predict.py after making some display customizations and it gives great results for my .pt weights. Here you can see the original image on the left, and the inference results with instance masks and bounding boxes on the right.

I only run into trouble when trying to run the converted .blob on my OAK-D.

[HonzaCuhel]

@aiManny thank you for the great description! Now we know what is the issue here. Yes, I tried the exported model you shared and got the same results as you did. The problem is that, unfortunately, as of now, our tools doesn't support conversion of instance segmentation models, just object detection ones. It is something that we want to look into in the future, but we don't have an estimated release date.

I am sorry for your trouble.

Best,
Jan

[aiManny]

I was beginning to suspect that was the case. Thank you for confirming. Do you know if the other luxonis BlobConveter should be able to handle instance segmentation models?

[tersekmatija]

Hey @aiManny ,

We are looking to add native support for Yolo-like instance segmentation. In terms of export, I believe it should be pretty easy, however, there will be some FW work we need to do. You should be able to export it yourself if you would manage the decoding yourself as well (NMS, mask computation). You could try pruning the model before the NMS, then converting it to ONNX and using blobconverter to compile it. Note that you will still need a host to process the results and perform then NMS in that case.

[aiManny]

Sounds good! I might have to look into that. I did go ahead and retrain my weights for detection-only and was able to get some great results with the .blob.

As a note to posterity, the YOLO training pipeline loads images in the BGR format. My color-based detections on the OAK-D were still poor until I realized I needed to set the OAK-D colorspace to BGR as well. Like so:

camRgb = pipeline.create(dai.node.ColorCamera)
camRgb.setColorOrder(dai.ColorCameraProperties.ColorOrder.BGR)