Error on loading onnx model

[jducnuigeen]

I've managed to get all the way to training on my own imagery and creating my first instance seg model and converted to onnx and added metadata using your add_model_metadata.py script.
I used Ultralytics yolov8 for training. I assume this is ok since your documentation said
Currently plugin supports YOLOv5, YOLOv6, YOLOv7, YOLOv9 and ULTRALYTICS output types. Detection model also supports the instance segmentation output type from ULTRALYTICS.
I assume this is ok even because the output was produced by Ultralytics as in

from ultralytics import YOLO
#Instance
model = YOLO('yolov8n-seg.pt')  # Transfer the weights from a pretrained model (recommended for training)

Or is this not correct, and I'd have to use ultralytics yolov9?
This is the error I get when I try to load my onnx into deepness:

I can send a link to the onnx model if that helps.

[bartoszptak]

Could you share a screenshot of your Deepness configuration and the model fine in this thread or send me by mail available in my profile? I'll check it tomorrow

[bartoszptak]

Thanks, I'll check it in spare time. One thing - YOLO's instance segmentation is not a pure Segmentor model - please, set the Detector type and then select detector type below to "YOLO_Ultralytics_segmentation" and try again - maybe it helps

[jducnuigeen]

Ok, I restarted QGIS and set the Detector type and Yolo_Ultralytics_segmentation, but on load of onnx model get the same error. Thank you very much for looking into it.
I plan to write up my full model training workflow and model and share it if I can get it to work.
Cheers.

[bartoszptak]

Hi, @jducnuigeen again!
I deeply got into the issue and discovered two things:

Model type issue

Your model has set the "model_type" metadata to Segmentor, so it is set automatically to it during model loading and it overwrite your manual change in QGIS (I used netron.app to the visualisation below):

Instance segmentation is not a pure segmentation task because it is based on object detection. Therefore, you should set following metadata:

Python 3.10.12 (main, Sep 11 2024, 15:47:36) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import json
>>> import onnx
>>> 
>>> model = onnx.load('train4_1000.onnx')

>>> m1 = model.metadata_props.add()
>>> m1.key = 'model_type'
>>> m1.value = json.dumps('Detector')

>>> m2 = model.metadata_props.add()
>>> m2.key = 'det_type'
>>> m2.value = json.dumps('YOLO_Ultralytics_segmentation')

>>> onnx.save(model, 'train4_1000_fixed.onnx')

You can download the changed model here, but I encourage you to update your pipeline and keep in mind that on the implementation side, the instance segmentation task is closer to object detection than semantic segmentation.

Onnxruntime incompatibility

I was confused because your model, after fixes, works correctly in the test environment, but it still crashes in QGIS. I discovered that my onnruntime library in QGIS is 1.17.0 while in testing env 1.19.2. When I downgraded the library to 1.17.0, I obtained the error:

onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Load model from /home/host/Downloads/train4_1000_fixed.onnx failed:/onnxruntime_src/onnxruntime/core/graph/model.cc:179 onnxruntime::Model::Model(onnx::ModelProto&&, const onnxruntime::PathString&, const onnxruntime::IOnnxRuntimeOpSchemaRegistryList*, const onnxruntime::logging::Logger&, const onnxruntime::ModelOptions&) 
Unsupported model IR version: 10, max supported IR version: 9

This was caused by the API change in the ONNX framework. Probably, Ultralytics does not care about back compatibility. I think. you can fix it in two ways:

• in the environment where you convert the Pytorch (.pt) model to ONNX (.onnx), downgrade of the onnx according to the comment

• update the ONNXRuntime library in QGIS:

  1. Open the Python console in QGIS (Ctrl + Alt + P)
  2. Import the PIP library by typing import pip
  3. Then, update ONNXRuntime by typing pip.main(['install', '--upgrade', 'onnxruntime'])
  4. You should see the last message, such as Successfully installed onnxruntime-1.19.2
  5. Restart QGIS

Final result

Thanks for your contribution. I look forward to your tutorial. Let me know if my suggestions are helpful and fix all issues!

[jducnuigeen]

Wow, thank you so very much ! I will go through this all carefully and let you know how it went - and then do my write up. Cheers !

[jducnuigeen]

Ok, those changes worked ! Thank you !
I still have a lot to tinker with to hopefully improve the detection and output: Confidence #, IoU threshold, tiles overlap, far more training images, and model type - all of which I will have to research to understand.

Two general questions:

1. You mention "instance segmentation task is closer to object detection than semantic segmentation". You suggested I change my metadata on the Ultralytics YoloV8 I trained on to "Detector" instead of "Segmentor" . Of the model types Deepness supports
   plugin supports YOLOv5, YOLOv6, YOLOv7, YOLOv9 and ULTRALYTICS output types. Detection model also supports the instance segmentation output type from ULTRALYTICS
   are there particular versions of Yolo which would provide instance segmentation using the Deepness "Segmentor", and if so, should I consider training with a different type to obtain possibly better results? Or is the "Detector" option likely to be about the same over other models and model types?

2. Looking at some of my output:
   
   I see

• very few false positive detections (great !!)
• some issues with instance segmentation not following the sharp corners of the buildings
• multiple polygons on top of each other
• missed building

Do you have a suggestions on what I should spend time on in order to obtain better results? Should I prioritize my time on one or more of these items over others?

A. train on different model, then choosing segmentor model type over detector maybe?
B. train with more training images (I started with 1000, which isn't large, but I have up to 65,000 images I could use)
C. increase or decrease tile overlap percentage
D. change/increase the contrast of my training imagery/ detection imagery (admittedly the colour is very flat in appearance)
E. try more variations of confidence/IoU threshold (for Detector)
F. larger tile size (I used 512x512 but I could increase this)
G. something else?

Instinct tells me I should train a new yolov7 instance segmentation model on the same training data and run that as a segmentor in Deepness - as I see the resulting output in Deepness is processed differently (ie map_processor_segmentation (mask_img) vs map_processor_detection (bounding_boxes: List[Detection]) ).

Any thoughts would be helpful, and I appreciate all of your spare time on this, and don't want to ask too much more of your time. The more I learn of different choices I've made my tutorial will be a bit more informative for new users.