Performance issue high resolution image using detectron2

[TTkgl]

Hello,
I currently experience performance issues using a Mask R-CNN trained with detectron2 on custom data.
My goal is to use SAHI, because the image resolution is around 4000x6000px. So my preferation would be to use it with a slice width&height of 640px.

result = predict( model_type='detectron2', model_path=model_path, model_config_path=config_path, model_device="cuda:0", source="predict_images/testData/img01.jpg", no_standard_prediction=True, slice_height=640, slice_width=640, overlap_height_ratio=0.2, overlap_width_ratio=0.2, verbose=2, novisual=True )

Using this configuration, the prediction takes about 120 seconds for a number of 120 slices.

If I slice the pictures using sahi and pass the folder containing the images to predict, the whole prediction process of all 120 images just takes 12(!) seconds instead of 120. This time consumption of 12s would be optimal for my use-case.

Is anyone aware of the possible issue and if so, knows a solution to it, I would be happy to hear about it.
Otherwise, if you need additional information please let me know.
If it helps, I can provide the output of cProfiler.

[TTkgl]

Hello,
I found out, that when you slice the big image into smaller slices (e.g., 640px x 640px) using slice_image and then perform a batch prediction as follows:
result = predict( model_type='detectron2', model_path=model_path, model_config_path=config_path, image_size=640, model_device="cuda", source="predict_images/slices/", no_standard_prediction=True, slice_height=640, slice_width=640, overlap_height_ratio=0.2, overlap_width_ratio=0.2, verbose=2, novisual=False )

Then, SAHI will perform a prediction on each of the images. For me, as seen in the below image, the whole prediction which beforehand took ~300sec now took place in 10.992 seconds and 8.34 it/s resulting in ~120 ms average image prediction time.
(The prediction result saved contains correct predicitions, so the batch prediction works)
Is it possible that this is a issue regarding the sahi implementation?

[ClemSc]

@TTkgl Interesting. I was in the process of wondering how this function was working. I am not the person for optimizing this unfortunately (not enough time, not enough skills).
I see that there is two calls to get_shifted_mask() per mask shifting (in annotation.get_shifted_object_prediction). Making only one call save me a few seconds...

I am then bit lost on how those "full shape" raster masks are used in the merging process...

[TTkgl]

@ClemSc Does the following work for you too?
If you change use_rle in annotation.py to False, the program does not use the changes made in (#599) which will lead to a higher ram usage. Besides this disadvantage of a higher ram usage, my prediction was ~40 seconds faster. (70sec -> 32sec)

[ClemSc]

@TTkgl Yes ! Much faster. First picture went down from 88 sec to 25 sec.

[TTkgl]

@ClemSc Also check #888 . This took my process down to 13sec.

[ClemSc]

@TTkgl Nice. The implementation using the masks as polygons makes more sense to me. I was actually confused to find out about the full shape array masks. I am relieved to see that someone else tackled it already ;-)
Thank you for finding this out, this is super helpful !

[fcakyon]

Segmentation mask support for yolov8, mmdetection, detectron2, torchvision models is live with the latest release! @TTkgl #1051

There is also 5x memory and speed improvement when working with masks!

Check the demo notebook on how it works: https://github.com/obss/sahi/blob/main/demo/inference_for_detectron2.ipynb