Adding yolov8 masks to detections

[hardikdava]

Description

Current Detection class only support for yolov8 object detection. Segmentation masks information are missing.

Type of change

Please delete options that are not relevant.

• New feature (non-breaking change which adds functionality)

How has this change been tested, please provide a testcase or example of how you tested the change?

model = YOLO(model="yolov8s.pt")
results = model.predict(img)[0]
detections = sv.Detections.from_yolov8(results)

mask_annotator = sv.MaskAnnotator()
box_annotator = sv.BoxAnnotator()

img = box_annotator.annotate(scene=img, detections=detections)
img = mask_annotator.annotate(scene=img, detections=detections)

Docs

• Docs updated? What were the changes:

[SkalskiP]

Hi 👋🏻 @hardikdava! We can't use ultralytics code directly like that :/ The moment we will do it, we need to change the LICENSE of supervision to a very restrictive YOLOv8 LICENSE.

But I took a look at the scale_image implementation. A lot of it is padding logic that we don't use. And if I'm not mistaken, all we need is :

masks = cv2.resize(masks, (im0_shape[1], im0_shape[0]))

Am I right?

[hardikdava]

Yes, @SkalskiP . I tested by just using cv2.resize function. It works fine without using any functionality from ultralytics. I will make necessary changes.

[SkalskiP]

Hi, @hardikdava 👋🏻! I took a look at the PR. I wanted to add this myself, so the PR is very much welcomed. I see one problem, however. We cant use ultralytics package directly in our code because the moment we do it, we need to change our license from MIT to a very restrictive YOLOv8 license. Please take a look at my comment for more details.

[hardikdava]

@SkalskiP I made necessary changes. Now take a look and let me know if I need to change anything else.

[SkalskiP]

Hi @hardikdava! First of all sorry for such a late response. I was quite busy. The code looks good. 💜 Let me test it quickly and if it works we are merging.

[SkalskiP]

Hi, @hardikdava just tested, and I'm getting: NameError: name 'cv2' is not defined. Looks like you didn't do the cv2 import at the top of the file.

[SkalskiP]

When I run the code I get:

0: 640x384 1 person, 1 car, 1 dog, 1 backpack, 1 handbag, 153.3ms
Speed: 15.4ms preprocess, 153.3ms inference, 37.7ms postprocess per image at shape (1, 3, 640, 640)
WARNING ⚠️ 'Masks.masks' is deprecated. Use 'Masks.data' instead.
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
[<ipython-input-10-6c1542ea3727>](https://localhost:8080/#) in <cell line: 2>()
      1 results = model.predict(source=image)[0]
----> 2 detections = sv.Detections.from_yolov8(results)

[/content/supervision/supervision/detection/core.py](https://localhost:8080/#) in from_yolov8(cls, yolov8_results)
    192             masks = yolov8_results.masks.masks.cpu().numpy()  # masks, (N, H, W)
    193             masks = np.moveaxis(masks, 0, -1)  # masks, (H, W, N)
--> 194             masks = cv2.resize(masks, (yolov8_results.masks.orig_shape[1], yolov8_results.masks.orig_shape[0]))
    195             masks = np.moveaxis(masks, -1, 0)  # masks, (N, H, W)
    196         return cls(

NameError: name 'cv2' is not defined

[hardikdava]

@SkalskiP I have fixed the error. Please feel free to change any part of the code.

[SkalskiP]

Let me test once again.

[SkalskiP]

I think something still does not work as expected. I've done some tests. Here are two images. First produced by YOLOv8 annotators and second by our annotators. Looks like masks are not scaled properly for some reason.

• YOLOv8 result:

• Supervision result:

[hardikdava]

@SkalskiP it is due to letterbox. Padding is not zero in this case. We need an information of inference input size.

[SkalskiP]

@hardikdava is it possible to extract that info from results?

[hardikdava]

@SkalskiP I rewrote the logic of mask scalling. Please test it again. I have tested it and attached the results (left=supervision result, right=yolov8 result).

[hardikdava]

@SkalskiP any update? I think we are ready to merge it.

[SkalskiP]

Hi, @hardikdava 👋🏻 ! There are a few formatting changes I'd do, but I don't want to bother you with those. I'll take care of it on my own. We are merging.