is there a way to make masks not to overlap?

[polinamalova0]

Search before asking

• I have searched the YOLOv8 issues and discussions and found no similar questions.

Question

Dear Ultralytics team,
After training my instance segmentation model, i tried to predict the objects and obtained the masks. However, the cases where they overlay one on another are very common. Therefore my question is: is there a way to set the prediction so that the masks are not overlapping (exclusive for each object)?

Additional

No response

[glenn-jocher]

Hello,

Thank you for reaching out! To prevent instance masks from overlapping in YOLOv8, you can set the overlap_mask argument to False when using the train or predict commands. Here's an example of how to implement it during prediction:

yolo predict model=path/to/your_model.pt source=path/to/your_image.jpg overlap_mask=False

This will ensure that each mask is exclusive to its respective detected object. Let us know if you need further assistance!

[polinamalova0]

Hello,

Thank you for reaching out! To prevent instance masks from overlapping in YOLOv8, you can set the overlap_mask argument to False when using the train or predict commands. Here's an example of how to implement it during prediction:

yolo predict model=path/to/your_model.pt source=path/to/your_image.jpg overlap_mask=False

This will ensure that each mask is exclusive to its respective detected object. Let us know if you need further assistance!

@glenn-jocher
My problem is that i used it both during training process:

results = model.train(data=r'path/dataset.yaml', epochs=100,  imgsz=(428,428), device = device, overlap_mask=False, verbose=True)

as well as during prediction:

model = YOLO(model_path)
results = model(img, overlap_mask=False, save=True)

but it still shows me overlapping masks.

[glenn-jocher]

@polinamalova0 hello,

Thanks for the detailed information! It seems like the overlap_mask=False should indeed prevent the masks from overlapping. If it's still not working as expected, it might be a good idea to ensure that your model and the Ultralytics YOLOv8 library are both up to date. Sometimes, such issues can be resolved with the latest updates.

If updating doesn't resolve the issue, it could be helpful to check if there's any custom modification in your model or data processing pipeline that might be influencing the mask behavior.

Please let us know if the problem persists after checking these points!

[polinamalova0]

@polinamalova0 hello,

Thanks for the detailed information! It seems like the overlap_mask=False should indeed prevent the masks from overlapping. If it's still not working as expected, it might be a good idea to ensure that your model and the Ultralytics YOLOv8 library are both up to date. Sometimes, such issues can be resolved with the latest updates.

If updating doesn't resolve the issue, it could be helpful to check if there's any custom modification in your model or data processing pipeline that might be influencing the mask behavior.

Please let us know if the problem persists after checking these points!

Unfortunately, i updated the libraries as well as re-checked all my annotations to make sure that none overlap, and the newly trained model still gives off overlapped masks as a result... is there another way that can help me with the problem?

[glenn-jocher]

Hello @polinamalova0,

Thank you for the update and for checking those aspects. It's puzzling that the issue persists despite these efforts. Here are a couple more steps we can try:

1. Debugging Output: Could you please run a few test predictions and share the output logs? Sometimes, the logs can provide insights into what might be going wrong.

2. Configuration Check: Double-check the configuration file for any overlooked settings that might affect mask generation.

3. Isolate the Issue: Try running a prediction with a minimal dataset (perhaps even synthetic data where you control overlap completely) to see if the issue is data-dependent.

4. Community Input: Consider posting this issue in the Ultralytics discussions forum. Sometimes, community members may have faced and resolved similar issues.

We're here to help you through this, so keep us posted on your progress! 🚀

[polinamalova0]

@glenn-jocher thank you for the answer. if i got right what you meant by "output logs", this is what i got before training info:
for results = model.train(data=r'Z:\Data\ODT\000_Paula\YOLOv8\labelme_json_dir\YOLODataset\dataset.yaml', epochs=100, imgsz=(428,428), device = device, overlap_mask=False, verbose=True, workers = 1)
i got:

New https://pypi.org/project/ultralytics/8.2.28 available 😃 Update with 'pip install -U ultralytics'
Ultralytics YOLOv8.2.16 🚀 Python-3.10.4 torch-2.3.0+cu121 CUDA:0 (NVIDIA GeForce RTX 2070, 8192MiB)
engine\trainer: task=segment, mode=train, model=yolov8s-seg.pt, data=Z:\Data\ODT\000_Paula\YOLOv8\labelme_json_dir\YOLODataset\dataset.yaml, epochs=100, time=None, patience=100, batch=16, imgsz=(428, 428), save=True, save_period=-1, cache=False, device=cuda, workers=1, project=None, name=train24, exist_ok=False, pretrained=True, optimizer=auto, verbose=True, seed=0, deterministic=True, single_cls=False, rect=False, cos_lr=False, close_mosaic=10, resume=False, amp=True, fraction=1.0, profile=False, freeze=None, multi_scale=False, overlap_mask=False, mask_ratio=4, dropout=0.0, val=True, split=val, save_json=False, save_hybrid=False, conf=None, iou=0.7, max_det=300, half=False, dnn=False, plots=True, source=None, vid_stride=1, stream_buffer=False, visualize=False, augment=False, agnostic_nms=False, classes=None, retina_masks=False, embed=None, show=False, save_frames=False, save_txt=False, save_conf=False, save_crop=False, show_labels=True, show_conf=True, show_boxes=True, line_width=None, format=torchscript, keras=False, optimize=False, int8=False, dynamic=False, simplify=False, opset=None, workspace=4, nms=False, lr0=0.01, lrf=0.01, momentum=0.937, weight_decay=0.0005, warmup_epochs=3.0, warmup_momentum=0.8, warmup_bias_lr=0.1, box=7.5, cls=0.5, dfl=1.5, pose=12.0, kobj=1.0, label_smoothing=0.0, nbs=64, hsv_h=0.015, hsv_s=0.7, hsv_v=0.4, degrees=0.0, translate=0.1, scale=0.5, shear=0.0, perspective=0.0, flipud=0.0, fliplr=0.5, bgr=0.0, mosaic=1.0, mixup=0.0, copy_paste=0.0, auto_augment=randaugment, erasing=0.4, crop_fraction=1.0, cfg=None, tracker=botsort.yaml, save_dir=runs\segment\train24
Overriding model.yaml nc=80 with nc=2

                   from  n    params  module                                       arguments                     
  0                  -1  1       928  ultralytics.nn.modules.conv.Conv             [3, 32, 3, 2]                 
  1                  -1  1     18560  ultralytics.nn.modules.conv.Conv             [32, 64, 3, 2]                
  2                  -1  1     29056  ultralytics.nn.modules.block.C2f             [64, 64, 1, True]             
  3                  -1  1     73984  ultralytics.nn.modules.conv.Conv             [64, 128, 3, 2]               
  4                  -1  2    197632  ultralytics.nn.modules.block.C2f             [128, 128, 2, True]           
  5                  -1  1    295424  ultralytics.nn.modules.conv.Conv             [128, 256, 3, 2]              
  6                  -1  2    788480  ultralytics.nn.modules.block.C2f             [256, 256, 2, True]           
  7                  -1  1   1180672  ultralytics.nn.modules.conv.Conv             [256, 512, 3, 2]              
  8                  -1  1   1838080  ultralytics.nn.modules.block.C2f             [512, 512, 1, True]           
  9                  -1  1    656896  ultralytics.nn.modules.block.SPPF            [512, 512, 5]                 
 10                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']          
 11             [-1, 6]  1         0  ultralytics.nn.modules.conv.Concat           [1]                           
 12                  -1  1    591360  ultralytics.nn.modules.block.C2f             [768, 256, 1]                 
 13                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']          
 14             [-1, 4]  1         0  ultralytics.nn.modules.conv.Concat           [1]                           
 15                  -1  1    148224  ultralytics.nn.modules.block.C2f             [384, 128, 1]                 
 16                  -1  1    147712  ultralytics.nn.modules.conv.Conv             [128, 128, 3, 2]              
 17            [-1, 12]  1         0  ultralytics.nn.modules.conv.Concat           [1]                           
 18                  -1  1    493056  ultralytics.nn.modules.block.C2f             [384, 256, 1]                 
 19                  -1  1    590336  ultralytics.nn.modules.conv.Conv             [256, 256, 3, 2]              
 20             [-1, 9]  1         0  ultralytics.nn.modules.conv.Concat           [1]                           
 21                  -1  1   1969152  ultralytics.nn.modules.block.C2f             [768, 512, 1]                 
 22        [15, 18, 21]  1   2771318  ultralytics.nn.modules.head.Segment          [2, 32, 128, [128, 256, 512]] 
YOLOv8s-seg summary: 261 layers, 11790870 parameters, 11790854 gradients, 42.7 GFLOPs

Transferred 411/417 items from pretrained weights
Freezing layer 'model.22.dfl.conv.weight'
AMP: running Automatic Mixed Precision (AMP) checks with YOLOv8n...
C:\Users\labadmin\PycharmProjects\pythonProject\venv\lib\site-packages\torch\nn\modules\conv.py:456: UserWarning: Plan failed with a cudnnException: CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR: cudnnFinalize Descriptor Failed cudnn_status: CUDNN_STATUS_NOT_SUPPORTED (Triggered internally at ..\aten\src\ATen\native\cudnn\Conv_v8.cpp:919.)
  return F.conv2d(input, weight, bias, self.stride,
AMP: checks passed ✅
WARNING ⚠️ updating to 'imgsz=428'. 'train' and 'val' imgsz must be an integer, while 'predict' and 'export' imgsz may be a [h, w] list or an integer, i.e. 'yolo export imgsz=640,480' or 'yolo export imgsz=640'
WARNING ⚠️ imgsz=[428] must be multiple of max stride 32, updating to [448]
train: Scanning \\syn03.frontend.storage.int.mpl.mpg.de\guck_division2\Data\ODT\000_Paula\YOLOv8\labelme_json_dir\YOLODataset\labels\train.cache... 146 images, 0 backgrounds, 0 corrupt: 100%|██████████| 146/146 [00:00<?, ?it/s]
Using device: cuda
val: Scanning \\syn03.frontend.storage.int.mpl.mpg.de\guck_division2\Data\ODT\000_Paula\YOLOv8\labelme_json_dir\YOLODataset\labels\val.cache... 36 images, 0 backgrounds, 0 corrupt: 100%|██████████| 36/36 [00:00<?, ?it/s]
Using device: cuda
Using device: cuda
Plotting labels to runs\segment\train24\labels.jpg... 
module 'backend_interagg' has no attribute 'FigureCanvas'
optimizer: 'optimizer=auto' found, ignoring 'lr0=0.01' and 'momentum=0.937' and determining best 'optimizer', 'lr0' and 'momentum' automatically... 
optimizer: AdamW(lr=0.001667, momentum=0.9) with parameter groups 66 weight(decay=0.0), 77 weight(decay=0.0005), 76 bias(decay=0.0)
Image sizes 448 train, 448 val
Using 1 dataloader workers
Logging results to runs\segment\train24
Starting training for 100 epochs...

as you might understand, the problem with overlap is persistent. I am using a small dataset, and i checked myself the annotation, there's also no overlap in the annotated images. I don't know where it might come from.

[glenn-jocher]

Hello @polinamalova0,

Thank you for providing the detailed logs and for your efforts in troubleshooting this issue. It's clear you've taken the necessary steps to ensure your setup and data are correctly configured. Given that the problem persists despite these measures, there might be an underlying issue with how the overlap_mask=False setting is being handled in the code or a bug that's not immediately apparent.

At this point, I recommend the following steps:

1. Review the Source Code: If possible, review the sections of the source code related to mask generation and the handling of the overlap_mask parameter. There might be a conditional check or a default setting overriding your configuration.

2. Raise an Issue on GitHub: Since this could potentially be a bug or an undocumented behavior, consider raising an issue on the Ultralytics YOLOv8 GitHub repository. Provide the details you've shared here, including your configuration and the unexpected behavior. The development team or other users may have insights or workarounds.

3. Community and Support Channels: Utilize the Ultralytics community channels such as the GitHub discussions or the Discord server. Other users might have encountered and resolved similar issues.

4. Temporary Workaround: As a temporary measure, you might consider post-processing the masks in your application to enforce non-overlapping conditions manually until a permanent fix or explanation is available.

We appreciate your patience and are here to assist you further as needed. Your detailed feedback is invaluable in improving YOLOv8 and helping us identify areas that may require attention.

[polinamalova0]

@glenn-jocher i am not sure it is a source code issue, since my code for training and predicting is like that:

from ultralytics import YOLO
import torch

device = 'cuda' if torch.cuda.is_available() else 'cpu'
print(f'Using device: {device}')

model = YOLO('yolov8s-seg.pt').to(device)

if __name__ == '__main__':
results = model.train(data=r'path\to\dataset.yaml', epochs=100,  imgsz=(428,428), device = device, overlap_mask=False, verbose=True, workers = 1)

path = r'C:\Users\labadmin\PycharmProjects\pythonProject\runs\segment\train25\weights\last.pt'
img = r'path\to\img.png'
model = YOLO(path)
prediction= model(img, save=True, overlap_mask=False, verbose =True)

[glenn-jocher]

Hello @polinamalova0,

Thank you for sharing your code snippet and providing detailed context. Let's work together to resolve this issue.

First, let's ensure that the bug occurs with the most recent versions of torch and ultralytics. Could you please confirm that you are using the latest versions? If not, please update your packages and try again:

pip install -U torch ultralytics

If the issue persists after updating, we need to verify that the problem can be reproduced. Could you provide a minimum reproducible example? This will help us investigate the issue more effectively. You can find guidelines on creating a minimum reproducible example here.

Here's a simplified version of your code to ensure we are on the same page:

from ultralytics import YOLO
import torch

device = 'cuda' if torch.cuda.is_available() else 'cpu'
print(f'Using device: {device}')

model = YOLO('yolov8s-seg.pt').to(device)

# Training
results = model.train(data='path/to/dataset.yaml', epochs=100, imgsz=428, device=device, overlap_mask=False, verbose=True, workers=1)

# Prediction
path = 'path/to/last.pt'
img = 'path/to/img.png'
model = YOLO(path)
prediction = model(img, save=True, overlap_mask=False, verbose=True)

If you have already ensured that your annotations are correct and there are no overlaps, and the issue still persists, it might be helpful to check if there are any specific conditions or edge cases in your dataset that could be causing this behavior.

Additionally, you can try visualizing the masks during training to see if the overlap issue starts there or only appears during prediction. This might give us more clues about where the problem lies.

If you continue to face issues, please consider raising an issue on our GitHub repository with the minimum reproducible example. This will allow the development team to investigate further and provide a solution.

Thank you for your patience and cooperation. We're here to help you get this resolved! 😊