The masks generated(Segmentation) are not accurate on my custom dataset. How to improve that?

[qaz0778899]

Search before asking

• I have searched the YOLOv8 issues and discussions and found no similar questions.

Question

Hey everyone.
I'm trying to solve an segmentation problem with yolov8x-seg model. However the map generated are not accurate in some specific circumstances. Here is my story. I trained the model for 1000 epochs first(it converged at the 814 epochs and stoped training). I ran it on the test set, and it turns out working very well excepting there are certian types of objects are not accurately segmented(mainly the vertical lying panels).

I'm a newbie in detection and segmentation area. Here are the thoughts I had about this problem and please correct if I'm wrong.

1. I firstly thought it was caused by imbalance of the instances of my training set, so I added more images containing the types of instances cannot be segmented correctly. I trained it for further 2000 epochs but nothing was learned(I increased the patience from 50 to 200 otherwise it stops training at the 51 epochs immediately).
2. Then I thought it got stuck in an local optimal and the learning rate might be too large to get the optimal solution. Therefore, I tried to lower the initial lrs and weight_decays but still not working.
3. I notice the seg_loss and cls_loss became very low while the box_loss and dfl_loss are relatively higher. I was thinking about adding coeffients to seg and cls losses to balance the losses, so it might stops focusing on updating the other two? I looked at the code and didn't find the lines of code for the loss calculations. Can anyone help me out?
4. Any suggetions would be appreciated. Guys help me please. (>0<)

Additional

No response

[github-actions]

👋 Hello @qaz0778899, thank you for your interest in YOLOv8 🚀! We recommend a visit to the YOLOv8 Docs for new users where you can find many Python and CLI usage examples and where many of the most common questions may already be answered.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset image examples and training logs, and verify you are following our Tips for Best Training Results.

Join the vibrant Ultralytics Discord 🎧 community for real-time conversations and collaborations. This platform offers a perfect space to inquire, showcase your work, and connect with fellow Ultralytics users.

Install

Pip install the ultralytics package including all requirements in a Python>=3.8 environment with PyTorch>=1.8.

pip install ultralytics

Environments

YOLOv8 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all Ultralytics CI tests are currently passing. CI tests verify correct operation of all YOLOv8 Modes and Tasks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

[SilverStarCoder]

@qaz0778899 , thank you for sharing the results of your experiments. Can you also provide details on the original image sizes and the image sizes used for training?

[qaz0778899]

@qaz0778899 , thank you for sharing the results of your experiments. Can you also provide details on the original image sizes and the image sizes used for training?

Hi, SilverStarCoder. Thanks for replying me. The sizes of the original images are arbitrary. They vary in 600-1500 x 400-800. I was training the model with imgsz=1280(I tried 680 before, but the bountries of the generated maps are less accurate than 1280).

[SilverStarCoder]

@qaz0778899 , I think I might be facing a similar issue to you, which I asked here. I suspect it has to do with the size of the anchor boxes that YOLOv8 uses, which doesn't adapt well to shapes that are too big/long. I think you might be able to confirm this by plotting the prediction results of those images. When I plotted the results from the predictions.json file, I noticed that it generated many boxes that all cover parts of the 'big' object, but not one box that full encloses the entire shape. I am also a newbie in segmentation, and hopefully someone will be able to help resolve your issue.
🙂

[qaz0778899]

@qaz0778899 , I think I might be facing a similar issue to you, which I asked here. I suspect it has to do with the size of the anchor boxes that YOLOv8 uses, which doesn't adapt well to shapes that are too big/long. I think you might be able to confirm this by plotting the prediction results of those images. When I plotted the results from the predictions.json file, I noticed that it generated many boxes that all cover parts of the 'big' object, but not one box that full encloses the entire shape. I am also a newbie in segmentation, and hopefully someone will be able to help resolve your issue. 🙂

I got it!!! I found my solution here. #2843 (comment) Yeah, you are right. The problem is caused by the hidden limitation of maximum length/width of the bounding boxes in yolov8. There is a variable named self.reg_max in ultralytics/nn/modules/head.py limits the size of the boxes. I changed the value from 16 to 32 and solved my problem. You might want to play with it accordingly based on the sizes of objects and the resolutions of images in your dataset.

[SilverStarCoder]

@qaz0778899 , it is great that you were able to find a solution that worked! I will try it myself. If you don't mind me asking, since the adjustment is in nn/modules/head.py, does that mean we have to rebuild the model using the yolov8-seg.yaml file?

[qaz0778899]

@qaz0778899 , it is great that you were able to find a solution that worked! I will try it myself. If you don't mind me asking, since the adjustment is in nn/modules/head.py, does that mean we have to rebuild the model using the yolov8-seg.yaml file?

No, I just loaded the model I trained previously and trained it for a couple of epochs after modifying the head.py file. There is no need to train from the scratch. (bs. I also adjust the learning rate and decay to smaller values to fine-tune my model, but I feel it doesn't matter because it never try to use these enlarged boxes. The box_loss, seg_loss went 20x higher than before at the first dozen of epochs but it quickly goes down and converge aound 100epochs. )