YOLOV8 OBB

[vanthang35]

Search before asking

• I have searched the YOLOv8 issues and discussions and found no similar questions.

Question

Hi guys I trained YOLOv8 obb with my custom dataset and almost my images is rotated object, but i got the results with no rotated bounding box, what should i do? I see some guys tell that i should replay nonmaximum by mse loss can fix it but i dont know how to fix it when i training it on colabs

Additional

No response

[github-actions]

👋 Hello @vanthang35, thank you for your interest in Ultralytics YOLOv8 🚀! We recommend a visit to the Docs for new users where you can find many Python and CLI usage examples and where many of the most common questions may already be answered.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset image examples and training logs, and verify you are following our Tips for Best Training Results.

Join the vibrant Ultralytics Discord 🎧 community for real-time conversations and collaborations. This platform offers a perfect space to inquire, showcase your work, and connect with fellow Ultralytics users.

Install

Pip install the ultralytics package including all requirements in a Python>=3.8 environment with PyTorch>=1.8.

pip install ultralytics

Environments

YOLOv8 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all Ultralytics CI tests are currently passing. CI tests verify correct operation of all YOLOv8 Modes and Tasks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

[glenn-jocher]

@vanthang35 hello! 🚀 Great to hear you're experimenting with YOLOv8 OBB on your custom dataset. If your objective is to achieve rotated bounding boxes (RBB) and you're noticing the output as axis-aligned bounding boxes (AABB), switching from Non-Maximum Suppression (NMS) to a mean squared error (MSE) loss might not directly address the issue with bounding box orientation.

Rotated bounding boxes require specific annotations and model support. Ensure your dataset annotations are in the correct format (e.g., [x_center, y_center, width, height, angle] for rotated objects) and that your model is indeed configured to predict rotated bounding boxes. If the model training process on platforms like Colab doesn't initially support rotated bounding boxes directly, it might need some custom modification in the code which could be complex to implement without altering the YOLOv8 source code directly.

Consider checking the documentation or tutorials specific to handling rotated objects or implementing OBB in YOLO models. Sometimes, the community might have shared experiences or code snippets on similar tasks which could offer a workaround.

If your current setup isn't yielding the desired results, could you provide a bit more info about the dataset annotation format and the exact version or variant of YOLOv8 you're using? This info might help in offering a more tailored suggestion. Keep experimenting and best of luck with your project! 🛠️

[Amberonetwo]

@glenn-jocher Hello, I seem to have also encountered a similar problem, I am training YOLOv8-obb with custom data, and all my detection targets are rotating objects, but the results of some objects detection are indeed non-rotating, that is, the output is axis aligned Bounding box (AABB), which generally occurs when the detection object is a square. While rectangles normally output a rotated bounding box (RBB), my data set format is [class_index, x1, y1, x2, y2, x3, y3, x4, y4].

[glenn-jocher]

Hey there! 👋 Great to see you diving into YOLOv8-obb for detecting rotating objects. It's indeed intriguing that AABBs are yielded for square objects while RBBs are output for rectangles. This behavior is often tied to how the model interprets the aspect ratio and orientation of the objects based on training data.

Your dataset format [class_index, x1, y1, x2, y2, x3, y3, x4, y4] looks correct for OBBs. However, the model may default to AABBs for objects close to a square shape due to the minimal rotational variance or if it predicts minimal advantage in rotating the bounding box.

One thing you could try is augmenting your dataset to include more variety in the orientation of square objects, which might help the model learn to apply rotation more consistently. Another point of investigation could be the loss function and whether it sufficiently penalizes misalignment in orientation for closer-to-square objects.

It might also be helpful to review the angle prediction part of the network to ensure it's learning effectively. Sometimes, adjusting the learning rate for this specific prediction can also yield better results.

Without delving too deep into the specifics, these suggestions are general pathways you could explore. Keep pushing forward, and feel free to share any updates or further questions!

[zyiiiiiii]

@glenn-jocher Hello, I seem to have also encountered a similar problem, I am training YOLOv8-obb with custom data, and all my detection targets are rotating objects, but the results of some objects detection are indeed non-rotating, that is, the output is axis aligned Bounding box (AABB), which generally occurs when the detection object is a square. While rectangles normally output a rotated bounding box (RBB), my data set format is [class_index, x1, y1, x2, y2, x3, y3, x4, y4].

@Amberonetwo Hi, I encountered the similar problem with you, have you resolve this problem?

[glenn-jocher]

@zyiiiiiii hey there! 😊 It looks like you're encountering issues where your YOLOv8 OBB is outputting axis-aligned bounding boxes for objects that are nearly square. This is actually expected behavior to some extent, because the network might determine that the rotational adjustment for near-square objects doesn't significantly improve the prediction accuracy for these particular cases.

A way to potentially enhance the model's sensitivity to rotations even in square-like objects is to include more varied rotation angles during the training phase. This might help the model learn to apply rotations more consistently across different shapes.

If the issue persists and is critical for your application, reviewing the loss function related to the rotation prediction component or potentially customizing it might also be a path worth exploring.

Keep pushing with the tweaks, and hopefully, you'll see the improvements in no time! 🚀

[zyiiiiiii]

@glenn-jocher
I have increased the number and the variety of the square rotational objects as much as possible, but the model still can not detect the square rotational objects well. So I want to try to focus on loss function to fix this problem, how should I do for it? Could you give me some guidance how to modify the loss function related to the rotation prediction component or how to potentially customizing it？

Besides loss function, is there any other ways to solve this problem? Many thanks for your help！

[glenn-jocher]

Hi @zyiiiiiii! Great job on varying the training dataset. To adjust the loss function specifically for improving rotation detection, you might want to experiment with increasing the weight of the rotation component within your total loss. This can help the model prioritize learning rotations more effectively.

Here’s a simple example of adjusting loss weights in your model configuration:

loss:
  bbox: 0.5  # weight for bounding box loss
  cls: 1.0   # weight for classification loss
  rot: 2.0   # increasing weight for rotation loss

As an alternative, consider augmenting your dataset further with synthetic data where you artificially rotate square objects, this might help enhance the model's ability to learn from varied orientations.

Let me know how it goes! 😊

[zyiiiiiii]

@glenn-jocher
Thanks for your reply, but in the loss function, I observed that it contained box_loss, cls_loss and dfl_loss, no rot.

[Wangfeng2394]

@glenn-jocher Thanks for your reply, but in the loss function, I observed that it contained box_loss, cls_loss and dfl_loss, no rot.

Although it's impolite to say this, I think the person answering the question（glenn-jocher） is a robot because he always lists some parameters that are not in the program。#11666

[glenn-jocher]

Hi there! 😊 It sounds like there might have been some confusion regarding the loss components. The rot component I mentioned was an example and might not directly reflect the parameters in your specific YOLOv8 configuration. In your setup, if the rotation adjustment is part of bounding box calculations, it could be integrated into the box_loss.

For customizing loss functions related to rotations, you’d generally look into adjusting parts of the box_loss where angles are involved. Here’s a quick pseudo-example of what this might look like:

# This is a simplified pseudo-code
def modified_box_loss(predictions, targets):
    # Normal bbox losses (x, y, width, height)
    bbox_loss = mse_loss(predictions[:,:4], targets[:,:4])
    
    # Increased weight for angle (rotation) loss
    rotation_loss = 5 * mse_loss(predictions[:,4], targets[:,4])
    
    return bbox_loss + rotation_loss

Regarding your observation about responses, I assure you that the aim is always to provide the most relevant information, though sometimes misunderstandings can occur, especially in technical discussions. If there's anything specific you need to dive deeper into, feel free to ask!

[Wangfeng2394]

hi，Do you know what are you saying？ what‘s loss function in yolov8 obb？“probiou loss” right？The probiou loss including boxloss and angle loss, so where is individual rotation_loss？？？

[Wangfeng2394]

hi，Do you know what are you saying？ what‘s loss function in yolov8 obb？“probloss” right？the probloss including boxloss and angle loss, so where is individual rotation_loss？？？

Why Your answer is usually incorrect、meaningless and useless？

[Wangfeng2394]

I can give you feedback on two optimization directions for yolov8 obb：
1，Cancel the “anchor-free”,
2，Using angle classification instead of loss function。

[Wangfeng2394]

bbox_loss & rotation_los are fused together， which is “probiou”，，，
How do you split them up？？？？？？？？？？？？

[Amberonetwo]

beronetwo Hi, I encountered the similar problem with you, have you resolve this problem?

Hello, the solution is to modify the loss function, the loss function uses MSE instead of iou, but the drawback is that it can detect square-like objects, but the detection effect is greatly discounted. I don't know whether it is the reason that my data is not diverse enough, but the same amount of data to train is poor, you can try to modify the code, if you can, I want to know the effect of your modification? This is my V: 157550166715
mse_loss = nn.MSELoss()
#iou = probiou(pred_bboxes[fg_mask], target_bboxes[fg_mask])
#loss_iou = ((1.0 - iou) * weight).sum() / target_scores_sum
loss_iou = mse_loss(pred_bboxes[fg_mask], target_bboxes[fg_mask])

[Wangfeng2394]

beronetwo Hi, I encountered the similar problem with you, have you resolve this problem?

Hello, the solution is to modify the loss function, the loss function uses MSE instead of iou, but the drawback is that it can detect square-like objects, but the detection effect is greatly discounted. I don't know whether it is the reason that my data is not diverse enough, but the same amount of data to train is poor, you can try to modify the code, if you can, I want to know the effect of your modification? This is my V: 157550166715 mse_loss = nn.MSELoss() #iou = probiou(pred_bboxes[fg_mask], target_bboxes[fg_mask]) #loss_iou = ((1.0 - iou) * weight).sum() / target_scores_sum loss_iou = mse_loss(pred_bboxes[fg_mask], target_bboxes[fg_mask])

微信不存在，发个微信号给我。这个回答问题的哥们儿有点虎拉吧唧的，驴头不对马嘴

[Amberonetwo]

beronetwo Hi, I encountered the similar problem with you, have you resolve this problem?

Hello, the solution is to modify the loss function, the loss function uses MSE instead of iou, but the drawback is that it can detect square-like objects, but the detection effect is greatly discounted. I don't know whether it is the reason that my data is not diverse enough, but the same amount of data to train is poor, you can try to modify the code, if you can, I want to know the effect of your modification? This is my V: 157550166715 mse_loss = nn.MSELoss() #iou = probiou(pred_bboxes[fg_mask], target_bboxes[fg_mask]) #loss_iou = ((1.0 - iou) * weight).sum() / target_scores_sum loss_iou = mse_loss(pred_bboxes[fg_mask], target_bboxes[fg_mask])

微信不存在，发个微信号给我。这个回答问题的哥们儿有点虎拉吧唧的，驴头不对马嘴

15750166715

[glenn-jocher]

Hello! It looks like you're exploring alternative loss functions to address specific issues with oriented bounding box prediction in YOLOv8 OBB. Using MSE loss might indeed change detection characteristics significantly as you've noticed. Rotational aspects and angular discrepancies typically need a dedicated component in the loss to be handled effectively without heavily compromising other detection metrics.

If you want to experiment with loss modifications or other adjustments, please ensure they align with the architecture and data specifics you are working with. Unfortunately, sharing contact details directly here might not be safe, so I'd recommend discussing further technical insights directly through GitHub issues or discussions for community safety and sharing knowledge openly. 😊

Happy coding!

[github-actions]

👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

For additional resources and information, please see the links below:

• Docs: https://docs.ultralytics.com
• HUB: https://hub.ultralytics.com
• Community: https://community.ultralytics.com

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLO 🚀 and Vision AI ⭐