Oriented Object Detection on Imbalanced Data

[xanderex-sid]

Hello Everyone,

I have been training on an imbalanced dataset. Here are the statistics for my cell tower dataset (before the 80% train / 20% validation split):

total images: 524

annotations:-
RRU :  5355 (cuboid shape)
Panel Antenna :  3165 (long rectangular antennas)
Microwave Antenna :  862 (circular dish)
5G Antenna :  643 (cuboid shape)

My raw images are of width: 5280 and height: 3956

One thing is that, RRU and 5G Antenna look roughly same shape and design.

Now I tried to train two models yolo11l-obb.pt and yolo11x-obb.pt, and in both models I got roughly same precision and recall, that is:-

In yolo11x-obb.pt :-
precision: 0.893
recall: 0.690

In yolo11l-obb.pt :-
precision: 0.887
recall: 0.688

My training configurations were:-

img_size: 1280
batch_size: 16
patience: 200

other values were set to default. 

later I tried to change v8OBBLoss class by commenting BCE loss and adding VFL loss like this:

target_labels = (target_scores > 0).float()
loss[1] = self.varifocal_loss(pred_scores, target_scores, target_labels) / target_scores_sum  # VFL way

But that didn’t help either; it actually worsened both precision and recall.

Can you help me determine the next steps—specifically which model to use, the appropriate hyperparameters, and the strategies to implement? My primary goal is to maximize recall while maintaining reasonable precision.

[UltralyticsAssistant]

👋 Hello @xanderex-sid, thank you for your interest in Ultralytics 🚀! This is an automated response to help you get unblocked quickly, and an Ultralytics engineer will also assist soon 🙂 We recommend a visit to the Docs for new users where you can find many Python and CLI usage examples and where many of the most common questions may already be answered.

If this is a 🐛 Bug Report (e.g., unexpected behavior from the OBB loss change or a regression between models), please provide a minimum reproducible example to help us debug it.

If this is a custom training ❓ Question (imbalanced OBB data + recall focus), please provide as much information as possible so we can give targeted guidance, including dataset image examples and training logs, and verify you are following our Tips for Best Training Results. In particular, please share:
-Your exact training command (Python/CLI) for yolo11l-obb.pt and yolo11x-obb.pt
-A linkable snippet of results.csv (or the key epoch metrics) and your val plots
-Your data.yaml class order + counts after the train/val split
-A few representative labeled examples for each class (especially where RRU vs 5G Antenna are ambiguous)
-Any non-default settings you used beyond imgsz, batch, and patience (including augmentations)
-Confirmation of whether you changed any code in the loss (and which commit/version you’re on)

Join the Ultralytics community where it suits you best. For real-time chat, head to Discord 🎧. Prefer in-depth discussions? Check out Discourse. Or dive into threads on our Subreddit to share knowledge with the community.

Upgrade

Upgrade to the latest ultralytics package including all requirements in a Python>=3.8 environment with PyTorch>=1.8 to verify your issue is not already resolved in the latest version:

pip install -U ultralytics

Environments

YOLO may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all Ultralytics CI tests are currently passing. CI tests verify correct operation of all YOLO Modes and Tasks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

[Y-T-G]

You can try using YOLO26-OBB

[glenn-jocher]

It looks like your loss-weight overrides may not be taking effect: the CLI/Python args are box, cls, dfl (and for OBB also angle), not box_wt/cls_wt/dfl_wt—see the official names in the Train mode arguments; also when fine-tuning from pretrained weights, try warmup_epochs=0 (long warmup can slow adaptation on small custom datasets), and avoid editing ultralytics/utils/loss.py unless you’re ready to maintain a fork. With 5280×3956 inputs, recall is often capped by downsampling, so you’ll usually gain more by tiling/cropping into 1024–1600 patches (with labels adjusted) than by going even bigger on the backbone; once trained, maximize recall by sweeping a lower conf at inference/validation and choosing an operating point from PR/F1 outputs as described in the performance metrics guide (and keep an eye on the RRU vs 5G confusion—if they’re visually indistinguishable, more “hard” examples or merging classes may be the only real fix). For reference, OBB specifics/constraints are in the OBB task docs.

yolo obb train model=yolo26x-obb.pt data=path/to/data.yaml imgsz=1600 batch=6 box=7.5 cls=0.8 dfl=1.2 angle=1.0 cos_lr=True warmup_epochs=0

If you paste your exact yolo obb train ... command plus per-class metrics (and yolo checks output as text), we can suggest the next most likely knob to turn for recall on your specific setup.

[xanderex-sid]

In the tiling process, since the annotations are oriented bounding boxes (OBBs). When an OBB lies partially at the boundary of a tiled image—such as at the right edge—using a Shapely polygon–based intersection can result in a polygon with five or more vertices instead of the original four. On the other hand, directly clipping the bounding box to the tile boundaries leads to a loss of geometric information and does not preserve the original orientation accurately.

This tiling issue can cause inconsistencies during inference, especially when merging detected oriented bounding boxes (OBBs) from adjacent tiles.

Any thoughts on this or I am missing something here ?

and... I am actually using code like this:

        if config.pretrained_model:
            model = YOLO(config.pretrained_model)
        elif config.model_yaml:
            model = YOLO(config.model_yaml)
        else:
            model = YOLO("yolo26x-obb.pt")
        model.info()
        
        results = model.train(
            data=yaml_path,
            epochs=config.epochs,
            imgsz=config.img_size, 
            batch=config.batch_size,
            cos_lr=config.cos_lr,
            box=config.box_wt,
            cls=config.cls_wt,
            dfl=config.dfl_wt,
            degrees=config.degrees,
            fliplr=config.fliplr,
            warmup_epochs=config.warmup_epochs,
            patience=config.patience,
            device=config.device,
            workers=config.workers,
            project=config.output_dir,
            name=config.name
        )

where, in config, I am using these values:

        where:
        box_wt = 7.5
        cls_wt = 0.8
        dfl_wt = 1.2
        cos_lr = True
        warmup_epochs = 5.0
        degrees = 20.0
        fliplr = 0.7
        
        other values are default.

[glenn-jocher]

Yes—if you intersect an OBB polygon with a tile boundary you’ll often get an N-vertex polygon, and YOLO OBB labels must stay as 4 points (or internally xywhr), so you either (a) drop “cut” objects during tiling (recommended) or (b) approximate the clipped shape back to a 4-corner rotated rectangle (e.g., min-area-rect) before writing labels, per the OBB dataset format.

For training, the simplest/most stable approach is to use overlapping tiles (e.g., 20–30% overlap) and only assign an object to a tile when all 4 vertices are inside (or at least when its center is inside + intersection area ratio passes a threshold)—that way every object appears fully in at least one neighboring tile and you avoid generating “invented” partial OBBs that can hurt learning and later merging.

If you do need to keep partials anyway, you can fit a 4-point OBB to the intersection polygon (note you’ll still lose some geometry by design, and you’ll want to normalize the angle to match the constraints in the OBB task docs):

import numpy as np
import cv2

def polygon_to_4pt_obb(poly_xy):  # poly_xy: (N,2)
    pts = np.asarray(poly_xy, dtype=np.float32).reshape(-1, 1, 2)
    (cx, cy), (w, h), angle_deg = cv2.minAreaRect(pts)
    box4 = cv2.boxPoints(((cx, cy), (w, h), angle_deg))  # (4,2)
    return box4

On your training call: your mapping is correct (box=config.box_wt, cls=config.cls_wt, dfl=config.dfl_wt), so the weights should apply; you can confirm by checking the saved run args.yaml in your output directory, and I’d still try warmup_epochs=0 for faster adaptation on a small custom dataset.

[xanderex-sid]

Thank you for the suggestion of using overlapping tiles—it did not occur to me earlier, and it makes sense.

I will run an experiment with overlapping tiles and also set the warmup_epochs=0.

I have another idea and would like your opinion. Instead of fixed tiling, I could group 2–3 nearby components and use their OBBs as references to crop a region from the original image that fully covers them. These cropped regions would then be used as training samples.

I see two main constraints with this approach:
(a) deciding the appropriate size range for the cropped images, and
(b) defining the maximum distance for components to be considered “neighbours.”

What do you think about this technique?

Also, what is your opinion on using overlapping tiles versus this neighbourhood-based cropping approach during inference?
Should I simply use standard tiling on the test data and run inference the same way for both approaches?

[glenn-jocher]

Your “neighbourhood-based” crops can work well for training (it’s essentially object-centric sampling), but it can also bias the model if most training views are “always near an object” and you lose true-background negatives; I’d treat it as an augmentation on top of uniform overlapping tiles, not a replacement.

A practical way to set crop size/neighbor rules is to build each crop from the union of the selected OBBs, then expand by a fixed padding + random jitter (so objects aren’t always centered), and clamp the final crop to a small set of sizes you can afford (e.g. 1024–1600) while avoiding heavy upscaling; for “neighbors,” use distance between OBB centers normalized by object size (e.g. within a few multiples of max(w, h)), and pick the threshold by looking at your dataset’s nearest-neighbor distance distribution rather than guessing. If you need to convert between 4-point corners and the internal xywhr conventions/angle normalization, see the xyxyxyxy2xywhr utility reference in the docs: OBB corner-to-xywhr conversion.

For inference, overlapping tiling is still the straightforward/robust option; neighbourhood-based cropping at inference only makes sense if you already have a proposal mechanism (e.g., a first pass on a downscaled image to propose regions, then a second high-res pass), otherwise you don’t know where to crop.