Custom training issue assert (boxes1[:, 2:] >= boxes1[:, :2]).all()

[pauamargant]

When training a custom dataset with the model, I consistently encounter the following error around epoch 9. The setup uses Python 3.11 and a single RTX 3090 GPU.

From my investigation, this issue appears in DETR-based models and related detectors, but I have not been able to find a definitive solution. The number of classes is correctly set to num_classes + 1, accounting for the background class.

---

🔥 Error Trace

tensor([[[nan, nan, nan, nan],
         ...
         [nan, nan, nan, nan]],

Epoch 9:  45%|████▍     | 460/1029 [07:40<10:23,  1.09s/it, loss=34.6598][rank0]: Traceback (most recent call last):
  File ".../train_deim_dev.py", line 64, in <module>
    trainer.fit(
  File ".../site-packages/deimkit/trainer.py", line 428, in fit
    train_stats = train_one_epoch(
  File ".../site-packages/deimkit/engine/solver/det_engine.py", line 72, in train_one_epoch
    loss_dict = criterion(outputs, targets, **metas)
  File ".../site-packages/deimkit/engine/deim/deim_criterion.py", line 276, in forward
    indices = self.matcher(outputs_without_aux, targets)['indices']
  File ".../site-packages/deimkit/engine/deim/matcher.py", line 101, in forward
    cost_giou = -generalized_box_iou(box_cxcywh_to_xyxy(out_bbox), box_cxcywh_to_xyxy(tgt_bbox))
  File ".../site-packages/deimkit/engine/deim/box_ops.py", line 56, in generalized_box_iou
    assert (boxes1[:, 2:] >= boxes1[:, :2]).all()
AssertionError

---

🧠 Notes

• All bounding boxes in my dataset are in [cx, cy, w, h] format and should be valid (value >0).
• I have double-checked the class count and annotations — nothing obviously wrong there.

Any guidance or suggestions for debugging this would be appreciated!

[dnth]

As far as I'm aware only the COCO format bounding box is supported.

Could you please try converting your bounding box to [x, y, w, h] format instead?

[pauamargant]

Sorry, my mistake when writing the issue. I have yolo labels, but I do convert them to coco format when training the DEIM model.
Until the model fails I do get valid results and the COCO metrics seems to indicate that the model is training properly. This DETR issue seems to be related.

[gebawe]

For anyone else facing a similar error:
I ran into the same issue, and in my case, it turned out that my class labels started from 1 instead of 0. Please make sure your label indices start from 0, and also verify that the number of classes in your labels matches the num_classes and associated parameter in your configuration. Mismatched class indices or counts can lead to unexpected errors during training.

[isu873]

First and foremost, thank you for your work and effort you put in this solution.

I'm experience the same issue, when i want to train with custom dataset.
Train is on a single GPU, RTX 4060 notebook, with 8 Gb VRAM, on Windows 11.

• Python 3.10.6
• torch 2.6.0+cu126
• torchaudio 2.6.0+cu126
• torchvision 0.21.0+cu126

My dataset contain 1617 train images, 203 test and 203 valid images, with 42 classes.

What I checked before training:

• dataset is in coco format (x,y,wh) converted from YOLO format
• ensure that there are no negative x and y values
• ensure that my _annotations.coco files contain 43 classes (bg + 42 class, index starts from 0)

In my train script, i set train_batch_size and val_batch_size to 1
My data contains 1280*1024 sized images, i have tried the training with multiple image_size, it produced the same problem.

After that, I made the suggested modification based on DETR issue last comment, suggested by pauamargant

After this, my training goes up to epoch 10-15, then a CUDA: Out of memory appears.

I didn't tried to train on CPU only, is it possible to train on CPU, then export the onnx to run the inference on GPU?

I dont know what am i doing wrong, any help or advice will be appriciated.

Thanks in advance!