AssertionError: Invalid boxes

[liming-ai]

Hi @SuperHenry2333 @SlongLiu @FengLi-ust

Update: some bboxes after data augmentation are nan:

---

I tried to reproduce the Objects365 pre-training with a large resolution, however, this error always occurs. I tried to filter the invalid boxes in dataloader but it still happens, which means this error is caused by data augmentation. Have you ever meet this? Could you please provide some advice?

Traceback (most recent call last):
  File "main.py", line 401, in <module>
    main(args)
  File "main.py", line 288, in main
    args.clip_max_norm, wo_class_error=wo_class_error, lr_scheduler=lr_scheduler, args=args, logger=(logger if args.save_log else None), ema_m=ema_m)
  File "/home/tiger/code/DINO/engine.py", line 52, in train_one_epoch
    loss_dict = criterion(outputs, targets)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1117, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/tiger/code/DINO/models/dino/dino.py", line 499, in forward
    indices = self.matcher(outputs_without_aux, targets)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1117, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
    return func(*args, **kwargs)
  File "/home/tiger/code/DINO/models/dino/matcher.py", line 87, in forward
    cost_giou = -generalized_box_iou(box_cxcywh_to_xyxy(out_bbox), box_cxcywh_to_xyxy(tgt_bbox))
  File "/home/tiger/code/DINO/util/box_ops.py", line 52, in generalized_box_iou
    assert (boxes1[:, 2:] >= boxes1[:, :2]).all()
AssertionError

[SlongLiu]

Thanks for your question. This discussion may be helpful: facebookresearch/detr#101