raise ValueError('No gt or bboxes') ValueError: No gt or bboxes

[lucasjinreal]

If per_img_gpu set to 1, which using 1 batchsize, this error should be handled:

    raise ValueError('No gt or bboxes')
ValueError: No gt or bboxes

it's lead by this single image has no ground truth boxes which can always occured when training on a big dataset. As far as I know, coco datasets has a lot of images that has no target inside image.

[joeythegod]

This also happens when my dataset has at least one bbox per image, a little bit weird.
I also discovered that it happens more frequently when I used bigger batch training. Hope someone can help.
E.g.:
Batch=1, GPU=8xV100 -> works
Batch=4, GPU=2xV100 0> no bboxes

[hellock]

Images with no gt bboxes are filtered out in the dataset automatically, see here. Could you help locate the error and see if there is no RPN proposals?

[artmortal93]

No RPN proposal could be sometimes generated when the learning rate are not following the linear scaling rule, finetune the lr could help.

[cowoopark]

The error sometimes occurs even after a few epochs.
I usually restart the model from that point, and sometimes it worked.
i agree that it's a little bit weird.

[FanghaiZhao]

I have also encountered the same problem of 'No gt or bbox' when training with single gpu using the code python train.py configs/faster_rcnn_r101_fpn_1x.py --validate. However when I tried to use the dist_train code and set the gpu num to 1, it works well, i.e., use the command ./dist_train.sh configs/faster_rcnn_r101_fpn_1x.py 1 --validate. don't know why.

[Maxfashko]

@hellock can you help with this problem? This occurs every time you start with the cascade_rcnn_x101_64x4d_fpn_1x.py config.

tools/dist_train.sh configs/cascade_rcnn_x101_64x4d_fpn_1x.py 4

i used:

imgs_per_gpu=16,
workers_per_gpu=8,
lr=0.08

[PAUPAUPAUL]

Hi, This occurs every time, do you have any solution?

[PAUPAUPAUL]

@hellock @joeythegod @cowoopark

[hellock]

There are mainly two reasons:

1. The dataset contains images without gt bboxes. You need to filter those images like in CocoDataset. We are preparing a PR to allow training with pure background images, but it may cover up bugs when there are other issues and make debugging more difficult for non-professional users.
2. The model is trained badly so that RPN cannot generate any valid proposals. This usually occurs when the hyper-parameters like lr are not properly set.

[hellock]

#1531