trained custom dataset uses coco_eval.py but got ap = -1.0

[CrystalSixone]

Hi, I have some problems when I use coco_eval.py after training with my own dataset, I don't know why all the AP is equal to -1.0.
I am sure that the path of val set is correct. The training seems to be successful and I can use efficientdet_test.py with trained weights to detect. This is my training log.
0.09451: 98%|█████████▊| 292/299 [03:49<00:04, 1.40it/s]
Step: 59793. Epoch: 199/200. Iteration: 293/299. Cls loss: 0.02686. Reg loss: 0.00648. Total loss: 0.03334: 98%|█████████▊| 292/299 [03:50<00:04, 1.40it/s]
Step: 59793. Epoch: 199/200. Iteration: 293/299. Cls loss: 0.02686. Reg loss: 0.00648. Total loss: 0.03334: 98%|█████████▊| 293/299 [03:50<00:04, 1.43it/s]
Step: 59794. Epoch: 199/200. Iteration: 294/299. Cls loss: 0.06801. Reg loss: 0.01800. Total loss: 0.08601: 98%|█████████▊| 293/299 [03:51<00:04, 1.43it/s]
Step: 59794. Epoch: 199/200. Iteration: 294/299. Cls loss: 0.06801. Reg loss: 0.01800. Total loss: 0.08601: 98%|█████████▊| 294/299 [03:51<00:03, 1.43it/s]
Step: 59795. Epoch: 199/200. Iteration: 295/299. Cls loss: 0.03731. Reg loss: 0.01866. Total loss: 0.05597: 98%|█████████▊| 294/299 [03:51<00:03, 1.43it/s]
Step: 59795. Epoch: 199/200. Iteration: 295/299. Cls loss: 0.03731. Reg loss: 0.01866. Total loss: 0.05597: 99%|█████████▊| 295/299 [03:51<00:02, 1.45it/s]
Step: 59796. Epoch: 199/200. Iteration: 296/299. Cls loss: 0.03701. Reg loss: 0.01164. Total loss: 0.04865: 99%|█████████▊| 295/299 [03:52<00:02, 1.45it/s]
Step: 59796. Epoch: 199/200. Iteration: 296/299. Cls loss: 0.03701. Reg loss: 0.01164. Total loss: 0.04865: 99%|█████████▉| 296/299 [03:52<00:02, 1.47it/s]
Step: 59797. Epoch: 199/200. Iteration: 297/299. Cls loss: 0.03641. Reg loss: 0.01481. Total loss: 0.05122: 99%|█████████▉| 296/299 [03:52<00:02, 1.47it/s]
Step: 59797. Epoch: 199/200. Iteration: 297/299. Cls loss: 0.03641. Reg loss: 0.01481. Total loss: 0.05122: 99%|█████████▉| 297/299 [03:52<00:01, 1.50it/s]
Step: 59798. Epoch: 199/200. Iteration: 298/299. Cls loss: 0.05714. Reg loss: 0.02491. Total loss: 0.08206: 99%|█████████▉| 297/299 [03:53<00:01, 1.50it/s]
Step: 59798. Epoch: 199/200. Iteration: 298/299. Cls loss: 0.05714. Reg loss: 0.02491. Total loss: 0.08206: 100%|█████████▉| 298/299 [03:53<00:00, 1.47it/s]
Step: 59799. Epoch: 199/200. Iteration: 299/299. Cls loss: 0.02466. Reg loss: 0.01264. Total loss: 0.03729: 100%|█████████▉| 298/299 [03:54<00:00, 1.47it/s]
Step: 59799. Epoch: 199/200. Iteration: 299/299. Cls loss: 0.02466. Reg loss: 0.01264. Total loss: 0.03729: 100%|██████████| 299/299 [03:54<00:00, 1.51it/s]
Step: 59799. Epoch: 199/200. Iteration: 299/299. Cls loss: 0.02466. Reg loss: 0.01264. Total loss: 0.03729: 100%|██████████| 299/299 [03:54<00:00, 1.27it/s]

[CrystalSixone]

This is my gt val json

And This is val_bbox_results.json generated after using coco_eval.py. (partial)

It is strange that there are many repeated predicted areas in val_bbox_results.json, I don't know if it is the point.

[zylo117]

What 's your training and eval command?

[CrystalSixone]

This is my training command:
nohup python train.py -c 0 -p smoke --batch_size 16 --lr 1e-5 --num_epochs 200 --load_weights weights/efficientdet-d0.pth >smoke.out 2>&1 &

And this is my eval command: ( I can use weights_smoke.pth to test and the result is not bad).
python coco_eval.py -p smoke -w weights_smoke.pth

Is there any code should I change when I use coco_eval.py?
Thx a lot.

[zylo117]

try lr 1e-3

[CrystalSixone]

THX. I will try.

[SSJIACV]

THX. I will try.

After setting lr=1e-3, is mAP up？

[CrystalSixone]

Hi, I find where the problem is!
for single class detection, add the code in coco_eval.py, def _eval(),
coco_eval.params.catIds = 1 # according to the doc, the category id should be started from 1.
and we can see the evaluation result:

[CrystalSixone]

THX. I will try.

After setting lr=1e-3, is mAP up？

sorry, I have not tried the new lr yet, but I find why my coco_eval results are all equal to -1. I will do more experiments later.

[fightingaaa]

Hi, I find where the problem is!
for single class detection, add the code in coco_eval.py, def _eval(),
coco_eval.params.catIds = 1 # according to the doc, the category id should be started from 1.
and we can see the evaluation result:

I met the same question , also for single class detection,but after add coco_eval.params.catIds = 1. the results is the same .

[fightingaaa]

Below is my results

[fightingaaa]

Box results:

gt:

Please Help Me ! Thanks !!

[CrystalSixone]

Box results:

gt:

Please Help Me ! Thanks !!

I think your training is successful too. Maybe you can compare coco_eval official instructions, and check whether your own database format is correct. Good luck!

[zylo117]

@fightingaaa why is the gt id always -1?

[fightingaaa]

Box results:

gt:

Please Help Me ! Thanks !!

I think your training is successful too. Maybe you can compare coco_eval official instructions, and check whether your own database format is correct. Good luck!

Thanks！ I have find my problem. id is wrong

[fightingaaa]

@fightingaaa why is the gt id always -1?

Thanks! I find this problem ,Now the map is correct,map is higher ,but I find the detection score is lower than before.

[zylo117]

@fightingaaa keep training until it overfits