mAP calculated by pycocotools collapsed #2

XYQDUT · 2024-09-02T08:36:22Z

I saw in the paper that this method improves small object detection. However, when I trained with the VisDrone dataset, I found that the mAP calculated by pycocotools collapsed. Here is the output from the last epoch of training.

Evaluating speed.

Evaluating mAP by pycocotools.
Saving MAF-YOLO/exp4/predictions.json...
Results saved to MAF-YOLO/exp4
Epoch: 99 | mAP@0.5: 0.03880035055291523 | mAP@0.50:0.95: 0.017914316025343748
best mAP@0.50:0.95: 0.04084159832035066 | best epoch: 0

Training completed in 4.064 hours.
loading annotations into memory...
Done (t=0.13s)
creating index...
index created!
Loading and preparing results...
DONE (t=1.07s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type bbox
DONE (t=23.53s).
Accumulating evaluation results...
DONE (t=1.74s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.018
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.039
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.015
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.019
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.019
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.033
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.012
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.060
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.098
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.098
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.117
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.150

However, using the weight file last_ckpt.pt saved from the final training session results in a better mAP. Here is the result.

Class Images Labels P@.5iou R@.5iou F1@.5iou mAP@.5 mAP@.5:.95
all 548 38759 0.591 0.43 0.492 0.436 0.261

Evaluating speed.
Average pre-process time: 0.24 ms
Average inference time: 6.46 ms
Average NMS time: 3.61 ms

Evaluating mAP by pycocotools.
Saving runs/val/result/predictions.json...
loading annotations into memory...
Done (t=0.29s)
creating index...
index created!
Loading and preparing results...
DONE (t=1.13s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type bbox
DONE (t=19.56s).
Accumulating evaluation results...
DONE (t=1.64s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.018
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.039
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.015
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.019
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.019
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.033
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.012
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.059
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.098
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.098
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.116
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.150

The text was updated successfully, but these errors were encountered:

yang-0201 · 2024-09-02T16:06:15Z

Thank you very much for your interest in this project，Is the training log in the middle working properly？ How accurate is it? I'd like you to provide me with the model you used and the commands you used to train it.

XYQDUT · 2024-09-05T02:34:55Z

Thank you very much for your interest in this project，Is the training log in the middle working properly？ How accurate is it? I'd like you to provide me with the model you used and the commands you used to train it.

Except for the results calculated by pycocotools, everything else seems normal.
The training command
python tools/train.py --data-path data/Visdrone.yaml --conf-file configs/pretrain/MAF-YOLO-m-pretrain.py

yang-0201 · 2024-09-11T08:29:50Z

Hello, I re-uploaded a new version of 1.1.0 and did not have a similar problem, you can re-download the code and try again!

yang-0201 · 2024-09-11T08:46:06Z

train cmd:
python tools/train.py --conf configs/pretrain/MAF-YOLO-m-pretrain.py --data data/VisDrone.yaml --device 0 --batch 8 --epochs 100
get:

transform_weight: 1562/1568 from MAFYOLOm.pt
transform_model_weight: 1562/1568 from MAFYOLOm.pt

Training start...

 Epoch  iou_loss  dfl_loss  cls_loss
  0/99    0.6247     0.375     1.288: 100%|██████████| 809/809 [04:17<00:00,  3.14it/s]

Inferencing model in train datasets.: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 35/35 [00:17<00:00, 2.02it/s]

Evaluating speed.

Evaluating mAP by pycocotools.
Saving MAF-YOLO\exp1\predictions.json...
loading annotations into memory...
Done (t=0.10s)
creating index...
index created!
Loading and preparing results...
DONE (t=0.83s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type bbox
DONE (t=19.35s).
Accumulating evaluation results...
DONE (t=1.16s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.126
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.209
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.127
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.059
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.211
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.357
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.068
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.215
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.296
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.177
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.474
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.656
Results saved to MAF-YOLO\exp1
Epoch: 0 | mAP@0.5: 0.20894816711035757 | mAP@0.50:0.95: 0.1255171991137001
best mAP@0.50:0.95: 0.1255171991137001 | best epoch: 0

 Epoch  iou_loss  dfl_loss  cls_loss
  1/99    0.5766    0.3498    0.9683: 100%|██████████| 809/809 [04:00<00:00,  3.37it/s]

Inferencing model in train datasets.: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 35/35 [00:15<00:00, 2.23it/s]

Evaluating speed.

Evaluating mAP by pycocotools.
Saving MAF-YOLO\exp1\predictions.json...
loading annotations into memory...
Done (t=0.09s)
creating index...
index created!
Loading and preparing results...
DONE (t=0.78s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type bbox
DONE (t=19.79s).
Accumulating evaluation results...
DONE (t=1.08s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.159
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.264
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.161
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.073
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.267
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.461
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.082
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.234
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.308
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.186
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.494
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.713
Results saved to MAF-YOLO\exp1
Epoch: 1 | mAP@0.5: 0.2639859069451592 | mAP@0.50:0.95: 0.1586531816945784
best mAP@0.50:0.95: 0.1586531816945784 | best epoch: 1

 Epoch  iou_loss  dfl_loss  cls_loss
  2/99    0.5607    0.3416    0.9162: 100%|██████████| 809/809 [03:55<00:00,  3.43it/s]

Inferencing model in train datasets.: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 35/35 [00:15<00:00, 2.19it/s]

Evaluating speed.

Evaluating mAP by pycocotools.
Saving MAF-YOLO\exp1\predictions.json...
loading annotations into memory...
Done (t=0.10s)
creating index...
index created!
Loading and preparing results...
DONE (t=0.88s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type bbox
DONE (t=19.76s).
Accumulating evaluation results...
DONE (t=1.13s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.182
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.303
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.186
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.088
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.299
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.468
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.089
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.259
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.340
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.216
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.526
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.736
Results saved to MAF-YOLO\exp1
Epoch: 2 | mAP@0.5: 0.3027632490114197 | mAP@0.50:0.95: 0.18194999742836987
best mAP@0.50:0.95: 0.18194999742836987 | best epoch: 2

XYQDUT · 2024-09-11T10:00:01Z

Thank you very much. I will download and re-run the program.

…

________________________________ 发件人: zhiqiang yang ***@***.***> 发送时间: 2024年9月11日 16:30 收件人: yang-0201/MAF-YOLO ***@***.***> 抄送: xyq ***@***.***>; Author ***@***.***> 主题: Re: [yang-0201/MAF-YOLO] mAP calculated by pycocotools collapsed (Issue #2) Hello, I re-uploaded a new version of 1.1.0 and did not have a similar problem, you can re-download the code and try again! ― Reply to this email directly, view it on GitHub<#2 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BCYE6RRO46EFWSR7343MO63ZV75RJAVCNFSM6AAAAABNP24NL6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNBSHE4TINRUGA>. You are receiving this because you authored the thread.Message ID: ***@***.***>

yang-0201 closed this as completed Oct 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

mAP calculated by pycocotools collapsed #2

mAP calculated by pycocotools collapsed #2

XYQDUT commented Sep 2, 2024

yang-0201 commented Sep 2, 2024

XYQDUT commented Sep 5, 2024

yang-0201 commented Sep 11, 2024

yang-0201 commented Sep 11, 2024

XYQDUT commented Sep 11, 2024 via email

mAP calculated by pycocotools collapsed #2

mAP calculated by pycocotools collapsed #2

Comments

XYQDUT commented Sep 2, 2024

yang-0201 commented Sep 2, 2024

XYQDUT commented Sep 5, 2024

yang-0201 commented Sep 11, 2024

yang-0201 commented Sep 11, 2024

XYQDUT commented Sep 11, 2024 via email