question about mAP for DOTA dataset

[teeinn]

I re-implemented your work for training Dota dataset but I got much lower mAP for dota test dataset.

*** training & validation**---------------------------------------------------------------------------------------------------------------

• dataset : DOTA - v1.0 (training, validation dataset -- all cropped by using your img_split.py)
• configuration : faster_rcnn_orpn_r101_fpn_1x_ms_rr_dota10.py
• parameters : not changed at all (same with your implementation)

*** test**-------------------------------------------------------------------------------------------------------------------------------------

• dataset : DOTA - v1.0 (test dataset -- all cropped by using your img_split.py)
• configuration : faster_rcnn_orpn_r101_fpn_1x_ms_rr_dota10.py
• used epoch 12 checkpoint
• sent to DOTA v 1.0 server - Evaluation of DOTA Task1
  and got this result ---> mAP: 0.41229681978600863
  ap of each class: plane:0.5396662332791309, baseball-diamond:0.2710892710892711,
  bridge:0.5154165616433038, ground-track-field:0.1484848484848485, small-vehicle:0.27250426663927574,
  large-vehicle:0.2622058637252267, ship:0.7108541529636384, tennis-court:0.5453417550191744,
  basketball-court:0.33560606060606063, storage-tank:0.09090909090909091, soccer-ball-
  field:0.22898154477101842, roundabout:0.19935158396696856, harbor:0.7828865404399028, swimming-
  pool:0.7302785521014554, helicopter:0.5508759711517627

According to your report, it should be 80.52 (backbone resnet101).
What have I done wrong? What should I change?
Waiting for your quick response! Thank you!

[jbwang1997]

Could you provide the number of trainval images and test images.
The test set of the DOTA dataset consists of two zip files. I consider if you forget to unzip one of zip files.

[teeinn]

I think you are right.
I thought I unzipped both of zip files (part1, part2).
However, after cropping images, test images were 30087, which is smaller number than number of cropped validation images.

Before crop images ----> After img_split.py ----> After I fix
train : 1411 ----> 52057
val : 458 ----> 35462
test : 937 ----> 30087 ----> 71888

now, it is 71888. Is this correct number of data images?

[jbwang1997]

Yes, the 71888 is the correct number of test set.

[teeinn]

Thank you! I will try one more time with this test set.

[colorfully-ybf]

Yes, the 71888 is the correct number of test set.

why is 71888? I have the same train val and test.After siplit,my picture is 10833 test and 12800 trainval.The parameter of gap is 200 and the pixel is 1024*1024.

[jbwang1997]

71888 images are generated by multiple-scale mode. 10833 images are generated by single-scale mode.

[colorfully-ybf]

Tanks

[fhaji75]

@teeinn
When I'm running test.py on DOTA dataset I got mAP=0 with this comment :
Result is Only for reference, final result is subject to DOTA_devkit

+--------------------+-----+--------+--------+--------+
| class | gts | dets | recall | ap |
+--------------------+-----+--------+--------+--------+
| large-vehicle | 0 | 102855 | 0.0000 | 0.0000 |
| swimming-pool | 0 | 12696 | 0.0000 | 0.0000 |
| helicopter | 0 | 11973 | 0.0000 | 0.0000 |
| bridge | 0 | 33974 | 0.0000 | 0.0000 |
| plane | 0 | 19639 | 0.0000 | 0.0000 |
| ship | 0 | 66079 | 0.0000 | 0.0000 |
| soccer-ball-field | 0 | 6102 | 0.0000 | 0.0000 |
| basketball-court | 0 | 4562 | 0.0000 | 0.0000 |
| ground-track-field | 0 | 7147 | 0.0000 | 0.0000 |
| small-vehicle | 0 | 126984 | 0.0000 | 0.0000 |
| baseball-diamond | 0 | 7429 | 0.0000 | 0.0000 |
| tennis-court | 0 | 9675 | 0.0000 | 0.0000 |
| roundabout | 0 | 10242 | 0.0000 | 0.0000 |
| storage-tank | 0 | 41839 | 0.0000 | 0.0000 |
| harbor | 0 | 25446 | 0.0000 | 0.0000 |
+--------------------+-----+--------+--------+--------+
| mAP | | | | 0.0000 |
+--------------------+-----+--------+--------+--------+
I use the ss_test.json file for split test data, Is that correct?
Do you know why this happened?