Can not reproduce results in your paper

[AIZOOTech]

Hi, I downloaded your codes, and train ATSS-R50 and RankLoss-ATSS-R50, the results are:

algorithm	exp1 mAP	exp2 mAP
ATSS	0.396	0.393
RS-ATSS	0.393	0.392

The mAP of RS-ATSS is lower than ATSS, however, in the paper, RS-ATSS is 0.399 , ATSS is 0.393。
I wonder why can not reproduce the results, is there something wrong?

[kemaloksuz]

Hi,

Thanks for your interest in our work. I will try to assist reproducing the results.

What is your batch size (how many GPUs and how many images on each GPU) during training? Also, just to confirm: have you made any changes in the code or config file?

[AIZOOTech]

@kemaloksuz I only change workers_per_gpu from 4 to 2, nothing changed else.

 55 data = dict(
 56     samples_per_gpu=4,
 57     workers_per_gpu=2)

[kemaloksuz]

What about number of GPUs?, i.e. How many GPUs do you use for training? You can also directly share your training command to see the number of GPUs.

[AIZOOTech]

What about number of GPUs?, i.e. How many GPUs do you use for training? You can also directly share your training command to see the number of GPUs.

8 V100 16G GPU were used to train.

[kemaloksuz]

We used 4 GPUs for training ATSS and the learning rate is tuned for 4 images/GPU, that is 16 images in batch. So, that's the mismatch with our configuration. Hence, I think decreasing the number of GPUs to 4 (and keeping 4 images/GPU) should solve your reproduction problem.

[kemaloksuz]

Just to update: Upon your worries, we tried to reproduce ATSS and obtained 39.8 AP, which aligns with the reported result 39.9 AP. If you use 4 GPUs for training and 4 images/GPU, you should get ~39.9AP.

[kemaloksuz]

Using the provided info above, have you been able to reproduce our results? If so, could you please close the issue?