Unable to reproduce the results on Pascal VOC datasets

[hugotong6425]

Hi, I tried to reproduce the G-FSOD results on Pascal VOC using the default config, but I am not able to reproduce the 2/3/5/10 shot results. Only the 1-shot result can be reproduced. The only change I made is the GPU number, which is changed from 8 to 2. Batch size and other config are kept unchanged.

G-FSOD, split 1, average results of seed 0-9

shot	reported AP (github repo)	reproduced AP
1-shot	43.8	41.95
2-shot	57.5	44.24
3-shot	61.4	45.09
5-shot	65.3	46.70
10-shot	67.0	47.51

training_script.txt
results.txt

Is there anything wrong with my experiment? Also, could you please provide a 10-shot trained model? Thank you very much.

[er-muyue]

If you change the number of GPUs, I think both the batch-size and the number of iterations should be changed accordingly to ensure that the model is fully trained (to ensure fair comparison with this repo), see issue #19 for more details.

[zhtmike]

@er-muyue What if the batch size is same, but the total number of GPU is changed. DO we need to modify the setting to reproduce the result?

[hugotong6425]

@er-muyue If I understand correctly, the stated batch size (which is 16) means total batch size instead of batch size per GPU. In the above experiments, I kept the total batch size unchanged (16) even though I changed the GPU number from 8 to 2.

In your last comment, do you mean to suggest changing the total batch size and number of iterations if I changed the GPU number? If yes, could you please suggest a direction to change (e.g. in 10-shot) so that the result gap between reported and reproduced can be reduced? Thanks.

[hugotong6425]

Just realized the result reported of Pascal voc is AP50 and result reported of coco is mAP. The result is able to be reproduced. Thanks.

[fenghuo-cao]

Hello, I would like to ask if your versions of python, cuda and pytorch are the same as the author's, because I use a 3090 graphics card, and the versions of cuda and pytorch cannot be consistent with the author's. Only the number of GPUs has been modified in the code, and the others have not been modified. The reproduced experimental results are far from the author's.Thank you very much.