Questions about experiments

[Hongbin98]

Thanks for your ICCV work. However, I find you directly use the test set of ImageNet-LT to store the best models, which may lead to overfitting in practice and seems to be unfair to other compared methods. Could you please provide the results on ImageNet-LT using the validation set to store models? It would be easier for us to compare with PaCo in our work. Thanks very much.

[jiequancui]

Hi,

Very thanks for your suggestion! In fact, we observe there is no difference for selecting models by validation set or test set in our experiments. Further, for all re-implemented baselines, we use the same setting for comparisons.

[xxxzhi]

hi, do you run experiment on 4 gpus with total batch size 128? what's the running time? I find I can not reproduce the result with batch size 128 (single gpu). Much worse than the reported.

[jiequancui]

Hi,
Thank you for your question.
We run all experiments on 4gpus with a batch size of 128 in 400 epochs, that's 32 images on a single GPU.
what network is used in your experiment? Can you provide me a training log to check the problem?

[xxxzhi]

Thanks for your response. 400 epochs, 128 batch size. That might be very time-consuming. My experiment has not been finished. I use 4 gpus with a batch size of 128*4 in 400 epochs. here is the log.
R50_mocot0.2_augrandcls_sim_400epochs_lr0.02_t1_gpu4.log

here is the log of 2 gpu
R50_mocot0.2_augrandcls_sim_400epochs_lr0.02_t1_gpu2.log

how many gpu hours your code run with 400 epochs on Imagenet-LT?

[jiequancui]

Hi,
You may misunderstand it.
We use 4gpus with a batch size of 128 (32 images per gpu) and train 400 epochs. It takes about 2 days with 4gpus for ResNet-50 on ImageNet-LT.

I think that, with 4gpus, running with our scripts in this repo can reproduce our results reported in the paper.

[xxxzhi]

Thanks, I will check your released log. I want to use larger batch size to reduce the training time.