(oicrbaseline+mist wi reg) setup map50 result 41.05

[liz6688]

Thanks for your reply.

As our computing resources are limited, there is no way to use 8 gpus. In order to get the performance of the paper, I setup my experiments as follows (oicrbaseline+mist wi reg):
removed the cbd module
BATCH_SIZE=2
base lr = 0.0025 (0.01/4)
torch = 1.5.0
cuda=9.2
The result of voc_2007_test data is map50 = 41.5. There is still a big gap from the results 51.4 in the paper. Do you have any suggestions for further improvement?

[jason718]

have you tried the ITER_SIZE flag? Also, in addition to decrease the LR, you should also train longer (4x).

Unfortunately we never test our model using 1 or 2 GPUs, so I'm not sure about the best parameters.

[liz6688]

Do the all the related parameters WARMUP_ITERS, STEPS, MAX_ITER (4x)?
I will try the ITER_SIZE flag later.

[bradezard131]

You should only have to set ITER_SIZE, the rest should be automatically scaled for you, it'll throw up a warning to let you know.
Try:
BATCH_SIZE=1
BASE_LR=0.01
ITER_SIZE=8

I was able to reproduce with these settings

[liz6688]

ok, thanks. I will try.

[liz6688]

Setup:
BATCH_SIZE=1
BASE_LR=0.01
ITER_SIZE=8
As there is no test during training, your result is based on the best model or the last iteration model?

[bityangke]

Setup:
BATCH_SIZE=2
BASE_LR=0.01
ITER_SIZE=4
Results:
mAP: 0.4327 aeroplane : 0.6249 bicycle : 0.7092 bird : 0.3490 boat : 0.1578 bottle : 0.1341 bus : 0.6963 car : 0.7411 cat : 0.4068 chair : 0.0145 cow : 0.6304 diningtable : 0.4927 dog : 0.4893 horse : 0.2174 motorbike : 0.7023 person : 0.3542 pottedplant : 0.1714 sheep : 0.4337 sofa : 0.5014 train : 0.5322 tvmonitor : 0.2950

[bradezard131]

I got ~50 at last epoch with BS=1, ITER=8. ITER_SIZE is an approximation and a somewhat shaky one at that, it isn't surprising for it to perform differently. If we want to reproduce their results then using their original hyperparameters is the only way to be sure

[jason718]

Thanks @bradezard131 @bradezard131 @bityangke @liz6688 for sharing.

@bradezard131 contributed to this feature "ITER_SIZE". I believe he knows better than me how to use it. Unfortunately we didn't test under 1/2 GPUs setting.

We report the best performance as we found it's usually not achieved at the last iteration, especially for small dataset like VOC.

The things I don't understand is why the variance is so big between @bityangke @bradezard131 reported. Seems strange. I would still recommend using 8GPUs and I'll look into this. Thanks again.

[bradezard131]

Hopefully I will be able to clean up and release my re-implementation soon, which works with 1 GPU and is somewhat simplified

[liz6688]

Thanks. Looking forward to your re-implementation.