How to improve training speed?

[lengly]

I think your work is really great. I try to run your code on Tesla V100 x 4, the speed of VGG16_VOC2007_512.py is 1.2 seconds/iter, and it may take around 40 hours to train the whole 120k itertions.

Do you have any idea to improve the speed?

I think connections between TCB may cause the whole model slow down. I think your idea is inspired by FPN. Did you try to remove connections between TCB? Then the whole model'll be more parallel. ( I know this may hurt the performance...)

[sfzhang15]

@lengly Hi, comparing with our Titan X (Maxwell) x 4, your speed is much faster! You can try to train SSD_VOC2007_512_VGG16, which do not have TCBs and ODM module. His speed may be similar to ours, so we think the connections between TCB is not the main reason. Every batch has 32 images with 512 x 512 resolution for VGG16, we think the speed 1.2 s/iter is totally acceptable. We do not try to remove connections between TCB, you can try it and check the speed.