Terrible Loss when training on SSD MobileNet v2 320x320

[Rodeazzo]

Hi, I'm trying to train using ssd mobilenet v2 320x320 using tensorflow 2 and a datatset consisting of 2.4k train and 600 test images. I've been struggling with finding a suitable setting to train on the aforementioned model. I tried modifying the learning_rate_base from its original .8 (which was giving me insane loss values), to 0.03, 0.2, 8e-2 and now 0.004. Pretty much the same occurrence happens as the image below:

Aside of learning rate, I changed warmup_learning_rate to 0.001, and use_dropout to true from convolutional_box_predictor.

Anyone has any idea why this is happening? Should I let it train for longer maybe it stabilises?

Thanks

[mrinal18]

Hey @Rodeazzo, could you share your code? its not possible to just judge it by looking at the learning rate or convolutional_box_predictor. Will take a look and let you know.

[Rodeazzo]

Here's a link to my Google Colab document:
https://colab.research.google.com/drive/19YKxOg2y4tEcIKeYrfl6TRkoPxmK_Aqw?usp=sharing