A problem for training SqueezeSeg model

[jsgaobiao]

Hello.
I'm trying to train the SqueezeSeg model from random inital weights. However, the iou seems too low after 16h training (about 16.2%).

GPU: a NVIDIA TITAN X.
Training set: sequence 0~7
Validation set: sequence 8
Arch config: squeezeseg.yaml (I just set batch_size=18 due to GPU memory limitation)

There are some information about my training in this pic may help.
Do I need more time for training? Could anyone give me some suggestions? @tano297
Thanks a lot !

[jbehley]

I would say that your curves indicate that you need more training time. You also see that the losses are still going down and the IoU goes still up.

[jsgaobiao]

I would say that your curves indicate that you need more training time. You also see that the losses are still going down and the IoU goes still up.

Thank you for your reply !
I will keep training it and see what will happen.
Do you know how long it usually takes to train a model like SqueezeSeg ?

[tano297]

I have just checked, and training was done for roughly 300 epochs.
Convergence will be faster than that, at 150-200 epochs, but I kept training for fairness to the original method, because I wanted to make sure there would be no improvement. I'm closing this now, if there is more questions feel free to keep commenting.