Training Loss is Puzzling

[XYZ-qiyh]

I have re-trained the network using the same cofiguration, but the training loss don't converge.

[XYZ-qiyh]

Hello @xy-guo . Thanks for your amazing work. I have re-trained the network using the same cofiguration, but the training loss don't converge.
I don't think the training loss is very plausible. Do you have this problem??
Regards.

[kwea123]

hi, did you continue training? What does the final loss curve look like? Also does the depth in the image tab seem improving? My training loss also fluctuates a lot.

[zhiwenfan]

Hello @xy-guo . Thanks for your amazing work. I have re-trained the network using the same cofiguration, but the training loss don't converge.
I don't think the training loss is very plausible. Do you have this problem??
Regards.

Have you ever try to use larger batchsize?

[xy-guo]

I use 8 gpus to train the model. It seems your training loss is converging, try using larger smoothing weight in tensorboard.

[kwea123]

Do you have the final metrics of abs_depth_error, thres2mm_error, thres4mm_error, thres8mm_error? I want to compare with my results, thank you.

[kwea123]

@xy-guo

[xy-guo]

Sorry I have lost the final metrics information. @kwea123

[whubaichuan]

@kwea123 Have you try bigger batchsize? What's the difference with the training loss of batchsize=1?

[kwea123]

The model consumes too much memory so I cannot try bigger batchsize (batchsize=1 requires ~8GB). You will need multiple gpus.

[whubaichuan]

@QTODD Hi, have you solve the problem that the training loss is not plausible?

[whubaichuan]

@xy-guo Do you use 8 GPUS to test?

[kwea123]

I tried, even 8GPUs with batchsize 4 results still in fluctuating losses. I think it's insolvable then; although it doesn't worsen the model performance..