Strange training loss and test result

[Owen-Liuyuxuan]

I ran

CUDA_VISIBLE_DEVICES=0 python3 -m manydepth.train --data_path /home/kitti_raw/ --log_dir workdirs/ --model_name manydepth

epoch 0 | batch 0 | examples/s: 2.8 | loss: 0.00810 | time elapsed: 00h00m09s | time left: 00h00m00s
epoch 0 | batch 250 | examples/s: 22.4 | loss: 0.00049 | time elapsed: 00h02m34s | time left: 11h19m40s
epoch 0 | batch 500 | examples/s: 20.6 | loss: 0.00024 | time elapsed: 00h05m00s | time left: 10h59m55s
epoch 0 | batch 750 | examples/s: 22.2 | loss: 0.00013 | time elapsed: 00h07m26s | time left: 10h50m49s
epoch 0 | batch 1000 | examples/s: 21.5 | loss: 0.00008 | time elapsed: 00h09m52s | time left: 10h44m55s
epoch 0 | batch 1250 | examples/s: 21.0 | loss: 0.00018 | time elapsed: 00h12m18s | time left: 10h41m15s
epoch 0 | batch 1500 | examples/s: 21.7 | loss: 0.00019 | time elapsed: 00h14m45s | time left: 10h37m37s
epoch 0 | batch 1750 | examples/s: 21.5 | loss: 0.00011 | time elapsed: 00h17m10s | time left: 10h33m45s

The loss is extremely small.

The result on the 12 epoch (it should be reasonable at this moment), but is not.

abs_rel |   sq_rel |     rmse | rmse_log |       a1 |       a2 |       a3 |
&   0.443  &   4.757  &  12.083  &   0.588  &   0.303  &   0.561  &   0.766  \\

[JamieWatson683]

Hi, thanks for trying out the repo!

Interesting - yes that loss does appear to be very low right from the start. I just tried a fresh github clone and see an initial loss (i.e. batch 0) of 0.34110, which seems more sensible.

Can you take a look at the tensorboard logs and switch to the images tab? Does it look like the images are being loaded correctly? And if so, how do the disparity predictions look? Feel free to post a screenshot.

A loss that low straight away suggests that it might be loading blank images. If you downloaded the raw KITTI and have png images then try adding a --png flag to your training command.

In the meantime I will push a small fix to raise an error if no image files are found (currently it's caught in a try except block).

Thanks a lot

[Owen-Liuyuxuan]

Hi, thanks for trying out the repo!

Interesting - yes that loss does appear to be very low right from the start. I just tried a fresh github clone and see an initial loss (i.e. batch 0) of 0.34110, which seems more sensible.

Can you take a look at the tensorboard logs and switch to the images tab? Does it look like the images are being loaded correctly? And if so, how do the disparity predictions look? Feel free to post a screenshot.

A loss that low straight away suggests that it might be loading blank images. If you downloaded the raw KITTI and have png images then try adding a --png flag to your training command.

In the meantime I will push a small fix to raise an error if no image files are found (currently it's caught in a try except block).

Thanks a lot

You are right, I am using png files, which needs a flag even when I am in monodepth2. But I did not expect this to be error-less/warning-less. Thank you for your answer

[mdfirman]

Thanks @Owen-Liuyuxuan for:

1. Being an early adopter of our repo
2. Reporting the bug when you found it, and
3. Reporting back when the fix worked!

And thanks @JamieWatson683 for the quick fix for this.