Unable to re-generate the numbers

[umariqb]

Hi,

I am trying to re-generate the numbers reported in your ICCV paper from scratch. Everything works fine when I use the pre-trained models. But when I train the models myself they seem to classify all pixels to the background. The only files I generated myself are LMDBs using the provided code (/extras/generate_lmdb.py).

Can you please also tell your loss value at 35,000 iterations?

Here is the snapshot of last 100 iterations in my case:

I0601 04:28:17.000000 34992 solver.cpp:489] Iteration 34700, lr = 1e-06
I0601 04:28:33.000000 34992 solver.cpp:214] Iteration 34720, loss = 0.0137566
I0601 04:28:33.000000 34992 solver.cpp:229] Train net output #0: loss = 0.0137567 (* 1 = 0.0137567 loss)
I0601 04:28:33.000000 34992 solver.cpp:489] Iteration 34720, lr = 1e-06
I0601 04:28:53.000000 34992 solver.cpp:214] Iteration 34740, loss = 0.0162213
I0601 04:28:53.000000 34992 solver.cpp:229] Train net output #0: loss = 0.0162215 (* 1 = 0.0162215 loss)
I0601 04:28:53.000000 34992 solver.cpp:489] Iteration 34740, lr = 1e-06
I0601 04:29:11.000000 34992 solver.cpp:214] Iteration 34760, loss = 0.0857187
I0601 04:29:11.000000 34992 solver.cpp:229] Train net output #0: loss = 0.0857188 (* 1 = 0.0857188 loss)
I0601 04:29:11.000000 34992 solver.cpp:489] Iteration 34760, lr = 1e-06
I0601 04:29:29.000000 34992 solver.cpp:214] Iteration 34780, loss = 0.365484
I0601 04:29:29.000000 34992 solver.cpp:229] Train net output #0: loss = 0.365484 (* 1 = 0.365484 loss)
I0601 04:29:29.000000 34992 solver.cpp:489] Iteration 34780, lr = 1e-06
I0601 04:29:47.000000 34992 solver.cpp:214] Iteration 34800, loss = 0.427323
I0601 04:29:47.000000 34992 solver.cpp:229] Train net output #0: loss = 0.427323 (* 1 = 0.427323 loss)
I0601 04:29:47.000000 34992 solver.cpp:489] Iteration 34800, lr = 1e-06
I0601 04:30:05.000000 34992 solver.cpp:214] Iteration 34820, loss = 0.272257
I0601 04:30:05.000000 34992 solver.cpp:229] Train net output #0: loss = 0.272258 (* 1 = 0.272258 loss)
I0601 04:30:05.000000 34992 solver.cpp:489] Iteration 34820, lr = 1e-06
I0601 04:30:23.000000 34992 solver.cpp:214] Iteration 34840, loss = 0.0660284
I0601 04:30:23.000000 34992 solver.cpp:229] Train net output #0: loss = 0.0660286 (* 1 = 0.0660286 loss)
I0601 04:30:23.000000 34992 solver.cpp:489] Iteration 34840, lr = 1e-06
I0601 04:30:40.000000 34992 solver.cpp:214] Iteration 34860, loss = 0.696421
I0601 04:30:40.000000 34992 solver.cpp:229] Train net output #0: loss = 0.696421 (* 1 = 0.696421 loss)
I0601 04:30:40.000000 34992 solver.cpp:489] Iteration 34860, lr = 1e-06
I0601 04:30:58.000000 34992 solver.cpp:214] Iteration 34880, loss = 0.096779
I0601 04:30:58.000000 34992 solver.cpp:229] Train net output #0: loss = 0.0967792 (* 1 = 0.0967792 loss)
I0601 04:30:58.000000 34992 solver.cpp:489] Iteration 34880, lr = 1e-06
I0601 04:31:16.000000 34992 solver.cpp:214] Iteration 34900, loss = 0.216992
I0601 04:31:16.000000 34992 solver.cpp:229] Train net output #0: loss = 0.216992 (* 1 = 0.216992 loss)
I0601 04:31:16.000000 34992 solver.cpp:489] Iteration 34900, lr = 1e-06
I0601 04:31:33.000000 34992 solver.cpp:214] Iteration 34920, loss = 0.047195
I0601 04:31:33.000000 34992 solver.cpp:229] Train net output #0: loss = 0.0471952 (* 1 = 0.0471952 loss)
I0601 04:31:33.000000 34992 solver.cpp:489] Iteration 34920, lr = 1e-06
I0601 04:31:51.000000 34992 solver.cpp:214] Iteration 34940, loss = 0.32279
I0601 04:31:51.000000 34992 solver.cpp:229] Train net output #0: loss = 0.32279 (* 1 = 0.32279 loss)
I0601 04:31:51.000000 34992 solver.cpp:489] Iteration 34940, lr = 1e-06
I0601 04:32:08.000000 34992 solver.cpp:214] Iteration 34960, loss = 0.0153671
I0601 04:32:08.000000 34992 solver.cpp:229] Train net output #0: loss = 0.0153672 (* 1 = 0.0153672 loss)
I0601 04:32:08.000000 34992 solver.cpp:489] Iteration 34960, lr = 1e-06
I0601 04:32:26.000000 34992 solver.cpp:214] Iteration 34980, loss = 0.0963567
I0601 04:32:26.000000 34992 solver.cpp:229] Train net output #0: loss = 0.0963568 (* 1 = 0.0963568 loss)
I0601 04:32:26.000000 34992 solver.cpp:489] Iteration 34980, lr = 1e-06
35000 iterations t = 892.211614132

Any help is welcome.

Thanks!

[umariqb]

Works now! I generated my own LMDB files, and used the provided hf5 files. And ran for 35000 iterations.