training loss not decrease

[skylian]

I'm using the training scripts from the examples. I used batch size 15 instead of 30 due to memory limits. I tried learning 1e-4 or 5e-5. The training loss does not decrease once reaching around 4.5. The following is some part of the log.

I0310 15:34:42.297404 10565 blocking_queue.cpp:50] Data layer prefetch queue empty
I0310 15:34:43.389981 10565 solver.cpp:406] Test net output #0: accuracy/top-1 = 0.133333
I0310 15:34:43.390017 10565 solver.cpp:406] Test net output #1: accuracy/top-5 = 0.4
I0310 15:34:43.390025 10565 solver.cpp:406] Test net output #2: loss = 44.5392 (* 1 = 44.5392 loss)
I0310 15:34:44.678773 10565 solver.cpp:229] Iteration 0, loss = 70.298
I0310 15:34:44.678804 10565 solver.cpp:245] Train net output #0: loss = 70.298 (* 1 = 70.298 loss)
I0310 15:34:44.678817 10565 sgd_solver.cpp:106] Iteration 0, lr = 0.0001
I0310 15:35:58.922013 10565 solver.cpp:229] Iteration 20, loss = 17.0077
I0310 15:35:58.922139 10565 solver.cpp:245] Train net output #0: loss = 17.0077 (* 1 = 17.0077 loss)
I0310 15:35:58.922149 10565 sgd_solver.cpp:106] Iteration 20, lr = 0.0001
I0310 15:37:15.157956 10565 solver.cpp:229] Iteration 40, loss = 11.0352
I0310 15:37:15.158082 10565 solver.cpp:245] Train net output #0: loss = 11.0352 (* 1 = 11.0352 loss)
I0310 15:37:15.158095 10565 sgd_solver.cpp:106] Iteration 40, lr = 0.0001
I0310 15:38:32.929126 10565 solver.cpp:229] Iteration 60, loss = 6.5498
I0310 15:38:32.929203 10565 solver.cpp:245] Train net output #0: loss = 6.5498 (* 1 = 6.5498 loss)
I0310 15:38:32.929211 10565 sgd_solver.cpp:106] Iteration 60, lr = 0.0001
I0310 15:39:49.492420 10565 solver.cpp:229] Iteration 80, loss = 6.02491
I0310 15:39:49.492509 10565 solver.cpp:245] Train net output #0: loss = 6.02491 (* 1 = 6.02491 loss)
I0310 15:39:49.492518 10565 sgd_solver.cpp:106] Iteration 80, lr = 0.0001
I0310 15:41:04.777462 10565 solver.cpp:229] Iteration 100, loss = 5.15338
I0310 15:41:04.778811 10565 solver.cpp:245] Train net output #0: loss = 5.15338 (* 1 = 5.15338 loss)
I0310 15:41:04.778821 10565 sgd_solver.cpp:106] Iteration 100, lr = 0.0001
I0310 15:42:20.587851 10565 solver.cpp:229] Iteration 120, loss = 4.76454
I0310 15:42:20.588001 10565 solver.cpp:245] Train net output #0: loss = 4.76454 (* 1 = 4.76454 loss)
I0310 15:42:20.588029 10565 sgd_solver.cpp:106] Iteration 120, lr = 0.0001
I0310 15:43:34.851495 10565 solver.cpp:229] Iteration 140, loss = 5.13133
I0310 15:43:34.851580 10565 solver.cpp:245] Train net output #0: loss = 5.13133 (* 1 = 5.13133 loss)
I0310 15:43:34.851598 10565 sgd_solver.cpp:106] Iteration 140, lr = 0.0001
I0310 15:44:50.394881 10565 solver.cpp:229] Iteration 160, loss = 4.92464
I0310 15:44:50.395270 10565 solver.cpp:245] Train net output #0: loss = 4.92464 (* 1 = 4.92464 loss)
I0310 15:44:50.395283 10565 sgd_solver.cpp:106] Iteration 160, lr = 0.0001
I0310 15:46:04.264691 10565 solver.cpp:229] Iteration 180, loss = 4.92802
I0310 15:46:04.264755 10565 solver.cpp:245] Train net output #0: loss = 4.92802 (* 1 = 4.92802 loss)
I0310 15:46:04.264765 10565 sgd_solver.cpp:106] Iteration 180, lr = 0.0001
I0310 15:47:20.315587 10565 solver.cpp:229] Iteration 200, loss = 4.70705
I0310 15:47:20.315706 10565 solver.cpp:245] Train net output #0: loss = 4.70705 (* 1 = 4.70705 loss)
I0310 15:47:20.315716 10565 sgd_solver.cpp:106] Iteration 200, lr = 0.0001
I0310 15:48:35.375715 10565 solver.cpp:229] Iteration 220, loss = 4.65431
I0310 15:48:35.375816 10565 solver.cpp:245] Train net output #0: loss = 4.65431 (* 1 = 4.65431 loss)
I0310 15:48:35.375834 10565 sgd_solver.cpp:106] Iteration 220, lr = 0.0001
I0310 15:49:48.752303 10565 solver.cpp:229] Iteration 240, loss = 4.70415
I0310 15:49:48.752403 10565 solver.cpp:245] Train net output #0: loss = 4.70415 (* 1 = 4.70415 loss)
I0310 15:49:48.752413 10565 sgd_solver.cpp:106] Iteration 240, lr = 0.0001
I0310 15:51:03.857997 10565 solver.cpp:229] Iteration 260, loss = 4.74508
I0310 15:51:03.858083 10565 solver.cpp:245] Train net output #0: loss = 4.74508 (* 1 = 4.74508 loss)

[chuckcho]

Could you let me know how the learning goes if you leave it running for some more time (a couple of hours)?

[skylian]

The loss is still around 4.5 up to 3000 iterations. I suspect it is the problem of image codecs. Looking at your train and test list files, I observed that the number of frames I extracted from some videos are different from yours. I managed to find a version of ffmpeg which generates the same frames as yours, and the loss looks normal now.

[chuckcho]

@skylian Good to hear that.

[yuta1125tp]

@skylian I got the same problem. Could you tell me the version of ffmpeg which generates the same frames.

[pelun]

Can I run this project in window? @chuckcho

[chuckcho]

@pelun not tested on Windows.

[pelun]

Thanks.