understand multiple loss debug info from caffe_output.log

[RSly]

Hi

I try to understand the loss information in the caffe_output.log

I have two normalized loss layers, each with weight 0.5.

1. however, for some reason the accumulated loss is not exactly the weighted sum of the loss values printed for each loss layer (see below)

2. also I use softmaxwithloss layer (called smloss) with normalize: true. How is that this smloss exceeds 1 in my example!?

any suggestion is appreciated :)
I am using digits 5.1-dev + NVcaffe 0.15.14
here is the example:

I0222 14:28:37.458081 10646 caffe.cpp:231] Starting Optimization
I0222 14:28:37.458094 10646 solver.cpp:304] Solving
I0222 14:28:37.458099 10646 solver.cpp:305] Learning Rate Policy: exp
I0222 14:28:37.461014 10646 solver.cpp:362] Iteration 0, Testing net (#0)
I0222 14:28:43.160092 10646 solver.cpp:429] Test net output #0: accuracy = 0.981503
I0222 14:28:43.160148 10646 solver.cpp:429] Test net output #1: diceloss = 0.624784 (* 0.5 = 0.312392 loss)
I0222 14:28:43.160156 10646 solver.cpp:429] Test net output #2: smloss = 0.044091 (* 0.5 = 0.0220455 loss)
I0222 14:28:46.598737 10646 solver.cpp:242] Iteration 0 (0 iter/s, 9.14068s/4 iter), loss = 0.335846
I0222 14:28:46.598793 10646 solver.cpp:261] Train net output #0: accuracy = 0.986954
I0222 14:28:46.598804 10646 solver.cpp:261] Train net output #1: diceloss = 0.643004 (* 0.5 = 0.321502 loss)
I0222 14:28:46.598811 10646 solver.cpp:261] Train net output #2: smloss = 0.0286878 (* 0.5 = 0.0143439 loss)
I0222 14:28:46.598834 10646 sgd_solver.cpp:106] Iteration 0, lr = 0.01
I0222 14:29:00.067633 10646 solver.cpp:242] Iteration 4 (0.296978 iter/s, 13.469s/4 iter), loss = 1.44211
I0222 14:29:00.067687 10646 solver.cpp:261] Train net output #0: accuracy = 0.631965
I0222 14:29:00.067698 10646 solver.cpp:261] Train net output #1: diceloss = 0.716986 (* 0.5 = 0.358493 loss)
I0222 14:29:00.067703 10646 solver.cpp:261] Train net output #2: smloss = 2.16724 (* 0.5 = 1.08362 loss)
I0222 14:29:00.067713 10646 sgd_solver.cpp:106] Iteration 4, lr = 0.0099799
I0222 14:29:14.040252 10646 solver.cpp:242] Iteration 8 (0.286272 iter/s, 13.9727s/4 iter), loss = 0.860196

[gheinrich]

This is a Caffe question, not a DIGITS question...

however, for some reason the accumulated loss is not exactly the weighted sum of the loss values printed for each loss layer (see below)

Is your Diceloss really a loss?

also I use softmaxwithloss layer (called smloss) with normalize: true. How is that this smloss exceeds 1 in my example!?

"Normalize" does not mean that the loss will be less than 1. It means the loss does not depend on the number of samples.

[RSly]

@gheinrich thanks, yes the diceloss is a loss,
maybe there is something going on with the diff values, something similar to this
BVLC/caffe#2895

I close this as it is not related to digits.