Local Response Normalization backpropagation

[4fur4]

Hi!

Im checking the implementation in C++ of the local response normalization here:

https://github.com/vlfeat/matconvnet/blob/master/matlab/src/bits/normalize.cpp

Based on Hinton´s paper http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf we have that the response-normalized activity is given by the formula (notation adapted for readability):

which totally fits with the implementation mentioned above. To get the back-propagation formulas we have that if

then

If im not wrong, this maps to the C++ implementation as

which should lead to the formula

however, the implementation is (lines 276-280)

Note the change zat(q) -> zat(t). Is there anything wrong there that I didnt notice?

Thank you!

Urko

[johnny5550822]

I am interested in the answer of this question too.

[jroose]

I'm not convinced that the negative term really makes a difference. In Hinton's paper he sets the value of K=2, beta=0.75, and alpha=1e-4. Using those values, I believe the negative term is almost certainly negligible because the positive term is likely to be so much larger.

Math isn't my specialty though, so I'd be interested in your opinion Urko. Did you happen to come to a similar conclusion?

[easten20]

I think the result is the same, it just assigned to different variable though,
instead of assigning it to yat(t) they assign it to yat(q).

https://github.com/vlfeat/matconvnet/blob/b7dd9c963541582faa04572f510e0cc20545e086/matlab/src/bits/impl/normalize_cpu.cpp

line 271 -275

yat(q) -= zat(t) * xat(t) * xat(q) * ab2 * Lbeta1 ;

is that line of code did you mean?