Local Response Normalization

[Kaixhin]

Functional implementation + module for LRN based on @jiecaoyu's implementation. Let me know if there should be any naming changes/if we want an across/within channel flag like Caffe/what to do with CrossMapLRN2d etc.

Originally AlexNet doesn't seem to do an average (just sums), whilst Caffe does do an average... I've written the docs as a sum as in the original, so will either need to change the docs or implementation depending on which one we want (or even make it another option).

[apaszke]

idk what the formula for LRN is, so I haven't checked that, but the code looks good in general

[Kaixhin]

Pinging @jiecaoyu @fmassa for an additional review (to check the formula/address any points in my initial comment) before merging.

[vadimkantorov]

+1 for clearing up the difference with CrossMapLRN2d (is it across-channels / within-channel thing?)

[Kaixhin]

@vadimkantorov this is across-channels, same as CrossMapLRN2d (which should be deprecated). The within-channel version seems to be far less common, so I'd go for merging this across-channel version in (which actually now supports 1D, 2D, 3D, 4D etc.), and then adding that as another option with an extra optional flag if someone wants it.

@apaszke The canonical reference is AlexNet, section 3.3, which sums the normalisation bit across channels. However, Torch documents SpatialCrossMapLRN as being a mean over channels, as does Caffe, so I'm changing the implementation and docs back to using the mean to be consistent with most implementations.

[vadimkantorov]

@Kaixhin Python disallows starting a variable name with a digit (minor docs thing):

>>> 2d_signal = autograd.Variable(torch.randn(32, 5, 24, 24))
>>> 4d_signal = autograd.Variable(torch.randn(16, 5, 7, 7, 7, 7))
>>> 2d_output = lrn(2d_signal)
>>> 4d_output = lrn(4d_signal)

[Kaixhin]

@vadimkantorov thanks for noting - have adjusted in #4681 since I forgot to add LRN to the autogenerated docs.