Local Response Normalization#4667
Conversation
|
|
||
|
|
||
| def local_response_norm(input, size, alpha=1e-4, beta=0.75, k=1): | ||
| div = avg_pool3d(input.pow(2).unsqueeze(1), (size, 1, 1), |
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
| def local_response_norm(input, size, alpha=1e-4, beta=0.75, k=1): | ||
| div = avg_pool3d(input.pow(2).unsqueeze(1), (size, 1, 1), | ||
| stride=1, padding=(size // 2, 0, 0)).squeeze(1) | ||
| div.mul_(alpha).add_(k).pow_(beta) |
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
| from .. import functional as F | ||
|
|
||
|
|
||
| class LocalResponseNorm2d(Module): |
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
| ) | ||
|
|
||
|
|
||
| def local_response_norm(input, size, alpha=1e-4, beta=0.75, k=1): |
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
apaszke
left a comment
apaszke
left a comment
There was a problem hiding this comment.
idk what the formula for LRN is, so I haven't checked that, but the code looks good in general
| div = div.view(sizes[0], 1, sizes[1], sizes[2], sizes[3] * sizes[4]) | ||
| div = pad(div, (0, 0, 0, 0, size // 2, (size - 1) // 2)) | ||
| div = avg_pool3d(div, (size, 1, 1), stride=1).squeeze(1) | ||
| div = div.view(sizes[0], sizes[1], sizes[2], sizes[3], sizes[4]) |
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
|
+1 for clearing up the difference with CrossMapLRN2d (is it across-channels / within-channel thing?) |
Specify 'across-channels'
|
@vadimkantorov this is across-channels, same as CrossMapLRN2d (which should be deprecated). The within-channel version seems to be far less common, so I'd go for merging this across-channel version in (which actually now supports 1D, 2D, 3D, 4D etc.), and then adding that as another option with an extra optional flag if someone wants it. @apaszke The canonical reference is AlexNet, section 3.3, which sums the normalisation bit across channels. However, Torch documents SpatialCrossMapLRN as being a mean over channels, as does Caffe, so I'm changing the implementation and docs back to using the mean to be consistent with most implementations. |
|
@Kaixhin Python disallows starting a variable name with a digit (minor docs thing): |
|
@vadimkantorov thanks for noting - have adjusted in #4681 since I forgot to add LRN to the autogenerated docs. |
* Local Response Normalization * Add 1D and 3D LRN * Generalise LRN to higher dims * Use mean instead of sum Specify 'across-channels'
5 participants
Functional implementation + module for LRN based on @jiecaoyu's implementation. Let me know if there should be any naming changes/if we want an across/within channel flag like Caffe/what to do with CrossMapLRN2d etc.
Originally AlexNet doesn't seem to do an average (just sums), whilst Caffe does do an average... I've written the docs as a sum as in the original, so will either need to change the docs or implementation depending on which one we want (or even make it another option).