BatchNormalization invalid configuration argument

[octavian-ganea]

Hi,

Can someone help with this hard to understand BatchNormalization error:

THCudaCheck FAIL file=/local/home/ganeao/torch/extra/cunn/lib/THCUNN/BatchNormalization.cu line=257 error=9 : invalid configuration argument /local/home/ganeao/torch/install/bin/luajit: ...home/ganeao/torch/install/share/lua/5.1/nn/Container.lua:67: In 1 module of nn.Sequential: In 1 module of nn.ConcatTable: In 2 module of nn.Sequential: In 4 module of nn.Sequential: In 1 module of nn.ParallelTable: /local/home/ganeao/torch/install/share/lua/5.1/nn/THNN.lua:109: cuda runtime error (9) : invalid configuration argument at /local/home/ganeao/torch/extra/cunn/lib/THCUNN/BatchNormalization.cu:257 stack traceback: ...ome/ganeao/torch/install/share/lua/5.1/fb/util/error.lua:76: in function <...ome/ganeao/torch/install/share/lua/5.1/fb/util/error.lua:72> [C]: in function 'v' /local/home/ganeao/torch/install/share/lua/5.1/nn/THNN.lua:109: in function 'BatchNormalization_updateOutput' ...ao/torch/install/share/lua/5.1/nn/BatchNormalization.lua:112: in function <...ao/torch/install/share/lua/5.1/nn/BatchNormalization.lua:101> [C]: in function 'real_xpcall' ...ome/ganeao/torch/install/share/lua/5.1/fb/util/error.lua:85: in function 'xpcall' ...home/ganeao/torch/install/share/lua/5.1/nn/Container.lua:63: in function 'rethrowErrors' .../ganeao/torch/install/share/lua/5.1/nn/ParallelTable.lua:12: in function <.../ganeao/torch/install/share/lua/5.1/nn/ParallelTable.lua:10> [C]: in function 'real_xpcall' ...ome/ganeao/torch/install/share/lua/5.1/fb/util/error.lua:85: in function 'xpcall' ... ...ome/ganeao/torch/install/share/lua/5.1/fb/util/error.lua:85: in function 'xpcall' ...home/ganeao/torch/install/share/lua/5.1/nn/Container.lua:63: in function 'rethrowErrors' ...ome/ganeao/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'forward'

The above error happens in a forward pass over the network. My network configuration is a bit complicated, but essentially I add nn.BatchNormalization immediately after an nn.LookupTable or after a nn.Linear , each time receiving the same error. If I remove BN, everything works fine. Any hints on how to debug this would be appreciated.

[soumith]

if you insert an nn.Probe() layer right between the nn.Linear and nn.BatchNormalization and get the shape of the input to the batchnorm layer, you will get the: input shape that is failing.
Then in a small script, construct a random input of that shape, construct a BatchNormalization layer, and push the same input shape forward. See if it also fails here.

[soumith]

nn.Probe is defined in nnx. require 'nnx'

[octavian-ganea]

I tried using nn.Probe. That's what I get:
`.output

• size = 147000x100
• mean = -0.0071470360271633
• std = 0.24890649318695
• min = -1.2087908983231
• max = 1.3189998865128
• time since last probe = 3.3ms `

The size is what I expected.

[soumith]

ok, this is likely a bug in nn.BatchNormalization. @colesbury who implemented it def didn't expect a batch size of 147,000 :)
We'll fix it.

[soumith]

The bug comes from kernel launch, where you define the number of blocks and threads in launching the CUDA kernel here: https://github.com/torch/cunn/blob/master/lib/THCUNN/BatchNormalization.cu#L248-L250

[octavian-ganea]

Ok, thanks! Actually this 147K comes from 700 x 210: I'm using a 700 length mini batch containing 210 word embeddings of 100 dimensions. However, BN works only with 2D tensors so I had to apply a View before BN. If there is a better way to do this, please let me know.

[soumith]

fixed now in trunk.