Convolution too slow for depthwise when input channels != output channels

[dskhudia]

This is the root cause of https://discuss.pytorch.org/t/got-slow-speed-on-quantized-model-with-fbgemm-on-x86/74439

fbgemmConv takes the im2col route for such cases and it's too slow. Here are the results for a shape I benchmarked.

"N", "IC", "OC", "H", "W", "G", "kernel", "stride", "pad"
1, 128, 256, 32, 100, 128, 3, 1, 1

FP32 time: 3.55 ms
fbgemmConv time: 86.76ms

cc @jspark1105

[jspark1105]

This is not technically depth-wise convolution because we have 2 output channels per group. We should see if this case should be handled by generalizing the depth-wise kernel or the group conv kernel.

[dskhudia]

PyTorch defines it as depthwise :-| https://pytorch.org/docs/stable/nn.html#conv2d

When groups == in_channels and out_channels == K * in_channels, where K is a positive integer, this operation is also termed in literature as depthwise convolution.

[jspark1105]

I see. Let me see if we can generalize group conv kernel (first generalize it to support C_per_G != K_per_G and then C_per_G == 1 and K_per_G == 2).

[dskhudia]

Would it be easier to add this to depthwise?

[dskhudia]

Also if we end up modifying groupwise, we should go for K_per_G == N

[jspark1105]

Yes just looked at groupwise again. I agree it will be easier to add this to depthwise.