Wrong output of single-channel channels_last convolution with channels_first input

[emilyfy]

🐛 Describe the bug

Asked about some unexpected behavior on this thread and was asked to raise an issue here.

I'm trying to run convolution with torch.channels_last memory format for the weights but contiguous input, and the output is wrong if the input channels is 1.

import torch

input = torch.randn(1, 1, 100, 100)
conv = torch.nn.Conv2d(1, 64, kernel_size=(3, 3), padding=(1, 1), bias=False)

with torch.no_grad():
    out_ref = conv(input)

conv.to(memory_format=torch.channels_last)
with torch.no_grad():
    out = conv(input)

print(torch.mean(torch.abs(out - out_ref)))

this prints some non-zero value.

Note:

• if input channel is not 1, the output is correct.
• this is observed in CPU only. If input and conv are both sent to .cuda(), the error will be zero.

Versions

PyTorch version: 1.12.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 11.2.3 (x86_64)
GCC version: Could not collect
Clang version: 12.0.0 (clang-1200.0.32.27)
CMake version: version 3.16.2
Libc version: N/A

Python version: 3.10.4 (main, Mar 31 2022, 03:38:35) [Clang 12.0.0 ] (64-bit runtime)
Python platform: macOS-10.16-x86_64-i386-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.1
[pip3] torch==1.12.0
[pip3] torchaudio==0.12.0
[conda] numpy 1.23.1 pypi_0 pypi
[conda] torch 1.12.0 pypi_0 pypi
[conda] torchaudio 0.12.0 pypi_0 pypi

cc @ezyang @gchanan @zou3519 @VitalyFedyunin @gujinghui @PenghuiCheng @XiaobingSuper @jianyuh @jamesr66a

[jingxu10]

Thanks for reporting this issue. We will look into it.

[jbschlosser]

Is this a mkldnn issue? I can repro this even using with torch.backends.mkldnn.flags(enabled=False):

[jingxu10]

I tested the following script with the latest commit 24d702d. It worked correctly on Ubuntu.
Could you verify this in your environment?

import torch

input = torch.randn(1, 1, 100, 100)
conv = torch.nn.Conv2d(1, 64, kernel_size=(3, 3), padding=(1, 1), bias=False)

with torch.no_grad():
    out_ref = conv(input)

input = input.to(memory_format=torch.channels_last) # Without this line, prints non-zero value.
conv.to(memory_format=torch.channels_last)
with torch.no_grad():
    out = conv(input)

print(torch.mean(torch.abs(out - out_ref)))

[mingfeima]

Fixed with #82392

When input is not explicitly converted to channels last while conv has, the output should also be in channels last. The root cause is that when input has IC of 1, compute_columns2d from \aten\src\ATen\native\ConvolutionMM2d.cpp would consider it as channels first:

We do have logic to make sure both input and weight have the same memory format even if they are given differently, like:

const Tensor input = self.contiguous(memory_format);

But for a N1HW input, .contiguous(MemoryFormat::ChannelsLast) would not change its stride , and its suggest_memory_format() still returns MemoryFormat::Contiguous. That's how it went wrong.

[ILoveSE]

Hello, I have a little question about this fix. @mingfeima
In PyTorch 1.12.0, if the input channel is 1, LazyConv2d with channels_last memory format will also return the wrong results.

import torch

torch.manual_seed(0)
input = torch.randn(1, 1, 100, 100)
def test():
	tmp_result= torch.nn.LazyConv2d(out_channels=64, kernel_size=(3, 3), padding=(1, 1), bias=False)
	return tmp_result
conv = test()

with torch.no_grad():
    out_ref = conv(input)

conv.to(memory_format=torch.channels_last)
with torch.no_grad():
    out = conv(input)

result=torch.mean(torch.abs(out - out_ref))

print(result)
# tensor(0.6507)

In 1.13.0 version, it returns tensor(0. ) which seems to be an expected result.
I want to confirm whether this bug in LazyConv2d is also related to \aten\src\ATen\native\ConvolutionMM2d.cpp and has been solved in this #82392.
Thank you very much!