SyncBatchNorm doesn't support 2 dimensions input? #194

flymark2010 · 2019-03-11T09:58:39Z

Hi,
I'm facing the issue that the program crash when the input for SyncBatchNorm is two dimensions. Here's the code:

import torch
import apex

model = apex.parallel.SyncBatchNorm(4).cuda()
data = torch.rand((8,4)).cuda()
output = model(data)

When running the code, error raised like this:

Traceback (most recent call last):
  File "syncbn_test.by", line 7, in <module>
    output = model(data)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/apex/parallel/optimized_sync_batchnorm.py", line 81, in forward
    return SyncBatchnormFunction.apply(input, self.weight, self.bias, self.running_mean, self.running_var, self.eps, self.training or not self.track_running_stats, exponential_average_factor, self.process_group, self.channel_last)
  File "/usr/local/lib/python3.5/dist-packages/apex/parallel/optimized_sync_batchnorm_kernel.py", line 27, in forward
    mean, var_biased = syncbn.welford_mean_var(input)
RuntimeError: Dimension out of range (expected to be in range of [-2, 1], but got 2) (maybe_wrap_dim at /pytorch/aten/src/ATen/core/WrapDimMinimal.h:18)

And everthing runs ok when data a 4 dims tensor.

Here is my environment:

Ubuntu 16.04
Python 3.5.2
Pytorch 1.0.1, installed with "pip install torch"
apex is installed with command:
  pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" .
cuda 10.0
nvidia driver 410.72

The text was updated successfully, but these errors were encountered:

mcarilli · 2019-03-11T16:37:05Z

@jjsjann123 have you observed this?

jjsjann123 · 2019-03-11T18:56:25Z

yeah, this is an overlook for my side.
I mistakenly thought this is not supported by torch.nn.BatchNormXd, but it is.

I'll patch it soon

flymark2010 · 2019-03-12T06:13:34Z

Thanks very much. Hope for reply once fixed!

jjsjann123 · 2019-03-12T23:48:48Z

I'll update on this thread once it's fixed, should be sometime this week (Currently occupied by some other work)

DTennant · 2019-03-15T12:22:31Z

I'm facing the same issue, hoping you guys can fix this soon..... many thanks!

jjsjann123 · 2019-03-15T18:25:22Z

The support should be really easy as we already have channel_last kernels there, that handles the same data stride as with 2 dimensional tensor.
I'll update later today or early tomorrow as soon as I finish my work at hand

supporting 2 dimensional input, resolving issue #194 Implementation: for 2d input, switching channel_last flag to true for better memory access pattern in the kernel.

jjsjann123 · 2019-03-15T23:19:39Z

@flymark2010 Sorry for the long wait for such a simple fix (priority issue on my side :/ )
PR with the fix has been issued. Feel free to cherry pick, I'll push for the merge as well.

supporting 2 dimensional input, resolving issue #194 Implementation: for 2d input, switching channel_last flag to true for better memory access pattern in the kernel.

mcarilli · 2019-03-25T16:13:07Z

Should be fixed via 0a99154.

hiyijian · 2019-03-26T08:05:31Z

I am using latest master and got below issue:

PS:
self.bn5 = SyncBatchNorm(512)

jjsjann123 · 2019-03-26T17:19:42Z

Thanks a lot for reporting this @hiyijian. I should have done a better job unit testing this.

Any chance I can have a repro? If it's something not related to 2-dimension tensor, maybe we want to have a new issue and tag me on that one :)

zhuyingSeu · 2019-09-24T08:35:17Z

Hello, I'm confused with the same problem.
The apex was installed on 16 Sep.

import torch
import apex
model = apex.parallel.SyncBatchNorm(4).cuda()
data = torch.rand((8,4)).cuda()
ouput = model(data)

Traceback (most recent call last):
File "", line 1, in
File "/gpfs01/user_home/zhuying/.conda/envs/py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "/gpfs01/user_home/zhuying/.conda/envs/py36/lib/python3.6/site-packages/apex/parallel/optimized_sync_batchnorm.py", line 85, in forward
return SyncBatchnormFunction.apply(input, z, self.weight, self.bias, self.running_mean, self.running_var, self.eps, self.training or not self.track_running_stats, exponential_average_factor, self.process_group, self.channel_last, self.fuse_relu)
File "/gpfs01/user_home/zhuying/.conda/envs/py36/lib/python3.6/site-packages/apex/parallel/optimized_sync_batchnorm_kernel.py", line 27, in forward
mean, var_biased = syncbn.welford_mean_var(input)
RuntimeError: Dimension out of range (expected to be in range of [-2, 1], but got 2) (maybe_wrap_dim at /tmp/pip-req-build-p5q91txh/c10/core/WrapDimMinimal.h:20)

diggerdu · 2019-09-29T07:44:33Z

Hello, I'm confused with the same problem.
The apex was installed on 16 Sep.

import torch
import apex
model = apex.parallel.SyncBatchNorm(4).cuda()
data = torch.rand((8,4)).cuda()
ouput = model(data)

Traceback (most recent call last):
File "", line 1, in
File "/gpfs01/user_home/zhuying/.conda/envs/py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "/gpfs01/user_home/zhuying/.conda/envs/py36/lib/python3.6/site-packages/apex/parallel/optimized_sync_batchnorm.py", line 85, in forward
return SyncBatchnormFunction.apply(input, z, self.weight, self.bias, self.running_mean, self.running_var, self.eps, self.training or not self.track_running_stats, exponential_average_factor, self.process_group, self.channel_last, self.fuse_relu)
File "/gpfs01/user_home/zhuying/.conda/envs/py36/lib/python3.6/site-packages/apex/parallel/optimized_sync_batchnorm_kernel.py", line 27, in forward
mean, var_biased = syncbn.welford_mean_var(input)
RuntimeError: Dimension out of range (expected to be in range of [-2, 1], but got 2) (maybe_wrap_dim at /tmp/pip-req-build-p5q91txh/c10/core/WrapDimMinimal.h:20)

same error

diggerdu · 2019-09-29T07:50:51Z

@zhuyingSeu a workaround is using python-fallback version

tycallen · 2019-11-02T15:48:18Z

Same error, and when try to use python-fallback, get another error.
Model is fine when not use syncbn.

Traceback (most recent call last):
  File "dist_train.py", line 74, in <module>
    main()
  File "dist_train.py", line 70, in main
    num_query
  File "/root/gg/tyc/processor/processor.py", line 82, in do_train
    scores, feats = model(img, target)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 539, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/apex/parallel/distributed.py", line 560, in forward
    result = self.module(*inputs, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 539, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/apex/amp/_initialize.py", line 197, in new_fwd
    **applier(kwargs, input_caster))
  File "/root/gg/tyc/model/make_model.py", line 230, in forward
    out = self.model.features(x)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 539, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 539, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torchvision/models/densenet.py", line 74, in forward
    new_features = layer(*features)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 539, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torchvision/models/densenet.py", line 50, in forward
    bottleneck_output = bn_function(*prev_features)
  File "/usr/local/lib/python3.6/dist-packages/torchvision/models/densenet.py", line 23, in bn_function
    bottleneck_output = conv(relu(norm(concated_features)))
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 539, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py", line 343, in forward
    return self.conv2d_forward(input, self.weight)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py", line 340, in conv2d_forward
    self.padding, self.dilation, self.groups)
RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.cuda.HalfTensor) should be the same

jjsjann123 · 2019-11-04T23:26:35Z

Oops, somehow missed this thread, let me take a look at this.

jjsjann123 · 2019-11-06T00:15:22Z

It was broken by me earlier at #275 https://github.com/NVIDIA/apex/pull/275/files#diff-f22c2ebe466a49fc0a46a5e7fabb2004R74-R85 🤦‍♂️

Will push a fix shortly

jjsjann123 · 2019-11-06T22:54:10Z

PR issued #590

tycallen · 2019-11-07T06:49:11Z

PR issued #590

It works! Thank you.

jjsjann123 · 2019-11-07T09:22:12Z

Glad to help and thanks for reporting the issue!

jjsjann123 added a commit that referenced this issue Mar 15, 2019

[SyncBatchNorm]

61ff8d3

supporting 2 dimensional input, resolving issue #194 Implementation: for 2d input, switching channel_last flag to true for better memory access pattern in the kernel.

jjsjann123 mentioned this issue Mar 15, 2019

[SyncBatchNorm] #206

Merged

mcarilli pushed a commit that referenced this issue Mar 22, 2019

[SyncBatchNorm] (#206)

0a99154

supporting 2 dimensional input, resolving issue #194 Implementation: for 2d input, switching channel_last flag to true for better memory access pattern in the kernel.

mcarilli closed this as completed Mar 25, 2019

jjsjann123 mentioned this issue Nov 6, 2019

fixing batchnorm 1d input #590

Merged

muzi2045 mentioned this issue Jan 7, 2020

multi-GPU training error V2AI/Det3D#36

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SyncBatchNorm doesn't support 2 dimensions input? #194

SyncBatchNorm doesn't support 2 dimensions input? #194

flymark2010 commented Mar 11, 2019

mcarilli commented Mar 11, 2019

jjsjann123 commented Mar 11, 2019

flymark2010 commented Mar 12, 2019

jjsjann123 commented Mar 12, 2019

DTennant commented Mar 15, 2019

jjsjann123 commented Mar 15, 2019

jjsjann123 commented Mar 15, 2019

mcarilli commented Mar 25, 2019

hiyijian commented Mar 26, 2019

jjsjann123 commented Mar 26, 2019

zhuyingSeu commented Sep 24, 2019

diggerdu commented Sep 29, 2019

diggerdu commented Sep 29, 2019

tycallen commented Nov 2, 2019

jjsjann123 commented Nov 4, 2019

jjsjann123 commented Nov 6, 2019

jjsjann123 commented Nov 6, 2019

tycallen commented Nov 7, 2019

jjsjann123 commented Nov 7, 2019

SyncBatchNorm doesn't support 2 dimensions input? #194

SyncBatchNorm doesn't support 2 dimensions input? #194

Comments

flymark2010 commented Mar 11, 2019

mcarilli commented Mar 11, 2019

jjsjann123 commented Mar 11, 2019

flymark2010 commented Mar 12, 2019

jjsjann123 commented Mar 12, 2019

DTennant commented Mar 15, 2019

jjsjann123 commented Mar 15, 2019

jjsjann123 commented Mar 15, 2019

mcarilli commented Mar 25, 2019

hiyijian commented Mar 26, 2019

jjsjann123 commented Mar 26, 2019

zhuyingSeu commented Sep 24, 2019

diggerdu commented Sep 29, 2019

diggerdu commented Sep 29, 2019

tycallen commented Nov 2, 2019

jjsjann123 commented Nov 4, 2019

jjsjann123 commented Nov 6, 2019

jjsjann123 commented Nov 6, 2019

tycallen commented Nov 7, 2019

jjsjann123 commented Nov 7, 2019