Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Weird Error while using multi GPU. #43

Open
AIprogrammer opened this issue Aug 9, 2019 · 0 comments
Open

Weird Error while using multi GPU. #43

AIprogrammer opened this issue Aug 9, 2019 · 0 comments

Comments

@AIprogrammer
Copy link

AIprogrammer commented Aug 9, 2019

RuntimeError: start (1431224) + length (0) exceeds dimension size (1431244). (narrow at /opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/ATen/native/TensorShape.cpp:157)
frame #0: at::Type::narrow(at::Tensor const&, long, long, long) const + 0x49 (0x7fe6365a1639 in /export/home/anaconda_install/anaconda_download/installed_conda/envs/pytorch_0_4_1/lib/python3.6/site-packages/torch/lib/libcaffe2.so)
frame #1: torch::autograd::VariableType::narrow(at::Tensor const&, long, long, long) const + 0x184 (0x7fe6382c3ae4 in /export/home/anaconda_install/anaconda_download/installed_conda/envs/pytorch_0_4_1/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #2: torch::cuda::broadcast_coalesced(at::ArrayRefat::Tensor, at::ArrayRef, unsigned long) + 0xbc0 (0x7fe6386b7210 in /export/home/anaconda_install/anaconda_download/installed_conda/envs/pytorch_0_4_1/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #3: + 0xc423cb (0x7fe6386bb3cb in /export/home/anaconda_install/anaconda_download/installed_conda/envs/pytorch_0_4_1/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #4: + 0x38a5cb (0x7fe637e035cb in /export/home/anaconda_install/anaconda_download/installed_conda/envs/pytorch_0_4_1/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)

frame #15: THPFunction_apply(_object*, _object*) + 0x38f (0x7fe6381e1a2f in /export/home/anaconda_install/anaconda_download/installed_conda/envs/pytorch_0_4_1/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #46: __libc_start_main + 0xf5 (0x7fe655cf2c05 in /usr/lib64/libc.so.6)
frame #47: python() [0x4009e9]


Hi, while adding the Att module in my own model, I encounter the error above. But it works well using single GPU.

Do anyone know why this happens? Thanks in advance.

@AIprogrammer AIprogrammer changed the title Wired Error while using multi GPU. Weird Error while using multi GPU. Aug 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant