Unable to build cuda extension with pytorch 1.5 #40663

lawpdas · 2020-06-27T13:48:16Z

Hi, I tried to build an extension correlation layer. This extension can be built with pytorch1.4. However, I cannot build it with pytorch1.5 and nightly.

The error message:

...
/usr/include/c++/7/bits/basic_string.tcc: In instantiation of ‘static std::basic_string<_CharT, _Traits, _Alloc>::_Rep* std::basic_string<_CharT, _Traits, _Alloc>::_Rep::_S_create(std::basic_string<_CharT, _Traits, _Alloc>::size_type, std::basic_string<_CharT, _Traits, _Alloc>::size_type, const _Alloc&) [with _CharT = char32_t; _Traits = std::char_traits<char32_t>; _Alloc = std::allocator<char32_t>; std::basic_string<_CharT, _Traits, _Alloc>::size_type = long unsigned int]’:
/usr/include/c++/7/bits/basic_string.tcc:578:28:   required from ‘static _CharT* std::basic_string<_CharT, _Traits, _Alloc>::_S_construct(_InIterator, _InIterator, const _Alloc&, std::forward_iterator_tag) [with _FwdIterator = const char32_t*; _CharT = char32_t; _Traits = std::char_traits<char32_t>; _Alloc = std::allocator<char32_t>]’
/usr/include/c++/7/bits/basic_string.h:5042:20:   required from ‘static _CharT* std::basic_string<_CharT, _Traits, _Alloc>::_S_construct_aux(_InIterator, _InIterator, const _Alloc&, std::__false_type) [with _InIterator = const char32_t*; _CharT = char32_t; _Traits = std::char_traits<char32_t>; _Alloc = std::allocator<char32_t>]’
/usr/include/c++/7/bits/basic_string.h:5063:24:   required from ‘static _CharT* std::basic_string<_CharT, _Traits, _Alloc>::_S_construct(_InIterator, _InIterator, const _Alloc&) [with _InIterator = const char32_t*; _CharT = char32_t; _Traits = std::char_traits<char32_t>; _Alloc = std::allocator<char32_t>]’
/usr/include/c++/7/bits/basic_string.tcc:656:134:   required from ‘std::basic_string<_CharT, _Traits, _Alloc>::basic_string(const _CharT*, std::basic_string<_CharT, _Traits, _Alloc>::size_type, const _Alloc&) [with _CharT = char32_t; _Traits = std::char_traits<char32_t>; _Alloc = std::allocator<char32_t>; std::basic_string<_CharT, _Traits, _Alloc>::size_type = long unsigned int]’
/usr/include/c++/7/bits/basic_string.h:6693:95:   required from here
/usr/include/c++/7/bits/basic_string.tcc:1067:16: error: cannot call member function ‘void std::basic_string<_CharT, _Traits, _Alloc>::_Rep::_M_set_sharable() [with _CharT = char32_t; _Traits = std::char_traits<char32_t>; _Alloc = std::allocator<char32_t>]’ without object
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "/home/space/Public/anaconda3/envs/nightly/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1423, in _run_ninja_build
    check=True)
  File "/home/space/Public/anaconda3/envs/nightly/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "setup.py", line 28, in <module>
    'build_ext': BuildExtension
  File "/home/space/Public/anaconda3/envs/nightly/lib/python3.7/site-packages/setuptools/__init__.py", line 161, in setup
    return distutils.core.setup(**attrs)
  File "/home/space/Public/anaconda3/envs/nightly/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/home/space/Public/anaconda3/envs/nightly/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/home/space/Public/anaconda3/envs/nightly/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/space/Public/anaconda3/envs/nightly/lib/python3.7/site-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "/home/space/Public/anaconda3/envs/nightly/lib/python3.7/site-packages/setuptools/command/install.py", line 109, in do_egg_install
    self.run_command('bdist_egg')
  File "/home/space/Public/anaconda3/envs/nightly/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/home/space/Public/anaconda3/envs/nightly/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/space/Public/anaconda3/envs/nightly/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 173, in run
    cmd = self.call_command('install_lib', warn_dir=0)
  File "/home/space/Public/anaconda3/envs/nightly/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 159, in call_command
    self.run_command(cmdname)
  File "/home/space/Public/anaconda3/envs/nightly/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/home/space/Public/anaconda3/envs/nightly/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/space/Public/anaconda3/envs/nightly/lib/python3.7/site-packages/setuptools/command/install_lib.py", line 11, in run
    self.build()
  File "/home/space/Public/anaconda3/envs/nightly/lib/python3.7/distutils/command/install_lib.py", line 107, in build
    self.run_command('build_ext')
  File "/home/space/Public/anaconda3/envs/nightly/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/home/space/Public/anaconda3/envs/nightly/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/space/Public/anaconda3/envs/nightly/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 87, in run
    _build_ext.run(self)
  File "/home/space/Public/anaconda3/envs/nightly/lib/python3.7/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/home/space/Public/anaconda3/envs/nightly/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 603, in build_extensions
    build_ext.build_extensions(self)
  File "/home/space/Public/anaconda3/envs/nightly/lib/python3.7/distutils/command/build_ext.py", line 449, in build_extensions
    self._build_extensions_serial()
  File "/home/space/Public/anaconda3/envs/nightly/lib/python3.7/distutils/command/build_ext.py", line 474, in _build_extensions_serial
    self.build_extension(ext)
  File "/home/space/Public/anaconda3/envs/nightly/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 208, in build_extension
    _build_ext.build_extension(self, ext)
  File "/home/space/Public/anaconda3/envs/nightly/lib/python3.7/distutils/command/build_ext.py", line 534, in build_extension
    depends=ext.depends)
  File "/home/space/Public/anaconda3/envs/nightly/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 437, in unix_wrap_ninja_compile
    with_cuda=with_cuda)
  File "/home/space/Public/anaconda3/envs/nightly/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1163, in _write_ninja_file_and_compile_objects
    error_prefix='Error compiling objects for extension')
  File "/home/space/Public/anaconda3/envs/nightly/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1436, in _run_ninja_build
    raise RuntimeError(message)
RuntimeError: Error compiling objects for extension

cc @malfet @yf225 @glaringlee

The text was updated successfully, but these errors were encountered:

522730312 · 2020-06-28T07:01:17Z

Did you set -std=c++14 in setup.py?

lawpdas · 2020-06-28T08:58:37Z

I have changed it from c++11 to c++14. It does not work.

glaringlee · 2020-06-29T22:13:46Z

@lawpdas
Do you have ninja installed?

lawpdas · 2020-06-30T07:50:32Z

@glaringlee Yes, ninja 1.9.0 has been installed in my conda environment.

nnuk · 2020-07-01T13:08:40Z

Stuck with same error kindly help. PyTorch 1.5.

glaringlee · 2020-07-01T16:44:52Z

@lawpdas @nnuk
I tried with the latest pytorch code in master branch with ubuntu16.02+gcc7+cuda9.2+cudnn7, it works fine for me.
Can you guys try pytorch 1.5.1 or just clone the code from master?
We do have some libs issues in pytorch 1.5 though I am not sure whether that is related.

nnuk · 2020-07-01T17:12:49Z

@glaringlee Actually I have PyTorch 1.5.1 and I am working with VS Code , Windows 10.

glaringlee · 2020-07-01T17:44:17Z

Can you guys post your cuda version, complier and version, and related env parameters?

@nnuk can you try latest master brunch? and make sure you compiled your code with c++14 supported

There is a similar error which is solved by upgrading cuda version.
pytorch/vision#1893

nnuk · 2020-07-01T23:27:49Z

Cuda -> 10.1
cudnn -> 7.0
PyTorch -> 1.5.1
Visual Studio Code -> 1.46.1
Python 3.7
OS Windows 10.

glaringlee · 2020-07-02T02:26:55Z

@nnuk can you try pytorch in master brunch?
@peterjc123 any thoughts on this? I saw a similar issue pytorch/vision#1893 and solved by upgrade cuda 10.1 to newer version.

lawpdas · 2020-07-06T09:30:03Z

@glaringlee I built it with pytorch1.5.1 and nightly. Both them are installed by `conda install'.
cuda10.1(also try 10.2)
python3.7
ubuntu18.04
gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0

I tried to clone the latest pytorch from master, but I got another error when I built it : (

I also found other errors

...
/home/space/Public/anaconda3/envs/pynightly/lib/python3.7/site-packages/torch/include/ATen/core/boxing/KernelFunction_impl.h: In instantiation of ‘Return c10::KernelFunction::callUnboxed(const c10::OperatorHandle&, Args ...) const [with Return = std::tuple<at::Tensor, at::Tensor, at::Tensor>; Args = {const at::Tensor&, const at::Tensor&, const at::Tensor&, c10::ArrayRef<long int>, c10::ArrayRef<long int>, c10::ArrayRef<long int>, c10::ArrayRef<long int>, std::array<bool, 3>}]’:
/home/space/Public/anaconda3/envs/pynightly/lib/python3.7/site-packages/torch/include/ATen/core/dispatch/Dispatcher.h:221:86:   required from ‘Return c10::Dispatcher::callUnboxedWithDispatchKey(const c10::OperatorHandle&, c10::DispatchKey, Args ...) const [with Return = std::tuple<at::Tensor, at::Tensor, at::Tensor>; Args = {const at::Tensor&, const at::Tensor&, const at::Tensor&, c10::ArrayRef<long int>, c10::ArrayRef<long int>, c10::ArrayRef<long int>, c10::ArrayRef<long int>, std::array<bool, 3>}]’
/home/space/Public/anaconda3/envs/pynightly/lib/python3.7/site-packages/torch/include/ATen/core/dispatch/Dispatcher.h:229:53:   required from ‘Return c10::Dispatcher::callUnboxed(const c10::OperatorHandle&, Args ...) const [with Return = std::tuple<at::Tensor, at::Tensor, at::Tensor>; Args = {const at::Tensor&, const at::Tensor&, const at::Tensor&, c10::ArrayRef<long int>, c10::ArrayRef<long int>, c10::ArrayRef<long int>, c10::ArrayRef<long int>, std::array<bool, 3>}]’
/home/space/Public/anaconda3/envs/pynightly/lib/python3.7/site-packages/torch/include/ATen/core/dispatch/Dispatcher.h:192:104:   required from ‘Return c10::OperatorHandle::callUnboxed(Args ...) const [with Return = std::tuple<at::Tensor, at::Tensor, at::Tensor>; Args = {const at::Tensor&, const at::Tensor&, const at::Tensor&, c10::ArrayRef<long int>, c10::ArrayRef<long int>, c10::ArrayRef<long int>, c10::ArrayRef<long int>, std::array<bool, 3>}]’
/home/space/Public/anaconda3/envs/pynightly/lib/python3.7/site-packages/torch/include/ATen/Functions.h:16662:259:   required from here
/home/space/Public/anaconda3/envs/pynightly/lib/python3.7/site-packages/torch/include/ATen/core/boxing/KernelFunction_impl.h:70:54: error: no matching function for call to ‘boxAndCallBoxedFunc<std::tuple<at::Tensor, at::Tensor, at::Tensor>, const at::Tensor&, const at::Tensor&, const at::Tensor&, c10::ArrayRef<long int>, c10::ArrayRef<long int>, c10::ArrayRef<long int>, c10::ArrayRef<long int>, std::array<bool, 3> >(void (* const&)(c10::OperatorKernel*, const c10::OperatorHandle&, c10::Stack*), c10::OperatorKernel*, const c10::OperatorHandle&, const at::Tensor&, const at::Tensor&, const at::Tensor&, c10::ArrayRef<long int>, c10::ArrayRef<long int>, c10::ArrayRef<long int>, c10::ArrayRef<long int>, std::array<bool, 3>)’

...
/home/space/Public/anaconda3/envs/pynightly/lib/python3.7/site-packages/torch/include/c10/util/Optional.h: In instantiation of ‘c10::optional<T>& c10::optional<T>::operator=(c10::optional<T>&&) [with T = torch::ExpandingArray<3, double>]’:
/home/space/Public/anaconda3/envs/pynightly/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/nn/options/pooling.h:428:8:   required from ‘void torch::nn::Cloneable<Derived>::clone_(torch::nn::Module&, const c10::optional<c10::Device>&) [with Derived = torch::nn::FractionalMaxPool3dImpl]’
/home/space/Public/anaconda3/envs/pynightly/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/optim/sgd.h:48:17:   required from here
/home/space/Public/anaconda3/envs/pynightly/lib/python3.7/site-packages/torch/include/c10/util/Optional.h:396:23: error: passing ‘const torch::ExpandingArray<3, double>’ as ‘this’ argument discards qualifiers [-fpermissive]

...

/home/space/Public/anaconda3/envs/pynightly/lib/python3.7/site-packages/torch/include/c10/util/Optional.h: In instantiation of ‘c10::optional<T>& c10::optional<T>::operator=(c10::optional<T>&&) [with T = torch::ExpandingArray<2, double>]’:
/home/space/Public/anaconda3/envs/pynightly/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/nn/options/pooling.h:428:8:   required from ‘void torch::nn::Cloneable<Derived>::clone_(torch::nn::Module&, const c10::optional<c10::Device>&) [with Derived = torch::nn::FractionalMaxPool2dImpl]’
/home/space/Public/anaconda3/envs/pynightly/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/optim/sgd.h:48:17:   required from here
/home/space/Public/anaconda3/envs/pynightly/lib/python3.7/site-packages/torch/include/c10/util/Optional.h:396:23: error: passing ‘const torch::ExpandingArray<2, double>’ as ‘this’ argument discards qualifiers [-fpermissive]
       contained_val() = std::move(*rhs);
       ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~
In file included from /home/space/Public/anaconda3/envs/pynightly/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/nn/options/conv.h:6:0,
                 from /home/space/Public/anaconda3/envs/pynightly/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/nn/functional/conv.h:3,
                 from /home/space/Public/anaconda3/envs/pynightly/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/nn/functional.h:4,
                 from /home/space/Public/anaconda3/envs/pynightly/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/nn.h:4,
                 from /home/space/Public/anaconda3/envs/pynightly/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/all.h:7,
                 from /home/space/Public/anaconda3/envs/pynightly/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/torch.h:3,
                 from /home/space/Documents/correlation_package/correlation_cuda.cc:1:
/home/space/Public/anaconda3/envs/pynightly/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/expanding_array.h:23:7: note:   in call to ‘torch::ExpandingArray<2, double>& torch::ExpandingArray<2, double>::operator=(const torch::ExpandingArray<2, double>&)’
 class ExpandingArray {
       ^~~~~~~~~~~~~~

malfet · 2020-07-06T13:56:22Z

@lawpdas can you please share a full build log (including output from configure)?

lawpdas · 2020-07-07T10:14:05Z

I cannot build pytorch source code with cuda10.2 #40909, but now I can build pytorch with cuda10.0.
By using master version and setting -std=c++14 in setup.py, this extension has been built successfully.
Thanks!

mrshenli added module: build Build system issues triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module labels Jun 29, 2020

glaringlee added the module: cpp-extensions Related to torch.utils.cpp_extension label Jul 1, 2020

lawpdas closed this as completed Jul 7, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to build cuda extension with pytorch 1.5 #40663

Unable to build cuda extension with pytorch 1.5 #40663

lawpdas commented Jun 27, 2020 •

edited by pytorch-probot bot

Loading

522730312 commented Jun 28, 2020

lawpdas commented Jun 28, 2020

glaringlee commented Jun 29, 2020

lawpdas commented Jun 30, 2020

nnuk commented Jul 1, 2020

glaringlee commented Jul 1, 2020

nnuk commented Jul 1, 2020

glaringlee commented Jul 1, 2020 •

edited

Loading

nnuk commented Jul 1, 2020

glaringlee commented Jul 2, 2020

lawpdas commented Jul 6, 2020 •

edited

Loading

malfet commented Jul 6, 2020

lawpdas commented Jul 7, 2020

Unable to build cuda extension with pytorch 1.5 #40663

Unable to build cuda extension with pytorch 1.5 #40663

Comments

lawpdas commented Jun 27, 2020 • edited by pytorch-probot bot Loading

522730312 commented Jun 28, 2020

lawpdas commented Jun 28, 2020

glaringlee commented Jun 29, 2020

lawpdas commented Jun 30, 2020

nnuk commented Jul 1, 2020

glaringlee commented Jul 1, 2020

nnuk commented Jul 1, 2020

glaringlee commented Jul 1, 2020 • edited Loading

nnuk commented Jul 1, 2020

glaringlee commented Jul 2, 2020

lawpdas commented Jul 6, 2020 • edited Loading

malfet commented Jul 6, 2020

lawpdas commented Jul 7, 2020

lawpdas commented Jun 27, 2020 •

edited by pytorch-probot bot

Loading

glaringlee commented Jul 1, 2020 •

edited

Loading

lawpdas commented Jul 6, 2020 •

edited

Loading