ninja: build stopped: subcommand failed.

[mannamman]

COMMAND
python3 main.py

ENVS
ubuntu 18.04

NVIDIA-SMI 470.86 Driver Version: 470.86 CUDA Version: 11.4
GeForce RTX 3090

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Wed_Jul_22_19:09:09_PDT_2020
Cuda compilation tools, release 11.0, V11.0.221
Build cuda_11.0_bu.TC445_37.28845127_0

ERROR

Traceback (most recent call last):
File "/opt/conda/envs/barber/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1539, in _run_ninja_build
env=env)
File "/opt/conda/envs/barber/lib/python3.7/subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "main.py", line 13, in
from models.Embedding import Embedding
File "/home/user/jupyter-work/barber/models/Embedding.py", line 3, in
from models.Net import Net
File "/home/user/jupyter-work/barber/models/Net.py", line 3, in
from models.stylegan2.model import Generator
File "/home/user/jupyter-work/barber/models/stylegan2/model.py", line 11, in
from models.stylegan2.op import FusedLeakyReLU, fused_leaky_relu, upfirdn2d
File "/home/user/jupyter-work/barber/models/stylegan2/op/init.py", line 1, in
from .fused_act import FusedLeakyReLU, fused_leaky_relu
File "/home/user/jupyter-work/barber/models/stylegan2/op/fused_act.py", line 17, in
os.path.join(module_path, "fused_bias_act_kernel.cu"),
File "/opt/conda/envs/barber/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 997, in load
keep_intermediates=keep_intermediates)
File "/opt/conda/envs/barber/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1202, in jit_compile
with_cuda=with_cuda)
File "/opt/conda/envs/barber/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1300, in write_ninja_file_and_build_library
error_prefix="Error building extension '{}'".format(name))
File "/opt/conda/envs/barber/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1555, in run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error building extension 'fused': [1/3] /usr/local/cuda-11.0/bin/nvcc -DTORCH_EXTENSION_NAME=fused -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include -isystem /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/TH -isystem /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/THC -isystem /usr/local/cuda-11.0/include -isystem /opt/conda/envs/barber/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -std=c++14 -c /home/user/jupyter-work/barber/models/stylegan2/op/fused_bias_act_kernel.cu -o fused_bias_act_kernel.cuda.o
FAILED: fused_bias_act_kernel.cuda.o
/usr/local/cuda-11.0/bin/nvcc -DTORCH_EXTENSION_NAME=fused -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include -isystem /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/TH -isystem /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/THC -isystem /usr/local/cuda-11.0/include -isystem /opt/conda/envs/barber/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -std=c++14 -c /home/user/jupyter-work/barber/models/stylegan2/op/fused_bias_act_kernel.cu -o fused_bias_act_kernel.cuda.o
nvcc fatal : Unsupported gpu architecture 'compute_86'
[2/3] c++ -MMD -MF fused_bias_act.o.d -DTORCH_EXTENSION_NAME=fused -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -isystem /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include -isystem /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/TH -isystem /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/THC -isystem /usr/local/cuda-11.0/include -isystem /opt/conda/envs/barber/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /home/user/jupyter-work/barber/models/stylegan2/op/fused_bias_act.cpp -o fused_bias_act.o
In file included from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/c10/core/DeviceType.h:8:0,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/c10/core/Device.h:3,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/c10/core/Allocator.h:6,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/ATen/ATen.h:7,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/all.h:8,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/extension.h:4,
from /home/user/jupyter-work/barber/models/stylegan2/op/fused_bias_act.cpp:1:
/home/user/jupyter-work/barber/models/stylegan2/op/fused_bias_act.cpp: In function ‘at::Tensor fused_bias_act(const at::Tensor&, const at::Tensor&, const at::Tensor&, int, int, float, float)’:
/home/user/jupyter-work/barber/models/stylegan2/op/fused_bias_act.cpp:7:42: warning: ‘at::DeprecatedTypeProperties& at::Tensor::type() const’ is deprecated: Tensor.type() is deprecated. Instead use Tensor.options(), which in many cases (e.g. in a constructor) is a drop-in replacement. If you were using data from type(), that is now available from Tensor itself, so instead of tensor.type().scalar_type(), use tensor.scalar_type() instead and instead of tensor.type().backend() use tensor.device(). [-Wdeprecated-declarations]
#define CHECK_CUDA(x) TORCH_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
^
/home/user/jupyter-work/barber/models/stylegan2/op/fused_bias_act.cpp:13:5: note: in expansion of macro ‘CHECK_CUDA’
CHECK_CUDA(input);
^~~~~~~~~~
In file included from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/ATen/Tensor.h:3:0,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/ATen/Context.h:4,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/ATen/ATen.h:9,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/all.h:8,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/extension.h:4,
from /home/user/jupyter-work/barber/models/stylegan2/op/fused_bias_act.cpp:1:
/opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/ATen/core/TensorBody.h:277:30: note: declared here
DeprecatedTypeProperties & type() const {
^~~~
In file included from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/c10/core/DeviceType.h:8:0,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/c10/core/Device.h:3,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/c10/core/Allocator.h:6,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/ATen/ATen.h:7,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/all.h:8,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/extension.h:4,
from /home/user/jupyter-work/barber/models/stylegan2/op/fused_bias_act.cpp:1:
/home/user/jupyter-work/barber/models/stylegan2/op/fused_bias_act.cpp:7:42: warning: ‘at::DeprecatedTypeProperties& at::Tensor::type() const’ is deprecated: Tensor.type() is deprecated. Instead use Tensor.options(), which in many cases (e.g. in a constructor) is a drop-in replacement. If you were using data from type(), that is now available from Tensor itself, so instead of tensor.type().scalar_type(), use tensor.scalar_type() instead and instead of tensor.type().backend() use tensor.device(). [-Wdeprecated-declarations]
#define CHECK_CUDA(x) TORCH_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
^
/home/user/jupyter-work/barber/models/stylegan2/op/fused_bias_act.cpp:14:5: note: in expansion of macro ‘CHECK_CUDA’
CHECK_CUDA(bias);
^~~~~~~~~~
In file included from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/ATen/Tensor.h:3:0,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/ATen/Context.h:4,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/ATen/ATen.h:9,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/all.h:8,
from /opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/torch/extension.h:4,
from /home/user/jupyter-work/barber/models/stylegan2/op/fused_bias_act.cpp:1:
/opt/conda/envs/barber/lib/python3.7/site-packages/torch/include/ATen/core/TensorBody.h:277:30: note: declared here
DeprecatedTypeProperties & type() const {
^~~~
ninja: build stopped: subcommand failed.

[evgdobr]

I had faced with the same issue,
for me helped next:

export TORCH_CUDA_ARCH_LIST="7.5"

[ZPdesu]

It's related to your CUDA version. Please install the newest CUDA and Pytorch.

[mrsaleh]

Sometimes it is because " ninja --version " return something more than just a version number. in this case, download Ninja executable file and add it to the path. That happened to me on windows.