Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NVCC error compiling fused_bias_act.cpp #72

Closed
arruw opened this issue Jan 13, 2021 · 5 comments
Closed

NVCC error compiling fused_bias_act.cpp #72

arruw opened this issue Jan 13, 2021 · 5 comments

Comments

@arruw
Copy link

arruw commented Jan 13, 2021

I get an error when running python run_generator.py generate-images ....

I solved some of the issues following this https://stackoverflow.com/questions/59342888/tensorflow-error-this-file-requires-compiler-and-library-support-for-the-iso-c#.

Error:

dnnlib: Running run_generator.generate_images() on localhost...
Loading networks from "./networks/stylegan2-ffhq-config-f.pkl"...
Setting up TensorFlow plugin "fused_bias_act.cu": Preprocessing... Compiling... Failed!
Traceback (most recent call last):
  File "run_generator.py", line 168, in <module>
    main()
  File "run_generator.py", line 163, in main
    dnnlib.submit_run(sc, func_name_map[subcmd], **kwargs)
  File "/home/matjazibb/dev/stylegan2/dnnlib/submission/submit.py", line 343, in submit_run
    return farm.submit(submit_config, host_run_dir)
  File "/home/matjazibb/dev/stylegan2/dnnlib/submission/internal/local.py", line 22, in submit
    return run_wrapper(submit_config)
  File "/home/matjazibb/dev/stylegan2/dnnlib/submission/submit.py", line 280, in run_wrapper
    run_func_obj(**submit_config.run_func_kwargs)
  File "/home/matjazibb/dev/stylegan2/run_generator.py", line 21, in generate_images
    _G, _D, Gs = pretrained_networks.load_networks(network_pkl)
  File "/home/matjazibb/dev/stylegan2/pretrained_networks.py", line 76, in load_networks
    G, D, Gs = pickle.load(stream, encoding='latin1')
  File "/home/matjazibb/dev/stylegan2/dnnlib/tflib/network.py", line 297, in __setstate__
    self._init_graph()
  File "/home/matjazibb/dev/stylegan2/dnnlib/tflib/network.py", line 154, in _init_graph
    out_expr = self._build_func(*self.input_templates, **build_kwargs)
  File "<string>", line 491, in G_synthesis_stylegan2
  File "<string>", line 455, in layer
  File "<string>", line 99, in modulated_conv2d_layer
  File "<string>", line 68, in apply_bias_act
  File "/home/matjazibb/dev/stylegan2/dnnlib/tflib/ops/fused_bias_act.py", line 68, in fused_bias_act
    return impl_dict[impl](x=x, b=b, axis=axis, act=act, alpha=alpha, gain=gain)
  File "/home/matjazibb/dev/stylegan2/dnnlib/tflib/ops/fused_bias_act.py", line 122, in _fused_bias_act_cuda
    cuda_kernel = _get_plugin().fused_bias_act
  File "/home/matjazibb/dev/stylegan2/dnnlib/tflib/ops/fused_bias_act.py", line 16, in _get_plugin
    return custom_ops.get_plugin(os.path.splitext(__file__)[0] + '.cu')
  File "/home/matjazibb/dev/stylegan2/dnnlib/tflib/custom_ops.py", line 147, in get_plugin
    _run_cmd(nvcc_cmd + ' "%s" --shared -o "%s" --keep --keep-dir "%s"' % (cuda_file, tmp_file, tmp_dir))
  File "/home/matjazibb/dev/stylegan2/dnnlib/tflib/custom_ops.py", line 61, in _run_cmd
    raise RuntimeError('NVCC returned an error. See below for full command line and output log:\n\n%s\n\n%s' % (cmd, output))
RuntimeError: NVCC returned an error. See below for full command line and output log:

nvcc --std=c++11 -DNDEBUG "/home/matjazibb/miniconda3/envs/stylegan2-try2/lib/python3.6/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so" --compiler-options '-fPIC -D_GLIBCXX_USE_CXX11_ABI=1' --gpu-architecture=sm_52 --use_fast_math --disable-warnings --include-path "/home/matjazibb/miniconda3/envs/stylegan2-try2/lib/python3.6/site-packages/tensorflow/include" --include-path "/home/matjazibb/miniconda3/envs/stylegan2-try2/lib/python3.6/site-packages/tensorflow/include/external/protobuf_archive/src" --include-path "/home/matjazibb/miniconda3/envs/stylegan2-try2/lib/python3.6/site-packages/tensorflow/include/external/com_google_absl" --include-path "/home/matjazibb/miniconda3/envs/stylegan2-try2/lib/python3.6/site-packages/tensorflow/include/external/eigen_archive" 2>&1 "/home/matjazibb/dev/stylegan2/dnnlib/tflib/ops/fused_bias_act.cu" --shared -o "/tmp/tmpn331x8yd/fused_bias_act_tmp.so" --keep --keep-dir "/tmp/tmpn331x8yd"

/usr/lib/gcc/x86_64-linux-gnu/5/include/mwaitxintrin.h(36): error: identifier "__builtin_ia32_monitorx" is undefined

/usr/lib/gcc/x86_64-linux-gnu/5/include/mwaitxintrin.h(42): error: identifier "__builtin_ia32_mwaitx" is undefined

/home/matjazibb/miniconda3/envs/stylegan2-try2/lib/python3.6/site-packages/tensorflow/include/absl/strings/str_cat.h(268): error: expression must have a constant value

/home/matjazibb/miniconda3/envs/stylegan2-try2/lib/python3.6/site-packages/tensorflow/include/absl/strings/str_cat.h(268): error: expression must have a constant value

/home/matjazibb/miniconda3/envs/stylegan2-try2/lib/python3.6/site-packages/tensorflow/include/absl/memory/memory.h(616): error: class "std::allocator<tensorflow::OpKernelContext::WrappedAllocator>" has no member "is_nothrow"
          detected during:
            instantiation of type "absl::memory_internal::GetIsNothrow<std::allocator<tensorflow::OpKernelContext::WrappedAllocator>>" 
(264): here
            instantiation of type "absl::memory_internal::ExtractOrT<absl::memory_internal::GetIsNothrow, std::allocator<tensorflow::OpKernelContext::WrappedAllocator>, std::false_type>" 
(642): here
            instantiation of class "absl::allocator_is_nothrow<Alloc> [with Alloc=std::allocator<tensorflow::OpKernelContext::WrappedAllocator>]" 
/home/matjazibb/miniconda3/envs/stylegan2-try2/lib/python3.6/site-packages/tensorflow/include/absl/container/inlined_vector.h(190): here
            instantiation of "absl::InlinedVector<T, N, A>::InlinedVector(absl::InlinedVector<T, N, A> &&) [with T=tensorflow::OpKernelContext::WrappedAllocator, N=4UL, A=std::allocator<tensorflow::OpKernelContext::WrappedAllocator>]" 
/home/matjazibb/miniconda3/envs/stylegan2-try2/lib/python3.6/site-packages/tensorflow/include/tensorflow/core/framework/op_kernel.h(1081): here

5 errors detected in the compilation of "/tmp/tmpn331x8yd/fused_bias_act.cpp1.ii".

OS info:

Distributor ID: Ubuntu
Description:    Ubuntu 16.04.6 LTS
Release:        16.04
Codename:       xenial

GPU info:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 455.45.01    Driver Version: 455.45.01    CUDA Version: 11.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 980 Ti  Off  | 00000000:01:00.0 Off |                  N/A |
| 20%   40C    P0    55W / 260W |      0MiB /  6082MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

nvcc:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2015 NVIDIA Corporation
Built on Tue_Aug_11_14:27:32_CDT_2015
Cuda compilation tools, release 7.5, V7.5.17

g++:

g++ --version
g++ (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Conda env:

_libgcc_mutex             0.1                        main  
_tflow_select             2.1.0                       gpu  
absl-py                   0.11.0             pyhd3eb1b0_1  
astor                     0.8.1                    py36_0  
blas                      1.0                         mkl  
brotlipy                  0.7.0           py36h27cfd23_1003  
c-ares                    1.17.1               h27cfd23_0  
ca-certificates           2020.12.8            h06a4308_0  
certifi                   2020.12.5        py36h06a4308_0  
cffi                      1.14.4           py36h261ae71_0  
chardet                   4.0.0           py36h06a4308_1003  
cryptography              3.3.1            py36h3c74f83_0  
cudatoolkit               10.1.243             h6bb024c_0  
cudnn                     7.6.5                cuda10.1_0  
cupti                     10.1.168                      0  
freetype                  2.10.4               h5ab3b9f_0  
gast                      0.4.0                      py_0  
google-pasta              0.2.0                      py_0  
grpcio                    1.31.0           py36hf8bcb03_0  
h5py                      2.10.0           py36hd6299e0_1  
hdf5                      1.10.6               hb1b8bf9_0  
idna                      2.10                       py_0  
importlib-metadata        2.0.0                      py_1  
intel-openmp              2020.2                      254  
jpeg                      9b                   h024ee3a_2  
keras-applications        1.0.8                      py_1  
keras-preprocessing       1.1.0                      py_1  
lcms2                     2.11                 h396b838_0  
ld_impl_linux-64          2.33.1               h53a641e_7  
libedit                   3.1.20191231         h14c3975_1  
libffi                    3.3                  he6710b0_2  
libgcc-ng                 9.1.0                hdf63c60_0  
libgfortran-ng            7.3.0                hdf63c60_0  
libpng                    1.6.37               hbc83047_0  
libprotobuf               3.13.0.1             hd408876_0  
libstdcxx-ng              9.1.0                hdf63c60_0  
libtiff                   4.1.0                h2733197_1  
lz4-c                     1.9.2                heb0550a_3  
markdown                  3.3.3            py36h06a4308_0  
mkl                       2020.2                      256  
mkl-service               2.3.0            py36he8ac12f_0  
mkl_fft                   1.2.0            py36h23d657b_0  
mkl_random                1.1.1            py36h0573a6f_0  
ncurses                   6.2                  he6710b0_1  
numpy                     1.19.2           py36h54aff64_0  
numpy-base                1.19.2           py36hfa32c7d_0  
olefile                   0.46                     py36_0  
openssl                   1.1.1i               h27cfd23_0  
pillow                    8.1.0            py36he98fc37_0  
pip                       20.3.3           py36h06a4308_0  
protobuf                  3.13.0.1         py36he6710b0_1  
pycparser                 2.20                       py_2  
pyopenssl                 20.0.1             pyhd3eb1b0_1  
pysocks                   1.7.1            py36h06a4308_0  
python                    3.6.12               hcff3b4d_2  
readline                  8.0                  h7b6447c_0  
requests                  2.25.1             pyhd3eb1b0_0  
scipy                     1.5.2            py36h0b6359f_0  
setuptools                51.1.2           py36h06a4308_4  
six                       1.15.0           py36h06a4308_0  
sqlite                    3.33.0               h62c20be_0  
tensorboard               1.14.0           py36hf484d3e_0  
tensorflow                1.14.0          gpu_py36h3fb9ad6_0  
tensorflow-base           1.14.0          gpu_py36he45bfe2_0  
tensorflow-estimator      1.14.0                     py_0  
tensorflow-gpu            1.14.0               h0d30ee6_0  
termcolor                 1.1.0                    py36_1  
tk                        8.6.10               hbc83047_0  
urllib3                   1.26.2             pyhd3eb1b0_0  
werkzeug                  1.0.1                      py_0  
wheel                     0.36.2             pyhd3eb1b0_0  
wrapt                     1.12.1           py36h7b6447c_1  
xz                        5.2.5                h7b6447c_0  
zipp                      3.4.0              pyhd3eb1b0_0  
zlib                      1.2.11               h7b6447c_3  
zstd                      1.4.5                h9ceee32_0
@nurpax
Copy link
Contributor

nurpax commented Jan 13, 2021

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2015 NVIDIA Corporation
Built on Tue_Aug_11_14:27:32_CDT_2015
Cuda compilation tools, release 7.5, V7.5.17

Looks like it's picking up CUDA 7.5 version of NVCC. You'd need nvcc to match with whatever CUDA your running with, most likely CUDA 10.0.

I'd recommend using the default Docker container template that's provided in the repo.

@arruw
Copy link
Author

arruw commented Jan 14, 2021

Thanks, tried with docker and all went smooth :)

@nurpax nurpax closed this as completed Jan 14, 2021
@xvdp
Copy link

xvdp commented May 24, 2021

This may be a bit ranty, but are there pre-compiled versions?

Compilation used to work for me I had to update cuda on my TitanRTX from 10 .1 to 11 and the code no longer works. Driver version 450.119.03

So I made a docker with
FROM nvidia/cudagl:10.0-devel-ubuntu18.04
But that doesn't work either.

As awesome are your graphics cards and the idea behind StyleGAN are, the code is a bit weak if it requires local .cu compilation.

@nurpax
Copy link
Contributor

nurpax commented May 25, 2021

@xvdp I understand your ranty position, we're also unhappy that it's difficult to build custom extensions to reliably.

I'd recommend completely switching to https://github.com/NVlabs/stylegan2-ada-pytorch which is our official PyTorch StyleGAN2 ADA remake. While it also suffers from tricky configuration issues with building custom extensions, you can use much newer software and containers with it as it works on the latest PyTorch versions.

I have done some investigation around making prebuilt custom ops for PyTorch but at least in my experience, there's no clean reliable ABI when building custom ops against PyTorch and CUDA. Thus the prebuilts often break right off the bat if you use them on even a slightly different version of PyTorch than what the extension was built against. I welcome any prior art and examples of successful custom extension binary releases that reliably work across multiple versions of PyTorch and CUDA.

@xvdp
Copy link

xvdp commented May 28, 2021

@nurpax Thank you! - I did,

I also simplified the projection code so its only one or two calls replacing dlib with mediapipe.
https://github.com/xvdp/stylegan2-ada-pytorch/blob/main/x_project_w_mediapipe_collab.ipynb
Mediapipe is not perfect either but its much simpler than dlib.

Of course that does not answer the compilation issue, but pytorch is more forward compatible than tensorflow. Anycase, it works both on my TitanRTX with Cuda 11 and on the collab.

I cant answer as to compilation details, but since StyleGAN has been such an influential project, its functionality deserves to be maintained, cant its sources be maintained in the pytorch repositories?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants