cupy.cuda.compiler.CompileException: nvrtc: error: failed to load builtins

[akjagadish]

When you run the demo:
python -m cmr.demo --name bird_net --num_train_epoch 500 --img_path cmr/demo_data/img1.jpg

I am getting this error:

/home/ubuntu/akshay/cmr/venv_cmr/local/lib/python2.7/site-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
Setting up model..
loading /home/ubuntu/akshay/cmr/nnutils/../cachedir/snapshots/bird_net/pred_net_500.pth..
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/ubuntu/akshay/cmr/demo.py", line 119, in
app.run(main)
File "/home/ubuntu/akshay/cmr/venv_cmr/local/lib/python2.7/site-packages/absl/app.py", line 274, in run
_run_main(main, argv)
File "/home/ubuntu/akshay/cmr/venv_cmr/local/lib/python2.7/site-packages/absl/app.py", line 238, in _run_main
sys.exit(main(argv))
File "/home/ubuntu/akshay/cmr/demo.py", line 108, in main
outputs = predictor.predict(batch)
File "cmr/nnutils/predictor.py", line 110, in predict
self.forward()
File "cmr/nnutils/predictor.py", line 149, in forward
self.cam_pred)
File "cmr/nnutils/nmr.py", line 183, in forward
return Render(self.renderer)(verts, faces)
File "cmr/nnutils/nmr.py", line 114, in forward
masks = self.renderer.forward_mask(vs, fs)
File "cmr/nnutils/nmr.py", line 50, in forward_mask
self.masks = self.renderer.render_silhouettes(self.vertices, self.faces)
File "build/bdist.linux-x86_64/egg/neural_renderer/renderer.py", line 38, in render_silhouettes
File "/home/ubuntu/akshay/cmr/venv_cmr/local/lib/python2.7/site-packages/chainer/functions/array/concat.py", line 90, in concat
y, = Concat(axis).apply(xs)
File "/home/ubuntu/akshay/cmr/venv_cmr/local/lib/python2.7/site-packages/chainer/function_node.py", line 245, in apply
outputs = self.forward(in_data)
File "/home/ubuntu/akshay/cmr/venv_cmr/local/lib/python2.7/site-packages/chainer/functions/array/concat.py", line 44, in forward
return xp.concatenate(xs, self.axis),
File "/home/ubuntu/akshay/cmr/venv_cmr/local/lib/python2.7/site-packages/cupy/manipulation/join.py", line 49, in concatenate
return core.concatenate_method(tup, axis)
File "cupy/core/core.pyx", line 2439, in cupy.core.core.concatenate_method
File "cupy/core/core.pyx", line 2482, in cupy.core.core.concatenate_method
File "cupy/core/core.pyx", line 2533, in cupy.core.core.concatenate
File "cupy/core/core.pyx", line 1630, in cupy.core.core.ndarray.setitem
File "cupy/core/core.pyx", line 3101, in cupy.core.core._scatter_op
File "cupy/core/elementwise.pxi", line 823, in cupy.core.core.ufunc.call
File "cupy/util.pyx", line 39, in cupy.util.memoize.decorator.ret
File "cupy/core/elementwise.pxi", line 622, in cupy.core.core._get_ufunc_kernel
File "cupy/core/elementwise.pxi", line 33, in cupy.core.core._get_simple_elementwise_kernel
File "cupy/core/carray.pxi", line 170, in cupy.core.core.compile_with_cache
File "/home/ubuntu/akshay/cmr/venv_cmr/local/lib/python2.7/site-packages/cupy/cuda/compiler.py", line 123, in compile_with_cache
base = _preprocess('', options, arch)
File "/home/ubuntu/akshay/cmr/venv_cmr/local/lib/python2.7/site-packages/cupy/cuda/compiler.py", line 86, in _preprocess
result = prog.compile(options)
File "/home/ubuntu/akshay/cmr/venv_cmr/local/lib/python2.7/site-packages/cupy/cuda/compiler.py", line 233, in compile
raise CompileException(log, self.src, self.name, options)
cupy.cuda.compiler.CompileException: nvrtc: error: failed to load builtins

I tried the solution mentioned in this: https://groups.google.com/forum/#!topic/chainer-jp/GKNe5KY_fm0 yet it is not working

[akjagadish]

The method suggested in https://groups.google.com/forum/#!topic/chainer-jp/GKNe5KY_fm0 worked. Had multiple versions of CUDA installed, need to figure out the right version.

[viralbthakar]

Hi @akshaykjagadish I am facing the same problem. Can you suggest the exact steps you followed to solve the problem?
Thank you

[akjagadish]

Hi, it has been some time since I solved this issue but I figure you just need to find out the right CUDA version (I had multiple versions on my AWS instance) and set path accordingly.

[PenroseWang]

Hi @akshaykjagadish , I am also facing the same problem. Do you still remember which CUDA version are you using? I currently only have CUDA 9.0 installed on my Linux, but still have the CompileException.

And hi @viralbthakar , did you solve the problem later?

Thank you all.

[charliememory]

I use the new version of neural_render git clone git@github.com:charliememory/neural_renderer.git and install the pre-built cupy with corresponding cuda version number, e.g. pip install cupy-cuda92. Then, I can avoid this issue.

[nikhilaravi]

@charliememory did you use chainer 3.3.0 or the latest version? With chainer==3.3.0 I keep getting an error:
RuntimeError: CUDA environment is not correctly set up (see https://github.com/chainer/chainer#installation).cannot import name sqrt_fixed
I checked chainer.backends.cuda.available and it returned False. With the new version of chainer, I get the nvrtc error above even though I have set the CUDA_PATH and LD_LIBRARY_PATH as instructed in the cupy documentation.

[BornInWater]

@charliememory I used neural renderer from the repo that you mentioned and have cuda 9 and hence used the same command but while installing neural renderer I get the below error:

Traceback (most recent call last):
File "setup.py", line 3, in
import neural_renderer
File "/home/nsajjan/WORK/Capstone/cmr/external/neural_renderer/neural_renderer/init.py", line 1, in
from cross import cross
File "/home/nsajjan/WORK/Capstone/cmr/external/neural_renderer/neural_renderer/cross.py", line 2, in
import cupy as cp
File "/home/nsajjan/WORK/Capstone/cmr/venv_cmr/lib/python2.7/site-packages/cupy/init.py", line 7, in
from cupy import _version
ImportError: cannot import name _version

Any idea why I am getting this?

[charliememory]

@nikhilaravi I did not have such issue. Maybe you can check whether the CUDA is accessible for other packages first to make sure the problem comes from chainer.

[charliememory]

@BornInWater In my case, I modified the setup.py file. I used cupy-cuda92, maybe you can try cupy-cuda9 instead.

[BornInWater]

@charliememory Tried that. Not working. Same error.

[ZilinZhou1995]

@charliememory did you use chainer 3.3.0 or the latest version? With chainer==3.3.0 I keep getting an error:
RuntimeError: CUDA environment is not correctly set up (see https://github.com/chainer/chainer#installation).cannot import name sqrt_fixed
I checked chainer.backends.cuda.available and it returned False. With the new version of chainer, I get the nvrtc error above even though I have set the CUDA_PATH and LD_LIBRARY_PATH as instructed in the cupy documentation.

Have you slove this problem？

[BornInWater]

@ZilinZhou1995 , No, I could not solve it.

[BornInWater]

@nikhilaravi , @ZilinZhou1995 Its working for me now.

I have CUDA 9. I first installed chainer 5.2.0 version and cupy 5.2.0[pip install cupy-cuda90] version separately. Then installed neural mesh renderer and Perceptual similarity in the external directory. But in the original script, branch v1.1.0 of neural mesh renderer gets installed. This was not working for me, so I installed the most recent one. That works.
After that, in the requirements.txt file, I removed chainer and cupy and just did pip install -r requirements.txt for the rest.
Also, I had to switch to Torch 1.0 version as I think torch 0.3 does not support CUDA 9.0.
Hope this helps!

[BornInWater]

@charliememory You are installing the newest version of NMR right? Were you able to train the model properly with the newest version of NMR? For me one of the losses was much higher than the cmr with the prescribed nmr. You can see the conversation here

[QHan1st]

@charliememory You are installing the newest version of NMR right? Were you able to train the model properly with the newest version of NMR? For me one of the losses was much higher than the cmr with the prescribed nmr. You can see the conversation here

Have you run the project successfully?

[EAST-J]

@nikhilaravi , @ZilinZhou1995 Its working for me now.

I have CUDA 9. I first installed chainer 5.2.0 version and cupy 5.2.0[pip install cupy-cuda90] version separately. Then installed neural mesh renderer and Perceptual similarity in the external directory. But in the original script, branch v1.1.0 of neural mesh renderer gets installed. This was not working for me, so I installed the most recent one. That works.
After that, in the requirements.txt file, I removed chainer and cupy and just did pip install -r requirements.txt for the rest.
Also, I had to switch to Torch 1.0 version as I think torch 0.3 does not support CUDA 9.0.
Hope this helps!

Have you run the project successfully? I have CUDA9.2 and use Torch1.0 but it has errors that says
a value of type "const float *" cannot be used to initialize an entity of type "float *"

[ShougangShen]

python2.7, CUDA8.0, chainer 3.3.0, cupy 2.3.0, pytorch 0.3.1 works for me. Before running the demo command, I run: export PATH="$PATH:/usr/local/cuda-8.0/bin" ， export LD_LIBRARY_PATH="/usr/local/cuda-8.0/lib64" and reinstall the chainer and cupy.