pycuda.driver.CompileError

[BoMingZhao]

Hello,
Thanks for the awesome work. However when I run the Per-scene Optimization on NeRF-Synthetics I found some error below:

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Debug Mode
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
/home/zhaoboming/anaconda3/envs/nerf-w/lib/python3.7/site-packages/numpy/core/shape_base.py:420: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
arrays = [asanyarray(arr) for arr in arrays]
/home/zhaoboming/anaconda3/envs/nerf-w/lib/python3.7/site-packages/torch/functional.py:568: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1646755953518/work/aten/src/ATen/native/TensorShape.cpp:2228.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
dataset total: train 100
dataset [NerfSynthFtDataset] was created
../checkpoints/nerfsynth/lego/*_net_ray_marching.pth
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Continue training from 200000 epoch
Iter: 200000
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
opt.act_type!!!!!!!!! LeakyReLU
self.points_embeding torch.Size([1, 479862, 32])
querier device cuda:3 3
Traceback (most recent call last):
File "train_ft.py", line 1084, in
main()
File "train_ft.py", line 637, in main
model = create_model(opt)
File "/mnt/data1/zhaoboming/pointnerf/run/../models/init.py", line 39, in create_model
instance.initialize(opt)
File "/mnt/data1/zhaoboming/pointnerf/run/../models/base_rendering_model.py", line 369, in initialize
self.create_network_models(opt)
File "/mnt/data1/zhaoboming/pointnerf/run/../models/mvs_points_volumetric_model.py", line 44, in create_network_models
super(MvsPointsVolumetricModel, self).create_network_models(opt)
File "/mnt/data1/zhaoboming/pointnerf/run/../models/neural_points_volumetric_model.py", line 157, in create_network_models
params = self.get_additional_network_params(opt)
File "/mnt/data1/zhaoboming/pointnerf/run/../models/neural_points_volumetric_model.py", line 142, in get_additional_network_params
self.neural_points = NeuralPoints(opt.point_features_dim, opt.num_point, opt, self.device, checkpoint=checkpoint_path, feature_init_method=opt.feature_init_method, reg_weight=0., feedforward=opt.feedforward)
File "/mnt/data1/zhaoboming/pointnerf/run/../models/neural_points/neural_points.py", line 331, in init
self.querier = self.lighting_fast_querier(device, self.opt)
File "/mnt/data1/zhaoboming/pointnerf/run/../models/neural_points/query_point_indices_worldcoords.py", line 39, in init
self.claim_occ, self.map_coor2occ, self.fill_occ2pnts, self.mask_raypos, self.get_shadingloc, self.query_along_ray = self.build_cuda()
File "/mnt/data1/zhaoboming/pointnerf/run/../models/neural_points/query_point_indices_worldcoords.py", line 524, in build_cuda
""", no_extern_c=True)
File "/home/zhaoboming/anaconda3/envs/nerf-w/lib/python3.7/site-packages/pycuda-2021.1-py3.7-linux-x86_64.egg/pycuda/compiler.py", line 358, in init
include_dirs,
File "/home/zhaoboming/anaconda3/envs/nerf-w/lib/python3.7/site-packages/pycuda-2021.1-py3.7-linux-x86_64.egg/pycuda/compiler.py", line 298, in compile
return compile_plain(source, options, keep, nvcc, cache_dir, target)
File "/home/zhaoboming/anaconda3/envs/nerf-w/lib/python3.7/site-packages/pycuda-2021.1-py3.7-linux-x86_64.egg/pycuda/compiler.py", line 87, in compile_plain
checksum.update(preprocess_source(source, options, nvcc).encode("utf-8"))
File "/home/zhaoboming/anaconda3/envs/nerf-w/lib/python3.7/site-packages/pycuda-2021.1-py3.7-linux-x86_64.egg/pycuda/compiler.py", line 59, in preprocess_source
"nvcc preprocessing of %s failed" % source_path, cmdline, stderr=stderr
pycuda.driver.CompileError: nvcc preprocessing of /tmp/tmpeytiys5x.cu failed
[command: nvcc --preprocess -arch sm_86 -I/mnt/data1/zhaoboming/anaconda3/envs/nerf-w/lib/python3.7/site-packages/pycuda-2021.1-py3.7-linux-x86_64.egg/pycuda/cuda /tmp/tmpeytiys5x.cu --compiler-options -P]
[stderr:
b'cc1plus: fatal error: cuda_runtime.h: No such file or directory\ncompilation terminated.\n']
end loading

---

PyCUDA ERROR: The context stack was not empty upon module cleanup.

A context was still active when the context stack was being
cleaned up. At this point in our execution, CUDA may already
have been deinitialized, so there is no way we can finish
cleanly. The program will be aborted now.
Use Context.pop() to avoid this problem.

So How can I fix it? Thanks.

[yjcaimeow]

The same error. How to fix it ?

[hdzmtsssw]

same problem

[Zoe-Yan123]

Same problem

[dongshengLiu]

Same problem

Same problem

[william2ai]

@BoMingZhao Did you solve it?

[BoMingZhao]

Yes. I think either CUDA is not installed correctly or added to the environment variable, or there are multiple versions of CUDA installed.
Please check your NVCC version by using

nvcc --version

Here is my solved env:
Nvidia 3090 with CUDA 11.1, PyCUDA 2021.1, and PyTorch 1.8.1.