Segmentation fault happened while calling DMTetGeometry

[mush881212]

Hi Team,

Thanks for your amazing work!
I try to run the program, but I get the segmentation fault happen when calling DMTetGeometry(FLAGS.dmtet_grid, FLAGS.mesh_scale, FLAGS) in train.py.
I tracked down the error and found that it happens when calling ou.OptiXContext() in dmtet.py.
I think the error might be happening because of calling _plugin.OptiXStateWrapper(os.path.dirname(file), torch.utils.cpp_extension.CUDA_HOME) in ops.py, but I don't know how to fix it.

I tried to reduce the batch size from 8 to 1 and the image train resolution from 512x512 to 128x128, but the problem persists.
Can you give some advice on how to solve this problem?

GPU Hardware:
Nvidia A100 (32G) on server
Console error:

Using /root/.cache/torch_extensions/py38_cu116 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /root/.cache/torch_extensions/py38_cu116/optixutils_plugin/build.ninja...
Building extension module optixutils_plugin...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module optixutils_plugin...
Config / Flags:
--
iter 5000
batch 8
spp 1
layers 1
train_res [512, 512]
display_res [512, 512]
texture_res [1024, 1024]
display_interval 0
save_interval 100
learning_rate [0.03, 0.005]
custom_mip False
background white
loss logl1
out_dir out/nerd_gold
config configs/nerd_gold.json
ref_mesh data/nerd/moldGoldCape_rescaled
base_mesh None
validate True
n_samples 12
bsdf pbr
denoiser bilateral
denoiser_demodulate True
mtl_override None
dmtet_grid 128
mesh_scale 2.5
envlight None
env_scale 1.0
probe_res 256
learn_lighting True
display [{'bsdf': 'kd'}, {'bsdf': 'ks'}, {'bsdf': 'normal'}]
transparency False
lock_light False
lock_pos False
sdf_regularizer 0.2
laplace relative
laplace_scale 3000.0
pre_load True
no_perturbed_nrm False
decorrelated False
kd_min [0.03, 0.03, 0.03]
kd_max [0.8, 0.8, 0.8]
ks_min [0, 0.08, 0.0]
ks_max [0, 1.0, 1.0]
nrm_min [-1.0, -1.0, 0.0]
nrm_max [1.0, 1.0, 1.0]
clip_max_norm 0.0
cam_near_far [0.1, 1000.0]
lambda_kd 0.1
lambda_ks 0.05
lambda_nrm 0.025
lambda_nrm2 0.25
lambda_chroma 0.025
lambda_diffuse 0.15
lambda_specular 0.0025
random_textures True
--
DatasetLLFF: 119 images with shape [512, 512]
DatasetLLFF: auto-centering at [-0.04492672 1.3252479 1.1068335 ]
DatasetLLFF: 119 images with shape [512, 512]
DatasetLLFF: auto-centering at [-0.04492672 1.3252479 1.1068335 ]
/opt/conda/lib/python3.8/site-packages/torch/functional.py:568: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/native/TensorShape.cpp:2156.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
Cuda path /usr/local/cuda
Segmentation fault (core dumped)

[jmunkberg]

Thanks @mush881212 ,

I suspect OptiX, which is quite sensitive to the driver installed on the machine. Our code uses OptiX 7.3, which requires an Nvidia display driver numbered 465 or higher. Perhaps verify that some standalone OptiX example from the OptiX 7.3 SDK runs fine on that machine.

One alternative may be to use our old code base, https://github.com/NVlabs/nvdiffrec, which is a similar reconstruction pipeline, but without the OptiX ray tracing step.

[jmunkberg]

Note also that the A100 GPU does not have any RT Cores (for ray tracing acceleration), so the ray tracing performance will be lower than what we reported in the paper (we measured on an A6000 RTX GPU).

[mush881212]

Hi @jmunkberg,

I think the problem is due to the driver version, because my driver version is too old to support OptiX 7.3.
I will try it on another device and upgrade the driver version.
Thanks for your help!

[Sheldonmao]

Hi, I wonder if the problem is solved? I have the same problem with a driver version 520.61.05 on V100, wonder how to solve it.

[mush881212]

Hi @Sheldonmao,

I solved this issue by updating the driver version and using an RTX3090 device instead.
Driver version: 465.19.01
CUDA version: 11.3
You could try using these settings.

[sungmin9939]

would relatively high version of GPU hardware be a problem? My GPU is RTX 3090 and driver version is 535.146.02 and I'm getting segmentation fault as the original author

[jmunkberg]

Newer GPUs and drivers shouldn't be an issue, I hope. It has been a while since we released this code, but I just tested on two setups without issues.

Setup 1: Windows Desktop
RTX 6000 Ada Gen w/ driver 545.84
PyTorch version: 2.0.0+cu117

Setup 2: Linux Server
V100 w/ Driver 515.86.01
Using the Dockerfile from the nvdiffrecmc repo https://github.com/NVlabs/nvdiffrecmc/blob/main/docker/Dockerfile