eval_img error

[zhouilu]

hi, i run lego dataset. train step is ok, bug got this error when eval_img. enc.shape is torch.Size([0, 48]), why caused this?

File "/nerfs/Tri-MipRF/trainer/trainer.py", line 184, in eval
metric, rb, target = self.eval_img(data)
File "/root/miniconda3/envs/neurbf/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/nerfs/Tri-MipRF/trainer/trainer.py", line 167, in eval_img
rb = self.model(
File "/root/miniconda3/envs/neurbf/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/nerfs/Tri-MipRF/neural_field/model/trimipRF.py", line 86, in forward
ray_indices, t_starts, t_ends = nerfacc.ray_marching(
File "/root/miniconda3/envs/neurbf/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/root/miniconda3/envs/neurbf/lib/python3.9/site-packages/nerfacc/ray_marching.py", line 196, in ray_marching
sigmas = sigma_fn(t_starts, t_ends, ray_indices)
File "/nerfs/Tri-MipRF/neural_field/model/trimipRF.py", line 84, in sigma_fn
return self.field.query_density(positions, level_vol)['density']
File "/nerfs/Tri-MipRF/neural_field/field/trimipRF.py", line 77, in query_density
self.mlp_base(enc)
File "/root/miniconda3/envs/neurbf/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/root/miniconda3/envs/neurbf/lib/python3.9/site-packages/tinycudann/modules.py", line 186, in forward
output = _module_function.apply(
File "/root/miniconda3/envs/neurbf/lib/python3.9/site-packages/tinycudann/modules.py", line 98, in forward
native_ctx, output = native_tcnn_module.fwd(input, params)
RuntimeError: /nerfs/tiny-cuda-nn/include/tiny-cuda-nn/cutlass_matmul.h:330 status failed with error Error Internal

[wbhu]

It seems the test ray_chunk is all sampled at empty space, did you use the default hyper-parameters? And what's your GPU type, if I can reproduce this bug I can help to address it.

[zhouilu]

It seems the test ray_chunk is all sampled at empty space, did you use the default hyper-parameters? And what's your GPU type, if I can reproduce this bug I can help to address it.

yes, i use the default hyper-parameters and gpu is v100, cuda version is 11.3.

[AI-slam]

I also meet this error when eval, could you provide some solutions to solve it.