ValueError: The histogram is empty, please file a bug report.

[ze-n-g]

Question 1
I added (Model.raydist_fn = @torch.reciprocal) in the llff_raw.gin file，Reported the error indicated in the title
RUN
CUDA_VISIBLE_DEVICES=2 bash scripts/train_raw.sh

Setting ds_accelerator to cuda (auto detect)
Found no checkpoint files in exp/raw/nightstreet with prefix checkpoint_
Number of parameters being optimized: 615740
Begin training...
0%| | 0/500000 [00:00<?, ?it/s]W0607 15:38:49.991008 140182311453056 x2num.py:14] NaN or Inf found in input tensor.
0%| | 0/500000 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/home/zenglongjian/multinerf-pytorch/train.py", line 367, in
app.run(main)
File "/home/zenglongjian/.conda/envs/multi-pytorch/lib/python3.9/site-packages/absl/app.py", line 308, in run
_run_main(main, args)
File "/home/zenglongjian/.conda/envs/multi-pytorch/lib/python3.9/site-packages/absl/app.py", line 254, in run_main
sys.exit(main(argv))
File "/home/zenglongjian/multinerf-pytorch/train.py", line 224, in main
summary_writer.add_histogram('train' + k, v, step)
File "/home/zenglongjian/.conda/envs/multi-pytorch/lib/python3.9/site-packages/tensorboardX/writer.py", line 562, in add_histogram
histogram(tag, values, bins, max_bins=max_bins), global_step, walltime)
File "/home/zenglongjian/.conda/envs/multi-pytorch/lib/python3.9/site-packages/tensorboardX/summary.py", line 209, in histogram
hist = make_histogram(values.astype(float), bins, max_bins)
File "/home/zenglongjian/.conda/envs/multi-pytorch/lib/python3.9/site-packages/tensorboardX/summary.py", line 247, in make_histogram
raise ValueError('The histogram is empty, please file a bug report.')
ValueError: The histogram is empty, please file a bug report.

Question 2
If I don't add this statement error
CUDA_VISIBLE_DEVICES=2 bash scripts/train_raw.sh

Setting ds_accelerator to cuda (auto detect)
Setting ds_accelerator to cuda (auto detect)
Found no checkpoint files in exp/raw/nightstreet with prefix checkpoint_
Number of parameters being optimized: 615740
Begin training...
0%| | 0/500000 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/home/zenglongjian/multinerf-pytorch/train.py", line 367, in
app.run(main)
File "/home/zenglongjian/.conda/envs/multi-pytorch/lib/python3.9/site-packages/absl/app.py", line 308, in run
_run_main(main, args)
File "/home/zenglongjian/.conda/envs/multi-pytorch/lib/python3.9/site-packages/absl/app.py", line 254, in _run_main
sys.exit(main(argv))
File "/home/zenglongjian/multinerf-pytorch/train.py", line 142, in main
renderings, ray_history = model(
File "/home/zenglongjian/.conda/envs/multi-pytorch/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/zenglongjian/.conda/envs/multi-pytorch/lib/python3.9/site-packages/accelerate/utils/operations.py", line 521, in forward
return model_forward(*args, **kwargs)
File "/home/zenglongjian/.conda/envs/multi-pytorch/lib/python3.9/site-packages/accelerate/utils/operations.py", line 509, in call
return convert_to_fp32(self.model_forward(*args, **kwargs))
File "/home/zenglongjian/.conda/envs/multi-pytorch/lib/python3.9/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
return func(*args, **kwargs)
File "/home/zenglongjian/multinerf-pytorch/internal/models.py", line 104, in forward
_, s_to_t = coord.construct_ray_warps(self.raydist_fn, batch['near'], batch['far'])
File "/home/zenglongjian/multinerf-pytorch/internal/coord.py", line 113, in construct_ray_warps
fn_inv = inv_mapping[fn.name]
AttributeError: 'str' object has no attribute 'name'

[jingyibo123]

The data need to contain at least 2 elements to be written as histogram..

[cannedpeanutbutter]

I also had this problem when training the zipnerf network. At first I drastically reduced the batchsize, which was useful, but the training results were not satisfactory. It turned out to be a problem with the custom dataset, and I didn't resize the picture well enough. I don't know if this will work for you.

[Lee-JaeWon]

@cannedpeanutbutter @jingyibo123 @ze-n-g

Is this resolved? I'm having the same issue and it's not working for me

[Lee-JaeWon]

@SuLvXiangXin
Is this resolved? I'm having the same issue and it's not working for me.
I also tried changing fp16 but got the same error.