Increasing training time and GPU memory

[kwea123]

The training time is increasing and the GPU memory too (although it doesn't cause OOM)

Training time at the beginning

Training time after a while (and it keeps increasing afterwards too)

The command is

ns-train instant-ngp --vis tensorboard --data /home/ubuntu/data/nerf_data/nerf_synthetic/lego \
--trainer.steps-per-save 10000 --trainer.steps-per-eval-batch 10000 \
--trainer.steps-per-eval-image 10000 --trainer.steps-per-eval-all-images 10000 \
--trainer.max-num-iterations 10000 \
blender-data

I disabled all intermediate evals so it's training only, and no visualization either.

[tancik]

This is likely due to the dynamic batching, although I would expect the number of rays/sec to also increase. Will need to investigate.

[97littleleaf11]

Has this issue been solved? I cannot reproduce the issue on latest main branch