add gradient accumulation support

[ksnzh]

This modification has no hurt to current repo. The train step number is same as before, so optimizer and scheduler works on old way. Users can easily increase batch size since nerf network has no batchnorm layer.

[tancik]

Run to fix issues.

pip install -e .[dev]
pre-commit install
pre-commit

[brentyi]

Would it make sense to:

• Divide the batch size by the gradient accumulation count?
• Provide an "auto" mode that computes the accumulation steps to achieve some effective batch size, while still respecting memory constraints?

(relevant to #2003, which won't fit on smaller GPUs)

[ksnzh]

Would it make sense to:

• Divide the batch size by the gradient accumulation count?
• Provide an "auto" mode that computes the accumulation steps to achieve some effective batch size, while still respecting memory constraints?

(relevant to #2003, which won't fit on smaller GPUs)

I think it's better to let user decide batchsize and coresponding learning rate.

[kerrj]

This would be clearer with a for loop, since internal_step is unnecessary except for counting iterations

[kerrj]

I think memory budgeting is a larger PR in itself, since going from batch size->memory consumption is a pretty nontrivial thing. I'd be in favor of merging in grad accumulation and thinking about how to automatically set these parameters later.

[ksnzh]

Here are two experients which use num rays of batch 8192 and accumulate two 4096 rays step.

https://wandb.ai/ksnzh/nerfstudio-public

[kerrj]

Looks good, thanks!