Approximate view matrix gradient for pose optimization

[oseiskar]

Allows camera pose optimization in Nerfstudio. A simpler drop-in replacement for #123. Compatible with nerfstudio-project/nerfstudio#2885

This gradient is not exact, but ignores certain terms that are harder to compute. A very similar approach is described in the GS-SLAM paper https://arxiv.org/pdf/2311.11700.pdf (see §3.3)

[kerrj]

Is there any comparison of how well this does vs the gradient including all terms?

[oseiskar]

Is there any comparison of how well this does vs the gradient including all terms?

Not yet, unfortunately. Adding all terms to this PR to this method would be a lot trickier, since that involves differentiating with respect to rotations and there are several different rotation representations involved (one for quats and another one for viewmat).

However, the end-to-end results (e.g. PSNR) could be compared to #123 (assuming the implementation there is correct). A good comparison would also need a non-COLMAP dataset that is agreed to be a good benchmark.

[kerrj]

I'm thinking of merging this PR since its so much simpler than the complete implementation (#123 ) and seems to work well, @oseiskar does this seem reasonable?

[jh-surh]

@kerrj @maturk @oseiskar My tests show significant deviations between the approximated gradient and the pure torch implementation gradient:

diff min max: diff.max()=tensor(761.4812, device='cuda:0') diff.mean()=tensor(151.8847, device='cuda:0')
a and b:
tensor([[ 1.6570e+02,  2.4303e+02, -5.1371e+02, -6.5904e+01],
        [-6.5802e+01,  1.1076e+02, -5.0377e+02, -7.6148e+02],
        [ 7.7113e-06, -1.4581e-05, -9.9915e-06, -6.2336e-06],
        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00]], device='cuda:0')
tensor([[ 3.1297e-06,  6.3989e-05, -8.0883e-06,  1.0110e-06],
        [ 5.5008e-05,  1.0953e-05, -1.5044e-04,  1.8805e-05],
        [ 9.8217e-06, -2.4316e-05, -2.2281e-05,  1.0247e-06],
        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00]], device='cuda:0')

Could you show an example of it working well? Or maybe rebasing on a commit later than 2d33520 and adding a test for viewmat like aa950a7.

[oseiskar]

This approach has now been independently battle tested in two different not-yet-peer-reviewed academic manuscripts:

1. Our paper: Gaussian Splatting on the Move (for the derivation, see page 20)
2. BAD-Gaussians. Their implementation changes/simplifies/reorderds the other part of the formula and is derived on page 15 of the corresponding paper.

Both approaches also use this PR #97. @jh-surh I'm not sure if the gradients are directly comparable before getting rid of projmat. @kerrj: merging both this and #97 would seem like a good idea at this point 👍

[kerrj]

Going to merge this version of gradients for now since they seem to work well in practice; can re-investigate the full implementation if needed down the line.