Is this the right way to compute Camera gradient backward

[kuldeepbrd1]

Hi @Snosixtyboo @grgkopanas

Thanks for this great work and your comment on camera gradients in #84 .

I tried adding backward gradients of the loss explicitly w.r.t the camera pose using an over-parameterized SE(3) pose matrix.
Would greatly appreciate it, if you could check whether this makes sense.

In __global__ void computeCov2DCUDA(...):

        
         // ---------------- Gradients w.r.t Tcw ------------
	// Gradients due to 2D covariance : 1st gradient portion from 2D covariance -> 3D means(t))
	
        // flattened_3x4_pose = {r00, r01, r02, r10, r11, r12,  r20, r21, r22, t0, t1, t2}
	// dL/dTcw = dL/dcov_c * dcov_c/dW * dW/dTcw
	
	// Loss w.r.t W:   dL/dW = dL/dT *dT/dW
	// dL/dW = J^T * dL/dT
	float dL_dW00 = J[0][0] * dL_dT00 + J[1][0] * dL_dT10;
	float dL_dW10 = J[0][0] * dL_dT01 + J[1][0] * dL_dT11;
	float dL_dW20 = J[0][0] * dL_dT02 + J[1][0] * dL_dT12;
	float dL_dW01 = J[0][1] * dL_dT00 + J[1][1] * dL_dT10;
	float dL_dW11 = J[0][1] * dL_dT01 + J[1][1] * dL_dT11;
	float dL_dW21 = J[0][1] * dL_dT02 + J[1][1] * dL_dT12;
	float dL_dW02 = J[0][2] * dL_dT00 + J[1][2] * dL_dT10;
	float dL_dW12 = J[0][2] * dL_dT01 + J[1][2] * dL_dT11;
	float dL_dW22 = J[0][2] * dL_dT02 + J[1][2] * dL_dT12;
	
	// Loss w.r.t Tcw elements
	dL_dTcw[0] += dL_dtx * m.x + dL_dW00;
	dL_dTcw[1] += dL_dtx * m.y + dL_dW10;
	dL_dTcw[2] += dL_dtx * m.z + dL_dW20;
	dL_dTcw[3] += dL_dty * m.x + dL_dW01;
	dL_dTcw[4] += dL_dty * m.y + dL_dW11;
	dL_dTcw[5] += dL_dty * m.z + dL_dW21;
	dL_dTcw[6] += dL_dtz * m.x + dL_dW02;
	dL_dTcw[7] += dL_dtz * m.y + dL_dW12;
	dL_dTcw[8] += dL_dtz * m.z + dL_dW22;
	dL_dTcw[9] += dL_dtx;
	dL_dTcw[10] += dL_dty;
	dL_dTcw[11] += dL_dtz;

And in __global__ void preprocessCUDA():

        // ---------------- Gradients w.r.t Tcw ------------
	//  Gradients for 3x4 elements due to 2D means
	// (2nd gradient portion from 2D means -> 3D means(t))
	// flattened_3x4_pose = {r00, r01, r02, r10, r11, r12,  r20, r21, r22, t0, t1, t2}
	dL_dTcw[0] += dL_dmean.x * mean.x;
	dL_dTcw[1] += dL_dmean.x * mean.y;
	dL_dTcw[2] += dL_dmean.x * mean.z;
	dL_dTcw[3] += dL_dmean.y * mean.x;
	dL_dTcw[4] += dL_dmean.y * mean.y;
	dL_dTcw[5] += dL_dmean.y * mean.z;
	dL_dTcw[6] += dL_dmean.z * mean.x;
	dL_dTcw[7] += dL_dmean.z * mean.y;
	dL_dTcw[8] += dL_dmean.z * mean.z;
	dL_dTcw[9] += dL_dmean.x;
	dL_dTcw[10] += dL_dmean.y;
	dL_dTcw[11] += dL_dmean.z;

PS: I would much like to do this on se3 logmap representation later. But this is easier for the moment.

[phongnhhn92]

Hi @kuldeepbrd1, I also want to optimize the camera extrinsic during training my 3DGS. Can you guide me through the steps to use your fork here ? https://github.com/kuldeepbrd1/diff-3dgs-cam-grad

[VladimirYugay]

@phongnhhn92 for analytical extrinsic optimization (if the gaussian parameters are frozen) you can check https://github.com/muskie82/MonoGS.