Rotation of Spherical harmonics data

[hardikdava]

Great work @yzslab , is it possible to add support for rotation and scalling of spherical harmonics data with the transformation of 3dgs model?

[yzslab]

Sorry that not planning yet. Contributions are welcomed!

…

On Mon, Apr 29, 2024 at 14:39 Hardik Dava ***@***.***> wrote: Great work @yzslab <https://github.com/yzslab> , is it possible to add support for rotation and scalling of spherical harmonics data with the transformation of 3dgs model? — Reply to this email directly, view it on GitHub <#26>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEJZCPLHIMYFPESFZ3OFEDY7XTLFAVCNFSM6AAAAABG5XGE6WVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI3DQMRUGM2DGOI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[yzslab]

Rotate the first extra degree has been implemented.
The calculation of the higher SH degrees are far more tedious, so only one degree currently.

[hardikdava]

@yzslab thanks a lot. Can you give me the direction how to achieve the rotation for 3 sh degree? I would like to try it out.

[yzslab]

@yzslab thanks a lot. Can you give me the direction how to achieve the rotation for 3 sh degree? I would like to try it out.

Already provided in the code comment:

gaussian-splatting-lightning/utils/gaussian_transform.py

Lines 86 to 96 in 098b0b7

	"""
	If the scene has been rotated,
	in order to get the correct color as without rotation,
	we should rotate the view direction opposite.
	E.g.,
	the scene is rotated by R: means3D' = R @ means3D,
	then when calculating colors,
	the correct view direction should be: R.T @ (means3D' - camera_center).
	Based on the how the SHs multiply with view directions, we know how to rotate the SHs.
	"""

[hardikdava]

@yzslab It's not possible to rotate higher degree of spherical harmonics without knowing camera matrix. Am I right?

P.S. I want to do it offline. Trained model -> rotate -> visualize

[yzslab]

@yzslab It's not possible to rotate higher degree of spherical harmonics without knowing camera matrix. Am I right?

P.S. I want to do it offline. Trained model -> rotate -> visualize

The camera matrix isn't required. The R is the matrix that rotate your scene.

[hardikdava]

@yzslab I tried your implementation and it's working perfectly for 1st degree. But still missing implementation for 2nd and higher degree.

[yzslab]

@yzslab I tried your implementation and it's working perfect for 1st degree. But still missing implementation for 2nd and higher degree.

The calculation of the higher SH degrees are far more tedious.
But rotating the view direction vectors are more easier: R.T @ (means3D' - camera_center). I think you can modifying the rasterizer to get correct colors by this way.

[yzslab]

@yzslab I tried your implementation and it's working perfect for 1st degree. But still missing implementation for 2nd and higher degree.

The calculation of the higher SH degrees are far more tedious. But rotating the view direction vectors are more easier: R.T @ (means3D' - camera_center). I think you can modifying the rasterizer to get correct colors by this way.

Just need to modify the rasterizer to accept a rotation matrix R as a parameter, and apply it when calculating colors.

[hardikdava]

@yzslab I think there is an error with your formula. It will create dim mismatch error. Can you recheck it once?

[yzslab]

@yzslab I think there is an error with your formula. It will create dim mismatch error. Can you recheck it once?

Would you mind providing more details?

[yzslab]

@yzslab I think there is an error with your formula. It will create dim mismatch error. Can you recheck it once?

Would you mind providing more details?

The means3D' - camera_center is a column vector.

[hardikdava]

@yzslab I am using modified version of gsplat rasterizer. I have rotated first degree as you implemented. I modified the color computation as you suggested and I am getting following error:

    viewdirs = camera.camera_to_worlds.detach()[..., :3, :3].T @ (means_crop.detach() - camera.camera_to_worlds.detach()[..., :3, 3])
RuntimeError: mat1 and mat2 shapes cannot be multiplied (9x1 and 82214x3)

[yzslab]

@yzslab I am using modified version of gsplat rasterizer. I have rotated first degree as you implemented. I modified the color computation as you suggested and I am getting following error:

    viewdirs = camera.camera_to_worlds.detach()[..., :3, :3].T @ (means_crop.detach() - camera.camera_to_worlds.detach()[..., :3, 3])
RuntimeError: mat1 and mat2 shapes cannot be multiplied (9x1 and 82214x3)

The means3D and camera_center in Python code are row vectors. So the correct calculation in code should be:

(means3D` - camera_center) @ R

NOTE:

• No transpose/inverse is not required for $R$ if placing on the right.
• The R isn't the camera matrix, should be the one rotating your scene.

[yzslab]

have rotated first degree

Besides, you should not rotate SHs and the view direction together. Rotation should be applied either for SHs or view directions.

[hardikdava]

@yzslab You are right. I got it working for applying Rotation directly to view direction. How can I achieve rotation without modifying the rasterizer? Is it possible?

[yzslab]

How can I achieve rotation without modifying the rasterizer? Is it possible?

I think rotating the SHs is the only way to achieve this.

[hardikdava]

@yzslab can you send me references on how can rotate SH for higher degree if you have it?

[yzslab]

@yzslab can you send me references on how can rotate SH for higher degree if you have it?

I think the theory is the same as the 1st extra degree. Someone mentioned that it could be done with Wigner D-matrix: graphdeco-inria/gaussian-splatting#176 (comment)
Here is an implementation: graphdeco-inria/gaussian-splatting#176 (comment)
I don't know whether it is correct. The results of the 1st extra degree are differ to the calculation here.

[hardikdava]

@yzslab I tried the mentioned method. But It's not yielding good result.

[yzslab]

@yzslab I tried the mentioned method. But It's not yielding good result.

Thanks to MstXy, this problem has been solved now: graphdeco-inria/gaussian-splatting#176 (comment)
I validated that the latest one is correct.

[hardikdava]

@yzslab Thanks for the support. I have also verified the result.