about pose optimizing

[hdzmtsssw]

Thanks for great work!
I would like to add a camera optimizer to optimize the camera pose during training. Specifically, I will introduce a learnable parameter ext_emb_(N * 6), translate it to SO3xR3(N * 3 * 4), and then 'multiply' poses_gt and ext_emb_. The resulting poses_ will be used to generate rays. [referenced nerfstudio here]
However, it seems that the gradient will be truncated at pts in scene_feat = scene_field_->AnchoredQuery(pts, anchors). Does this mean that I have to complete all the related CUDA kernel backward functions? Do you have any idea about this?

[Bin-ze]

Hi, I would like to ask if you have any new progress? I have the same needs as you

[Bin-ze]

@Totoro97 Sorry to bother you, can you take the time to reply, this is very important to me

[hdzmtsssw]

Hi, I would like to ask if you have any new progress? I have the same needs as you

Sorry, I haven't made any progress yet. I'm not very familiar with libtorch and CUDA, so this is a bit too difficult for me.

[lvmingzhe]

Hi, I am also curious about optimizing the poses under the architect of Tiny-Cuda-NN. It seems more difficult than optimizing poses with Pytorch. I don`t know whether someone could transfer f2-nerf into NeRFStudio. If not, maybe we can have a free discussion about optimizing the poses in Tiny-Cuda-NN. Please email me lmzo@vip.163.com

[Bin-ze]

Hi, I would like to ask if you have any new progress? I have the same needs as you

Sorry, I haven't made any progress yet. I'm not very familiar with libtorch and CUDA, so this is a bit too difficult for me.

You mentioned the problem of gradient cutoff, how did you find it? I tried to search, but couldn't find a description of this. I plan to introduce a learnable pose optimization variable in the dataset class and multiply it with the input pose, but this variable acts before the light is generated. If the gradient is only propagated after the light is sampled, it will not be possible to optimize the variable you want to improve.
I don’t have any experience with libtorch. This part of the work is very difficult for me. I hope I can find a friend to discuss and try to implement it. He will further improve the performance of f2-nerf in practical applications, which is very meaningful

[Bin-ze]

Hi, I am also curious about optimizing the poses under the architect of Tiny-Cuda-NN. It seems more difficult than optimizing poses with Pytorch. I don`t know whether someone could transfer f2-nerf into NeRFStudio. If not, maybe we can have a free discussion about optimizing the poses in Tiny-Cuda-NN. Please email me lmzo@vip.163.com

Moving f2-nerf to nerf-stidio, I feel that it can be realized with the help of pybinding, encapsulating the key c++ implementation. this is my personal opinion
Do you have any experience with libtorch, especially libtorch's features, such as neural networks are not combined in a class, so what kind of implementation can gradient propagation? I tried to implement pose optimization in nerfstudio using libtorch, but no progress yet

[Totoro97]

Hi @hdzmtsssw @Bin-ze Thanks for your interest in this project! and sorry for the late reply, I am recently buried from another project and might not be able to reply in time.

I think it needs some extra efforts to the implementation of pose optimization. More specifically, currently the gradient of the loss function can not be backpropagated to the pose parameters.

To implement a complete backpropagation to poses, we need some extra efforts to implment 1) The back-propagating of loss gradients to the input point positions in Hash3DAnchored function, 2) The back-propagating of the point gradients to the ray parameters (rays_o and rays_d) of point sampling functions, and 3) The back-propagating of the gradients from ray parameters to camera poses in the ray sampling function from the Dataset class.

They have not been finished yet. And currently I do not have a specific plan with timeline to support that. Pull requests are welcomed.

[Bin-ze]

Hi @hdzmtsssw @Bin-ze Thanks for your interest in this project! and sorry for the late reply, I am recently buried from another project and might not be able to reply in time.

I think it needs some extra efforts to the implementation of pose optimization. More specifically, currently the gradient of the loss function can not be backpropagated to the pose parameters.

To implement a complete backpropagation to poses, we need some extra efforts to implment 1) The back-propagating of loss gradients to the input point positions in Hash3DAnchored function, 2) The back-propagating of the point gradients to the ray parameters (rays_o and rays_d) of point sampling functions, and 3) The back-propagating of the gradients from ray parameters to camera poses in the ray sampling function from the Dataset class.

They have not been finished yet. And currently I do not have a specific plan with timeline to support that. Pull requests are welcomed.

@Totoro97
Thank you very much for your reply!
I have a few questions I want to ask you:

1. I would like to know what specific steps are needed to realize the backpropagation of the several parts you mentioned? For example, some parts have been implemented in your code base, which can be used as a reference to realize the missing parts?
2. I checked instant-ngp, which also provides the option of pose optimization. Is it very difficult to add the pose optimization pipeline proposed by instant-ngp to your framework?
3. What is your opinion on how pose optimization improves the reconstruction quality in real scenes?
   Hope to get your reply！

[hdzmtsssw]

Hi, I would like to ask if you have any new progress? I have the same needs as you

Sorry, I haven't made any progress yet. I'm not very familiar with libtorch and CUDA, so this is a bit too difficult for me.

You mentioned the problem of gradient cutoff, how did you find it? I tried to search, but couldn't find a description of this. I plan to introduce a learnable pose optimization variable in the dataset class and multiply it with the input pose, but this variable acts before the light is generated. If the gradient is only propagated after the light is sampled, it will not be possible to optimize the variable you want to improve. I don’t have any experience with libtorch. This part of the work is very difficult for me. I hope I can find a friend to discuss and try to implement it. He will further improve the performance of f2-nerf in practical applications, which is very meaningful

I identified the issue of gradient cutoff by using the requires_grad() function to track the variable and confirm if it needs gradients. Eventually, I discovered that the cutoff occurred at the AnchoredQuery function. Specifically, the Hash3DAnchoredFunction doesn't compute the gradient of the second input at the backward function.

[hdzmtsssw]

Hi @hdzmtsssw @Bin-ze Thanks for your interest in this project! and sorry for the late reply, I am recently buried from another project and might not be able to reply in time.

I think it needs some extra efforts to the implementation of pose optimization. More specifically, currently the gradient of the loss function can not be backpropagated to the pose parameters.

To implement a complete backpropagation to poses, we need some extra efforts to implment 1) The back-propagating of loss gradients to the input point positions in Hash3DAnchored function, 2) The back-propagating of the point gradients to the ray parameters (rays_o and rays_d) of point sampling functions, and 3) The back-propagating of the gradients from ray parameters to camera poses in the ray sampling function from the Dataset class.

They have not been finished yet. And currently I do not have a specific plan with timeline to support that. Pull requests are welcomed.

@Totoro97 Thank you for your reply and helpful suggestions!

1. During my attempt to implement the back-propagating of loss gradients to the input point positions in Hash3DAnchoredFunction, I encountered a challenge: the second input type in the forward function is IValue, and I am unsure how to properly return its gradient in the backward function.
2. What does the anchors mean in AnchoredQuery function? Does it also need a gradient in this case?
3. I noticed that the pts in Renderer.cpp is derived from sample_result_early_stop, which is within the torch::NoGradGuard no_grad_guard scope. Could it potentially affect the backpropagation?

[Bin-ze]

Hi, I would like to ask if you have any new progress? I have the same needs as you

Sorry, I haven't made any progress yet. I'm not very familiar with libtorch and CUDA, so this is a bit too difficult for me.

You mentioned the problem of gradient cutoff, how did you find it? I tried to search, but couldn't find a description of this. I plan to introduce a learnable pose optimization variable in the dataset class and multiply it with the input pose, but this variable acts before the light is generated. If the gradient is only propagated after the light is sampled, it will not be possible to optimize the variable you want to improve. I don’t have any experience with libtorch. This part of the work is very difficult for me. I hope I can find a friend to discuss and try to implement it. He will further improve the performance of f2-nerf in practical applications, which is very meaningful

I identified the issue of gradient cutoff by using the requires_grad() function to track the variable and confirm if it needs gradients. Eventually, I discovered that the cutoff occurred at the AnchoredQuery function. Specifically, the Hash3DAnchoredFunction doesn't compute the gradient of the second input at the backward function.

Can we use WeChat to discuss? Bin1978108038

[lvmingzhe]

Have you guys read this ICML2023 paper about pose optimization with instant-NGP?
Robust Camera Pose Refinement for Multi-Resolution Hash Encoding
I think it might be helpful for understanding the backpropagation process of Hash3DAnchored.