Add non-trainable camera indices to CameraOptimizer #2080
Conversation
| @@ -121,11 +123,13 @@ def forward( | |||
| outputs.append(exp_map_SE3(self.pose_adjustment[indices, :])) | |||
| else: | |||
| assert_never(self.config.mode) | |||
| # Detach non-trainable indices by setting to identity transform | |||
| if self.non_trainable_camera_indices is not None: | |||
| outputs[0][self.non_trainable_camera_indices] = torch.eye(4)[:3, :4].to(self.device) | |||
There was a problem hiding this comment.
Create on device
torch.eye(4, device=self.device)
|
LGTM with Matt's change, useful in robotics when you want to prevent the coord frame from drifting too much thru optimization. I wonder if eventually it could be good to optionally input a covariance matrix for each camera, then you can even have a per-camera confidence :). Not sure how you'd incorporate that into the optimization, but maybe just some simple regularization like weighting loss higher on lower confidence cameras |
|
Regarding the camera optimizer, on the top of the file it says |
|
Yes there is no intrinsic optimization. We're also thinking about refactoring camera optimization to move it out of the datamanager and into the Model, since this allows more natural development of different methods (BARF etc), and also would help speed up dataloading (right now ray generation cannot be parallelized because gradients need to flow from model to datamanager). Might be good to keep this in mind, and if you have thoughts on a refactor we'd love to hear them too |
…ct#2080) * add non-trainable indices to camera optimizer * improve comment * create on device --------- Co-authored-by: Matthew Tancik <tancik@berkeley.edu>
Here I added the possibility for a user to manually tell the Camera Optimizer not to train certain camera indices. This is needed when you know prior that only certain indices should be trained in the camera training_dataset.