Questions on Fine-tuning for Depth Map Estimation with Known Camera Parameters

[lnSong]

Thank you for your work! I have a few questions about fine-tuning that I'd like to ask regarding depth map estimation when camera parameters are known.

1. If the intrinsic and extrinsic parameters are known, and I want to obtain a depth map, during fine-tuning, should I use the default loss function, or only losses related to depth, scale, and mask?

2. Given that the intrinsic and extrinsic parameters are known, should I freeze the main encoder, and also freeze any encoder components related to processing these parameters?

3. After obtaining the depth map with known intrinsic and extrinsic parameters, would it be better to use the ground truth (GT) of intrinsic and extrinsic parameters or the predicted intrinsic and extrinsic to generate the point map for optimal results?

[Nik-V9]

Hi, I would recommend the following:

1. Use the following loss for both train and val:

"ExcludeTopNPercentPixelLoss(FactoredGeometryScaleRegr3DPlusNormalGMLoss(RobustRegressionLoss(alpha=0.5, scaling_c=0.05), norm_mode='avg_dis', depth_type_for_loss='depth_along_ray', loss_in_log=True, flatten_across_image_only=True, compute_pairwise_relative_pose_loss=False, compute_world_frame_points_loss=True, apply_normal_and_gm_loss_to_synthetic_data_only=True, cam_frame_points_loss_weight=0.1, depth_loss_weight=0.1, ray_directions_loss_weight=0.1, pose_quats_loss_weight=0.1, pose_trans_loss_weight=0.1, scale_loss_weight=0.1, world_frame_points_loss_weight=1, normal_loss_weight=0.3, gm_loss_weight=0.3), top_n_percent=5, apply_to_real_data_only=True, loss_set_indices=[0, 1, 2])"

2. I think freezing the encoder would be good since the magnitude of fine tuning data might be small. I would also recommend using a lower learning rate for the rest of the architecture.

3. I would recommend using the predicted intrinsic and extrinsic to generate the point map.

[lnSong]

Thanks for your suggestion! Experiments indicate that the default loss works perfectly well.