Why finetuning MVSNet

[ldz666666]

Hello, thanks for your excellent work!
In the original paper, it seems that you have claimed that during finetuning, MVSNerF is free from the 2d feature extractor and the 3d cnn (Correpsonding to the MVSNet class of the code), which means the MVSNet doesn't need to be optimized during fintuning.

In introduction

In essence, the encoding volume is a localized neural representation of the radiance field; once estimated, this volume can be used directly (dropping the 3D CNN) for final rendering by differentiable ray marching.

In section 3.4

Note that, we optimize only the encoding volume and the MLP, instead of our entire network

However, in train_mvs_nerf_finetuning_pl.py and models.py. It seems that the MVSNet also needs to be optimized during finetuning, which confuses me.

Could you please explain the difference of the paper and the code? Many thanks!

[apchenstu]

Thanks, we don't finetune the mvsnet, you may want to note that the image_feature is none during finetuning, so the gradient wouldn't pass through the mvsnet.

[ldz666666]

Got it. Thank you for your reply.

[basit-7]

@apchenstu img_feat are set to none in the training script as well, why is that?

mvsnerf/train_mvs_nerf_pl.py

Line 123 in 1fdf648

rgb, disp, acc, depth_pred, alpha, ret = rendering(args, pose_ref, rays_pts, rays_NDC, depth_candidates, rays_o, rays_dir,