Regarding information required to be able to train on custom dataset

[amanshenoy]

What exactly would I need to provide to be able to train LoFTR on a custom dataset? It was pointed out in some of the previous issues that all one would need would be depth map, extrinsics and intrinsics for every image and the ground truth is generated on the fly.

The npz files seem to have fields - 'image_paths', 'depth_paths', 'intrinsics', 'poses', 'pair_infos'. Assuming 'poses' is just the extrinsics (R | t), how exactly is 'pair_infos' generated and how are the covisibility scores calculated?

[zehongs]

Yes, you need rgb images, depth maps, extrinsic and intrinsic parameters to train LoFTR.
pair_infos contains the predefined pairs, i.e. [(img0_id, img1_id)].
The source/definition of covisibility for MegaDepth and ScanNet are different. For MegaDepth, we used the results provided by D2Net repo, please check their paper for more details. For ScanNet, we computed percentage of the covisible pixels for all possible image pairs. Please refer to this issue.

[amanshenoy]

Thanks for your response. Once the d2net processed npz file is created is there any specific way to be able to get the npz files that were used for training?

[amanshenoy]

Also wanted to ask regarding what the 'central_matches' value is in pair_infos

[whh14]

just wondering, is it possible to train the network with image pairs plus ground truth (matching keypoints), like the output from the network.

[zehongs]

just wondering, is it possible to train the network with image pairs plus ground truth (matching keypoints), like the output from the network.

Possibly not. Only if you have ground truth point on the 1/8 grids. And please open another issue to discuss this if you have more questions.

[zehongs]

central_matches

We didn't use this.

[zehongs]

Thanks for your response. Once the d2net processed npz file is created is there any specific way to be able to get the npz files that were used for training?

I'm sorry that I don't have the code now. I think you should set-up train/val/test splits and reorder the data dictionary. Have you solved the problem yet?

[amanshenoy]

Yes, thanks!