aachen evaluation

[wolf943134497]

Hi @GrumpyZhou Thanks for your great works!

I use patch2pix + superglue as matcher for aachen evaluation, and use eval_aachen.py script. I can get keypoints.h5 and matches.h5, but an error was encountered while running triangulation, the error as follows:
"Keypoint format not supported"

I find the format slightly different from the hloc, any suggestion? Did you test patch2pix + superglue on aachen ? Thanks!

[GrumpyZhou]

Hi @wolf943134497,

Yes, i tested patch2pix+superglue on aachen as shown in the paper. I have not encountered this error before. Can you show me your config for patch2pix+superglue on aachen and the command for the evaluation ?

[wolf943134497]

Hi @wolf943134497,

Yes, i tested patch2pix+superglue on aachen as shown in the paper. I have not encountered this error before. Can you show me your config for patch2pix+superglue on aachen and the command for the evaluation ?

Hi Thanks for your quick reply!

I just use the config as follows:

default: &default
class: 'Patch2PixRefined'
ckpt: 'image-matching-toolbox/third_party/patch2pix/pretrained/patch2pix_pretrained.pth'
imsize: 1024
coarse_default: &coarse_default
name: 'SuperGlue'
weights: 'outdoor'
sinkhorn_iterations: 50
max_keypoints: 4096
nms_radius: 4
imsize: 1024
aachen:
<<: *default
match_threshold: 0.1
imsize: 1600
rthres: 48
coarse:
<<: *coarse_default
match_threshold: 0.1
imsize: 1600

---

my code :
# Initialize Model
config_file = 'configs/patch2pix+superglue.yml'
with open(config_file, 'r') as f:
args = yaml.load(f, Loader=yaml.FullLoader)['aachen']
class_name = args['class']
model = immatch.dictclass_name
print('model:',model)
matcher = lambda im1, im2: model.match_pairs(im1, im2)

dataset_dir = Path('data')
im_dir = dataset_dir / 'images/'
output_dir = Path('output')

# Load db & query pairs
db_pairs_path = output_dir / 'pairs-db-covis20.txt'
query_pairs_path = output_dir / 'pairs-query-vlad20.txt'
pair_list = load_pairs(db_pairs_path, query_pairs_path)
if not pair_list:
    return


# Match pairs
match_pairs_exporth5(pair_list, matcher, im_dir, output_dir) 


# Extract keypoints

process_matches_and_keypoints_exporth5(
    pair_list, output_dir, output_dir )

# Localization
#init_empty_sfm(args)
result_sfm = Path('output') / 'result_sfm'
empty_sfm = Path('output') / 'sfm_base'
result_dir = output_dir
colmap = 'colmap'
reconstruct_database_pairs(result_sfm, empty_sfm, im_dir, db_pairs_path, result_dir, colmap)
localize_queries(result_sfm, dataset_dir, query_pairs_path, result_dir)

if name == 'main':
parser = argparse.ArgumentParser(description='Localize Aachen Day-Night')
parser.add_argument('--gpu', '-gpu', type=str, default=0)
parser.add_argument('--config', type=str, default=None)
parser.add_argument('--prefix', type=str, default=None)

args = parser.parse_args()

eval_aachen(args)

---

I just want to get the localize_queries results, and test the pipeline is right or not.

I get init_empty_sfm by sfm. I follow the pipeline as hloc, and just change the matcher. The feature and matches can import the colmap database.

Thanks!

[GrumpyZhou]

Um so you used my code to generate the matches and used original hloc repo to import matches and features? Am i understanding it correctly?

[wolf943134497]

Um so you used my code to generate the matches and used original hloc repo to import matches and features? Am i understanding it correctly?

sorry for the unclear description, I use your code to generate the matches and use your script localize_sfm_helper.py to import matches and features, the hloc are also your repo.

only the pairs-db-covis20.txt,pairs-query-vlad20.txt and init_empty_sfm generate by original hloc repo.

Thanks!

[GrumpyZhou]

I don't understand in your posted code

reconstruct_database_pairs(result_sfm, empty_sfm, im_dir, db_pairs_path, result_dir, colmap)
localize_queries(result_sfm, dataset_dir, query_pairs_path, result_dir)

In the original code, the input argument is different from what you are using:

image-matching-toolbox/immatch/utils/localize_sfm_helper.py

Line 117 in 6ec826d

def localize_queries(args):

Apart from this, i also found something not correct in your config, since to evaluate patch2pix + superglue on aachen, you need to perform quantization (we also mentioned it in the paper).
Without quantization, errors might occur during the triangulation or you get much worse accuracy. This is because patch2pix regressed matches are not repeatable compared to keypoint-based methods.

So to help you get this out, i will update the repo by including the configuration that i used and the running command tonight or tomorrow. Since i still need to work on something else before i touch this.

[wolf943134497]

I don't understand in your posted code

reconstruct_database_pairs(result_sfm, empty_sfm, im_dir, db_pairs_path, result_dir, colmap)
localize_queries(result_sfm, dataset_dir, query_pairs_path, result_dir)

In the original code, the input argument is different from what you are using:

image-matching-toolbox/immatch/utils/localize_sfm_helper.py

Line 117 in 6ec826d

def localize_queries(args):

Apart from this, i also found something not correct in your config, since to evaluate patch2pix + superglue on aachen, you need to perform quantization (we also mentioned it in the paper).
Without quantization, errors might occur during the triangulation or you get much worse accuracy. This is because patch2pix regressed matches are not repeatable compared to keypoint-based methods.

So to help you get this out, i will update the repo by including the configuration that i used and the running command tonight or tomorrow. Since i still need to work on something else before i touch this.

I also change the the function variables in localize_sfm_helper.py, just write it directly in your code. I also perform quantization by set qt_psize = 48, qt_dthres = 4 ,sc_thres = 0.25 in code. I will check my code and h5 file.

Looking forward to your updates, Thanks for your great works again, it is very meaningful !

[GrumpyZhou]

Hey, can you try with my example command in readme for evaluating patch2pix+superglue on aachen? Please inform me here if you succeed with it. Thanks~

[wolf943134497]

Hey, can you try with my example command in readme for evaluating patch2pix+superglue on aachen? Please inform me here if you succeed with it. Thanks~

I will try with your example command for evaluating patch2pix+superglue on aachen, I'll give you feedback. Thanks~

[wolf943134497]

Hey, can you try with my example command in readme for evaluating patch2pix+superglue on aachen? Please inform me here if you succeed with it. Thanks~

I will try with your example command for evaluating patch2pix+superglue on aachen, I'll give you feedback. Thanks~

@GrumpyZhou Hi The code runs successfully! Thanks~

[TruongKhang]

Hi @GrumpyZhou, I wonder how long your program takes to evaluate on Aachen DayNight v1.1 dataset? My current running is to match 140952 pairs estimated in 50 hours. That's too long.

[GrumpyZhou]

Hi @TruongKhang,

I do not remember the exact time now but yes it was taking a lot of time (* days) to run the whole evaluation on Aachen due to the large number of pairs.

[TruongKhang]

Thank you @GrumpyZhou. Also, I have one more question. I wonder how to get the results Local Feature Evaluation on Night-time Queries from the table in your paper. This current code produced the results of Full Localization with HLOC, am I right?

[GrumpyZhou]

Hi @TruongKhang ,

I just uploaded the config in this commit e3801b6 .

[TruongKhang]

Thank you so much for your engaging discussion! I wish you the best in your research :D