Multiple types of keypoints and descriptors

[ducha-aiki]

Hi,

I am trying to fit the kapture for my use case, which requires multiple types of local features, both detectors and descriptors, e.g. SuperPoint, R2D2, SIFT+HardNet.
The current kapture format assumes a single type of local feature -- which is hardcoded in the filenames "keypoints.txt", "descriptors.txt". If I would make a fork for support multiple feature types, would it be merged at some point, or do you not plan to change the format?

Best, Dmytro.

[humenbergerm]

Hi Dmytro!

Thanks for your interest in this. Indeed, we already discussed a lot how to support multiple types of local features. We will continue the discussion and get back to you as soon as possible.

Best,
Martin

[ducha-aiki]

Thank you, Martin! If possible, I would be happy to join the discussion :)
My email is ducha.aiki@gmail.com just in case

[humenbergerm]

Great! It would also be helpful if you could share some more details about your needs. At the moment we use kapture with multiple features, but not within the same reconstruction.
Best, Martin

[ducha-aiki]

Sure. The general idea is to revisit MODS, which combines the multiple type of features in order to speed-up matching of the easy pairs and to allow matching the hard pairs, where any single feature fails, but combined they work.
So, first we are matching everything which is matchable by ORB, then we add, say, SuperPoint, and so on.

https://github.com/ducha-aiki/mods-light-zmq
https://arxiv.org/abs/1503.02619

[yocabon]

Hi,
We've started to draft a new version of the format (1.1) that should hopefully answer your needs.
The new specifications are being written here
https://github.com/naver/kapture/blob/1.1_draft/kapture_format.adoc

There are no changes to the "sensor" part of the format.
In the "reconstruction" part, multiple keypoints/descriptors/global features types can be given. keypoints can be represented by multiple descriptors.
Observations can reference different types of keypoints.

What do you think of these changes ?

[ducha-aiki]

Hi,

It fits my use-case, thank you!
One thing, I would like to add is a metric for descriptors. So the descriptors.txt would be:

name, dtype, dsize, keypoints_type, metric_type

where the metric is "L2" for the most and "Hamming" for the binary features, for example.