Some questions about Key.Net

[javierttgg]

Hi @axelBarroso , congrats for the great work and thanks for open-sourcing it :)

I have tried a little the system and the detections that I'm getting look like the following examples:

My gut feeling tells me that the keypoints detected at the top may not be as distinctive as the ones detected in other regions. On the one hand they are highly concentrated and on the other hand they belong regions without texture.
Is this a sign that I may have broken something while installing and playing with the system?

Otherwise, Key.Net should be tuned with the following hyperparameters?:

1. Image sizes -> Does the system work better with certain image sizes?
2. Threshold in the response maps -> I have found that Key.Net uses a 0-threshold after applying NMS to extract the keypoints. However I feel that these extractions may be directly related to this threshold. Do you recommend a certain value or should we keep it at 0?
3. Since Key.Net has as input the image at different scales, do you recommend applying NMS at the end of the whole extraction process to avoid highly concentrated keypoints?

Thanks again and apologies for the long questions.

[axelBarroso]

Hello,

Thank you for your interest in our work, and for bringing this error up with us. The keypoints on top do look strange, and this behaviour is indeed not intended. We are looking into it, we will update this issue when we find the problem.

Thank you once again!

[javierttgg]

Hi again, thank you for your answer and time :)

In the meanwhile, I delved a little more into the code/model and gathered some findings.

The TL;DR is that setting a threshold in the response map of ~1.124 solves the problem. Below I explain why I think so, hoping it is useful,

Since we have a 0-threshold when extracting the keypoints from the response maps, this causes that even when an image completely lacks texture, the system will return keypoints. An example, where the colobar refers to the keypoint' scores is shown here:

As a sidenote, in this case the features outputted by the handcrafted block are 0-valued since the gradients are also 0-valued.

As can be seen, the scores have a constant value which approximately is 1.1235435,
This exact value is also present in the "strange" keypoints of the first post. For example (saturating scores within the same range):

The origin of this value comes from the bias of the last conv. layer, as shown below:

print(keynet_model.last_conv[0].bias)
# This gives:
# Parameter containing:
# tensor([1.1235], requires_grad=True)

Since the network during inference behaves linearly, then inputs (or receptive fields) with 0-valued gradients will always be offset by this final bias. In fact, imposing this threshold we effectively remove the previous strange keypoints:

As a final note, I have also experimented the system in its Tensorflow version and this is not present since the final bias is 0:

print([v for v in tf.trainable_variables() if v.name == 'model_deep_detector_conv_last_layer/bias:0'][0].eval())
# This gives:
# [0.]

Just in case it is deemed useful I'm preparing a pull request (for that I may use my main acc. 😅).

Best