Added support for DoG + patch descriptors.

[mihaidusmanu]

Added code for extracting patches from VLFeat keypoints followed by SOSNet description. Renamed SIFT by DoG as it can now be used with different descriptors. Uses extract-patches for patch extraction and Kornia for SOSNet architecture + pretrained weights download.

Reconstruction statistics on South Building.

RootSIFT

	num_reg_images = 128
	num_cameras = 1
	num_points3D = 41863
	num_observations = 225031
	mean_track_length = 5.37542
	mean_observations_per_image = 1758.05
	mean_reprojection_error = 0.933379
	num_input_images = 128

SOSNet

	num_reg_images = 128
	num_cameras = 1
	num_points3D = 47597
	num_observations = 252175
	mean_track_length = 5.29813
	mean_observations_per_image = 1970.12
	mean_reprojection_error = 0.984635
	num_input_images = 128

[sarlinpe]

I have replaced extract_patches with kornia.feature.laf.extract_patches_from_pyramid, which runs on GPU and makes the overall detection+description ~15% faster. I checked that the maths are identical (with a tiny modification), but the results are slightly lower.

rootsift
	num_reg_images = 128
	num_cameras = 1
	num_points3D = 32401
	num_observations = 186126
	mean_track_length = 5.74445
	mean_observations_per_image = 1454.11
	mean_reprojection_error = 0.872702

SOSNet - extract_patches
	num_reg_images = 128
	num_cameras = 1
	num_points3D = 33462
	num_observations = 195222
	mean_track_length = 5.83414
	mean_observations_per_image = 1525.17
	mean_reprojection_error = 0.892686

SOSNet - kornia
	num_reg_images = 128
	num_cameras = 1
	num_points3D = 33530
	num_observations = 195211
	mean_track_length = 5.82198
	mean_observations_per_image = 1525.09
	mean_reprojection_error = 0.892961

Patches: extract_patches / kornia / abs(diff)

import numpy as np
from hloc.utils.io import read_image
from hloc.utils.viz import plot_images, plot_keypoints
from hloc.extract_features import resize_image

path = 'datasets/South-Building/images/P1180141.JPG'
image = read_image(path, True)
s = 1024 / max(image.shape[:2])
size = tuple(int(round(x*s)) for x in image.shape[:2][::-1])
image = resize_image(image, size, 'cv2_area')
image = image / 255.

import pycolmap
from hloc.extractors.dog import DoG
keypoints, scores, descs = pycolmap.extract_sift(image, **DoG.default_conf['vlfeat'])
plot_images([image])
plot_keypoints([keypoints[:, :2]])

patch_size = 32
mr_size = 12.0
idxs = [0, 4, 500, 1000, 1400, 2001]

from extract_patches.core import extract_patches
x = keypoints[:, 0]
y = keypoints[:, 1]
scale = keypoints[:, 2]
ori = keypoints[:, 3]
a11 = scale * np.cos(ori)
a12 = -scale * np.sin(ori)
a21 = scale * np.sin(ori)
a22 = scale * np.cos(ori)
xyA = np.stack([x, y, a11, a12, a21, a22], axis=-1)
patches = extract_patches(xyA, img2, patch_size, mr_size, 'xyA')
plot_images(np.stack(patches)[idxs])

from kornia.feature.laf import extract_patches_from_pyramid, laf_from_center_scale_ori
imgt = torch.from_numpy(image)[None, None].float()
center = keypoints[:, :2] + 0.5
scale = keypoints[:, 2] * mr_size / 2
ori = -np.rad2deg(keypoints[:, 3])
lafs = laf_from_center_scale_ori(
    torch.from_numpy(center)[None],
    torch.from_numpy(scale)[None, :, None, None],
    torch.from_numpy(ori)[None, :, None])
patches2 = extract_patches_from_pyramid(imgt, lafs, PS=patch_size)[0, :, 0]

plot_images(patches2[idxs])

diff = np.abs(patches2.numpy() - patches)
print(diff.mean())
plot_images(diff[idxs])

[mihaidusmanu]

The change in results is super low (probably less than run-to-run delta during GV), so I'd stick with your Kornia implementation since it's faster!