Meaning of the Homography in HomographyWarper

[stefat77]

Hello,
I'm doing some tests with the HomographyWarper function to find out how it uses the homography to warp the tensor.
In the following code I'm trying to track a point in a image. Both the image and the point are warped using the same homography. The warped point must be mapped at the same location of the warped image:

#Load a sample image
img = io.imread("sintel.jpg")
img = img / 255.0

h = img.shape[0]
w = img.shape[1]

H = torch.tensor([[1.0, 0.0, 0.5],
                  [0.0, 1.0, 0.0],
                  [0.0, 0.0, 1.0]])

#Test point chosen in the sample image
point = torch.ones(3, 1)
point[0] = 66
point[1] = 372

#Normalize between [-1, 1] the point coordinates 
point[0] = ((point[0] * 2.0 / (h - 1)) - 1)
point[1] = ((point[1] * 2.0 / (w - 1)) - 1)

#Warp the point
point_ = torch.matmul(H, point)
point_ = point_ / point_[2]

#Normalize back the point coordinates
point_[0] = ((point_[0] + 1) * (h - 1)) / 2
point_[1] = ((point_[1] + 1) * (w - 1)) / 2

H = H.unsqueeze(0)
H = H.type(torch.float32)

img = img_np2tensor(img).type(torch.float32)

#Warp the image
warper = tgm.HomographyWarper(260, 615, normalized_coordinates=True)
warped = warper(img, H)

warped = img_tensor2np(warped)

In this case the image is translated to the left while the point is moved down, so the point moved in a different way with respect the image.
If I use this homography to move the point I match the image movement:

H_point = torch.tensor([[1.0, 0.0, 0.0],
                  [0.0, 1.0, -0.5],
                  [0.0, 0.0, 1.0]])

Why does this happen?
The normalized coordinates means that the top left corner of the image has indices [-1,-1], and the bottom right corner has indices [1,1], right?
It seems like in a point [v1, v2, 1] the first value (v1) describes the column index and the second (v2) describes the row index.
There's also something wrong with the sign of the homography values...

[edgarriba]

@stefat77 thanks to try kornia :D

I see one thing in your code example: when you normalize the sample point, what are your coordinates convention ? According to our convention, I suspect that you normalize in the other way around since we assume {x, y}

Check how we create the base grid: https://github.com/arraiyopensource/kornia/blob/master/kornia/utils/grid.py#L38

Hope this helps.

[stefat77]

@stefat77 thanks to try kornia :D

I see one thing in your code example: when you normalize the sample point, what are your coordinates convention ? According to our convention, I suspect that you normalize in the other way around since we assume {x, y}

Check how we create the base grid: https://github.com/arraiyopensource/kornia/blob/master/kornia/utils/grid.py#L38

Hope this helps.

@edgarriba Thank you for the reply!
I changed the convention, but I still have troubles.
I have also noticed that the top left corner should have [1,1] normalized coordinates, while the bottom right corner should have [-1,-1] (torch.grid_sample has the opposite convention).
I changed the normalization equations as follows:

img = io.imread("sintel.jpg")
img = img / 255.0

h = img.shape[0]
w = img.shape[1]

H = torch.tensor([[1.0, 0.3, 0.5],
                  [0.4, 1.0, -0.4],
                  [0.2, 0.2, 1.0]])

point = torch.ones(3, 1)
point[0] = 372 # j or x
point[1] = 66 # i or y
point[2] = 1

#Normalize between [-1, 1] the point coordinates 
point[0] = -((point[0] * 2.0) / (w - 1)) + 1
point[1] = -((point[1] * 2.0) / (h - 1)) + 1

#Warp the point
point_ = torch.matmul(H, point)
point_ = point_ / point_[2]

#Normalize back the point coordinates
point_[0] = (-point_[0] + 1) * (w - 1) / 2.0
point_[1] = (-point_[1] + 1) * (h - 1) / 2.0

H = H.unsqueeze(0)
H = H.type(torch.float32)

img = img_np2tensor(img).type(torch.float32)

warper = tgm.HomographyWarper(260, 615, normalized_coordinates=True)
warped = warper(img, H)

In this case with...

• {x,y} convention
• top left pixel having [1,1] coordinates
• bottom right pixel having [-1,-1] coordinates
  ...I almost obtain the right result. The warped point in mapped close to the right location but still not in the right place.

[edgarriba]

@stefat77 I think you are normalizing wrong. I compared it with kornia.normalize_pixel_coordinates and the sign you get is inverted.

[edgarriba]

another question, is this homography for normalized coordinates ?

[stefat77]

@stefat77 I think you are normalizing wrong. I compared it with kornia.normalize_pixel_coordinates and the sign you get is inverted.

@edgarriba thank you for your patience!
I wrote everything using kornia libraries:

def img_np2tensor(img):
    img = torch.from_numpy(img)
    img = img.unsqueeze(0)
    img = img.permute(0,3,1,2)
    return img

def img_tensor2np(img):
    img = img.permute(0,2,3,1)
    img = img.data.numpy()
    img = np.squeeze(img)
    return img

img = io.imread("sintel.jpg")
img = img / 255.0

h = img.shape[0]
w = img.shape[1]

H = torch.tensor([[1.0, 0.0, 0.4],
                  [0.0, 1.0, 0.0],
                  [0.0, 0.0, 1.0]])
H = H.unsqueeze(0)

point = torch.ones(1, 1, 2)
point[0, 0, 0] = 372 # j or X
point[0, 0, 1] = 66 # i or Y

not_warped = np.copy(img)
not_warped[int(point[0, 0, 1]), int(point[0, 0, 0]), :] = 1
io.imsave("not_warped.png", np.uint8(not_warped*255))

point_n = tgm.normalize_pixel_coordinates(point, h, w)
point_w = tgm.transform_points(H, point_n)
point_w = denormalize_pixel_coordinates(point_w, h, w)

img = img_np2tensor(img).type(torch.float32)
warper = tgm.HomographyWarper(h, w, normalized_coordinates=True)
warped = warper(img, H)
warped = img_tensor2np(warped)

warped[int(point_w[0, 0, 1]), int(point_w[0, 0, 0]), :] = 1
io.imsave("warped.png", np.uint8(warped*255))

But again, the point is not warped as the image.

another question, is this homography for normalized coordinates ?

It's just a homography I made up randomly. Why should the homography be specific for normalized coordinates?

[stefat77]

@edgarriba I solved the problem. I don't know why, but if I warp the image with the homography H and the point with the inverse homograpy H^-1 then the point and the image are warped in the same way.

Here's my code:

def img_np2tensor(img):
    img = torch.from_numpy(img)
    img = img.unsqueeze(0)
    img = img.permute(0,3,1,2)
    return img

def img_tensor2np(img):
    img = img.permute(0,2,3,1)
    img = img.data.numpy()
    img = np.squeeze(img)
    return img

def b_inv(b_mat):
    eye = b_mat.new_ones(b_mat.size(-1)).diag().expand_as(b_mat)
    b_inv, _ = torch.gesv(eye, b_mat)
    return b_inv

img = io.imread("sintel.jpg")
img = img / 255.0

h = img.shape[0]
w = img.shape[1]

H = torch.tensor([[1.0, -0.3, 0.4],
                  [0.2, 1.0, 0.0],
                  [0.2, 0.1, 0.8]])

H = H.unsqueeze(0)
Hp = b_inv(H)
print(H)
print(Hp)

point = torch.ones(1, 1, 2)
point[0, 0, 0] = 372 # j or X
point[0, 0, 1] = 66 # i or Y

not_warped = np.copy(img)
not_warped[int(point[0, 0, 1]), int(point[0, 0, 0]), :] = 1
io.imsave("not_warped.png", np.uint8(not_warped*255))

point_n = tgm.normalize_pixel_coordinates(point, h, w)
point_w = tgm.transform_points(Hp, point_n)
point_w = denormalize_pixel_coordinates(point_w, h, w)

img = img_np2tensor(img).type(torch.float32)
warper = tgm.HomographyWarper(h, w, normalized_coordinates=True)
warped = warper(img, H)
warped = img_tensor2np(warped)

warped[int(point_w[0, 0, 1]), int(point_w[0, 0, 0]), :] = 1
io.imsave("warped.png", np.uint8(warped*255))

[edgarriba]

@stefat77 oh, now I see what you want to check.

I played around with you code and I can confirm that for going from a -> b you need to pass the homography from b -> a. The reason is that since we want to generate patch b, we need to sample from a and for this we need to pass the inverse transform since we have to retrieve pixels in the other direction. I encourage you to go in deep to the homography warper code to fully understand. However, the documentation is wrong since the homography should from dst to src. I open a ticket to fix it.

In addition, I found that to pass homographies in pixel coordinates space we need to add an extra normalization step as I show in the example below.

import cv2
import kornia

import torch
import torch.nn.functional as F


img_a = torch.ones(1, 1, 60, 60)
img_a[..., 20:40, 20:40] = 0.5

h = img_a.shape[2]
w = img_a.shape[3]

b_H_a = torch.tensor([[
    [1.0, 0.0, 15.],
    [0.0, 1.0, 15.],
    [0.0, 0.0, 1.0]
]]) # in pixel

a_H_b = torch.inverse(b_H_a)
print(b_H_a)
print(a_H_b)

# define point in frame a and transform to b

point_a = torch.ones(1, 1, 2)
point_a[..., 0] = 21 # j or X
point_a[..., 1] = 11 # i or Y

point_b = kornia.transform_points(b_H_a, point_a)

# encode point a to img_a

img_a[..., int(point_a[..., 1]), int(point_a[..., 0])] = 0.75

# create warper in pixel space

# TODO: call normalize pixel coordinate inside warper 
#       when normalized_coordinates is False

warper = kornia.HomographyWarper(
    h, w, normalized_coordinates=False)

# NOTE: b -> a
# TODO: fix docs or call torch.inverse in case when want to
# keep this convention which seems more natural.

grid_pix = warper.warp_grid(a_H_b)
grid_norm = kornia.normalize_pixel_coordinates(grid_pix, h, w)

img_b = F.grid_sample(img_a, grid_norm)

assert img_a[..., int(point_a[..., 1]), int(point_a[..., 0])] == \
       img_b[..., int(point_b[..., 1]), int(point_b[..., 0])]

# write to disk

img_a_vis = kornia.tensor_to_image((img_a[0] * 255).byte())
img_b_vis = kornia.tensor_to_image((img_b[0] * 255).byte())

cv2.imwrite("img_a.png", img_a_vis)
cv2.imwrite("img_b.png", img_b_vis)

[edgarriba]

closing in favor of #159