Scene with known camera poses

[mertkiray]

Hello, thank you for this project :)

I got a problem while training on a scene.

When I extracted the camera poses from COLMAP this is the output

kitti_exp_2_spiral_100000_rgb.mp4

the result is not so crispy as the scene is a little bit hard to learn but

But when I use the camera poses from the dataset I got a result like this

kitti_exp_pose_1_spiral_025000_rgb.mp4

The difference between the two is really great. I changed the gt camera poses to the coordinate system of right, upwards, backwards.

What am I doing wrong?

Thanks.

[dunbar12138]

Hi, thanks for your attention to our work!

Unfortunately, I cannot give you a direct answer for lack of information. I would say using some tools to visualize the cameras will help a lot when you debug.

[mertkiray]

Hello @dunbar12138 ,
Thank you for your help.
I tried to visualize the rays with the package you suggested.

import pytransform3d.visualizer as p
import pytransform3d.transformations as pt

index_pose = i_train[0]
rays_o, rays_d = get_rays_by_coord_np(H, W, focal, poses[index_pose,:3,:4], depth_gts[index_pose]['coord'])
rays_o, rays_d = torch.Tensor(rays_o).to(device), torch.Tensor(rays_d).to(device)


pose = poses[index_pose, :, :]
pose_hom = np.eye(4)
pose_hom[:3, :4] = pose
pose = pose_hom

fig = pv.figure()

cam2world = pose
virtual_image_distance = 1
sensor_size = np.array([W, H])
 intrinsic = [
    [focal, 0, sensor_size[0] / 2], [0, focal, sensor_size[1] / 2], [0, 0, 1]
]
intrinsic_matrix = np.array(intrinsic)

fig.plot_transform(A2B=cam2world, s=0.2)
fig.plot_camera(
    cam2world=cam2world, M=intrinsic_matrix, sensor_size=sensor_size,
    virtual_image_distance=virtual_image_distance)

for i in range(rays_d.shape[0]):
    ray_o = rays_o[i].cpu()
    ray_d = rays_d[i].cpu()

    fig.plot_vector(start=ray_o,
        direction=ray_d,
        c=(1.0, 0.5, 0.0))

    P = np.zeros((2, 3))
    colors = np.empty((2, 3))
    P[0] = ray_o

    # ray_o + (depth_gts[index_pose]['depth'][0] * rays_d)
    P[1] = ray_o + (depth_gts[index_pose]['depth'][i] * -ray_d)
    colors[:, 0] = np.linspace(1, 0, len(colors))
    colors[:, 1] = np.linspace(0, 1, len(colors))
    fig.plot(P, colors)

pcd = np.load('colmap_pointcloud.npy')

pcd_colmap = open3d.geometry.PointCloud()
pcd_colmap.points = open3d.utility.Vector3dVector(pcd)
pcd_colmap.paint_uniform_color([1, 0, 0])

pcd_colmap.transform(pose)
 fig.add_geometry(pcd_colmap)

fig.view_init()
if "__file__" in globals():
    fig.show()
else:
    fig.save_image("__open3d_rendered_image.jpg")

But the resulting visualization is like this:

Shouldn't the rays intersect with the points as we go draw a line from camera in the ray direction with the depth value?

Also, the rays are pointing in the negative direction of the camera.

Is this correct?

When I try the above implementation on my own dataset with known poses and point cloud I get a similar output.

which I think is wrong.

Can you please tell me what am I doing wrong :)

[IncludeMathH]

I'm also interested in this problem, and I think it may have something to do with the t vector in the transform matrix. I would appreciate it if you could provide your own discovery or solution.