issues with cam1 pose

[yshen47]

Hello KITTI360 team, first thanks a lot for sharing this dataset with the community! During development, we are confused about the following three issues. Let me introduce notation first, system_to_world refers to the per-frame system poses from seq/poses.txt, cam0_to_world refers to the per-frame cam0 pose extracted from cam0_to_world.txt, cam0_to_system refers to the transformation extracted from calib_cam_to_pose.txt, "cam0_rectified_R" is rotation rectification matrix for cam0, similarly for cam1_rectified_R

1. For the poses data (cam0_to_world.txt), is that from rectified or unrectified camera coordinate to world coordinate? For the cam0_to_system, is that from rectified or unrectified camera coordate to system coordinate system? The reason I asked this is that we found system_to_world.dot(cam0_to_system) = cam0_to_world.dot(cam0_rectified_R) seems always hold for sequence00
2. How to generate cam1_to_world? Based on q1, it seems that it should be
   cam1_to_world = np.matmul(system_to_world, cam1_to_system).dot(R_rect_01)
3. I am running a sanity check for stereo camera by naively extracting color from image for per-frame velo point cloud data. I project lidar point cloud onto the pixel coordinate of cam0 and cam1, then extract color for point cloud based on the mapping from 2d image. I checked by using the velo lidar, cam0 and cam1 images for the first frame in the seq00. I find the cam0 is projected correctly, however, cam1 seems off, i am not sure if I am missing any transformation.
   Here are the visualizations:

The 2d cam0 image for the first frame of sequence00:

Lidar pcd coloring based on cam0:

Lidar pcd coloring based on cam1:

If you look at the small sign post in the middle right, my lidar pcd coloring based on cam0 seems got the correct white color, but for cam1 it is way off

In specific, i used the following transformation to project velo lidar to camera coordinate for cam0 and cam1,
lidar_points_in_cam0_pixel_coord = P_rect_00.dot(R_rect_00.dot(Rt_00.dot(np.linalg.inv(cam0_to_velo).dot(lidar_points_homo.transpose()))))

lidar_points_in_cam1_pixel_coord = P_rect_01.dot(R_rect_01.dot(Rt_01.dot(np.linalg.inv(cam0_to_velo).dot(lidar_points_homo.transpose()))))
, where P_rect_00, P_rect_01, RT_00, RT_01, R_rect_00, R_rect_01 are extracted from perspective.txt

`R_rect_00 = np.array([[0.999974, -0.007141,-0.000089, 0],
[0.007141, 0.999969, -0.003247, 0],
[0.000112, 0.003247, 0.999995, 0],
[0, 0, 0, 1]
])

Rt_00 = np.array([[1, 0, 0, 0],
[0, 1, 0, 0],
[0, 0, 1, 0],
[0, 0, 0, 1]])

P_rect_00 = np.array([[552.554261, 0.000000, 682.049453, 0.000000],
[0.000000, 552.554261, 238.769549, 0.000000],
[0.000000, 0.000000, 1.000000, 0.000000],
[0, 0, 0, 1]])

R_rect_01 = np.array([[0.999778, -0.012115, 0.017222, 0],
[0.012059, 0.999922, 0.003351, 0],
[-0.017261, -0.003143, 0.999846, 0],
[0,0,0,1]])

P_rect_01 = np.array([[552.554261, 0.000000, 682.049453, -328.318735],
[0.000000, 552.554261, 238.769549, 0.000000],
[0.000000, 0.000000, 1.000000, 0.000000],
[0, 0, 0, 1]])

Rt_01 = np.array([[0.999837, 0.004862, -0.017390, -0.594052],
[-0.004974, 0.999967, -0.006389, 0.007198],
[0.017358, 0.006474, 0.999828, -0.010233],
[0, 0, 0, 1]])`

Thanks a lot for your help! we actually have been stuck at this transformation for a bit long time now...