Ref frame for extrinsics

[nikhil-nakhate]

Hi @dyfcalid , This one is related to the cam_representation question. Why are the extrinsics re computed? What are the original extrinsics with respect to? The following is the piece of code that I am referring to:

                cam_extrinsics = np.array(info_dict['extrinsic'])
                # Re-calculate extrinsic matrix based on ground coordinate
                R_vg = np.array([[0, 1, 0],
                                    [-1, 0, 0],
                                    [0, 0, 1]], dtype=float)
                R_gc = np.array([[1, 0, 0],
                                    [0, 0, 1],
                                    [0, -1, 0]], dtype=float)
                cam_extrinsics[:3, :3] = np.matmul(np.matmul(
                                            np.matmul(np.linalg.inv(R_vg), cam_extrinsics[:3, :3]),
                                                R_vg), R_gc)
                cam_extrinsics[0:2, 3] = 0.0

[nickle-fang]

same questions

[RicardLee]

Hi , there are three camera coord sys:

• Waymo (OpenLane) camera coord sys. x-front, y-left, z-up

• normal (aka. standard) camera coord sys widely used. x-right, y-down, z-front

• LaneNet (3D-LaneNet) camera coord sys. x-right, y-front, z-up

The transformation matrices in the code are as following:

• cam_representation: Waymo to normal
  https://github.com/OpenPerceptionX/OpenLane/blob/1e8e61ccb456232955cf999b8db2aa4023945d25/eval/LANE_evaluation/lane3d/eval_3D_lane.py#L333-L338
• R_gc: normal to LaneNet
  https://github.com/OpenPerceptionX/OpenLane/blob/1e8e61ccb456232955cf999b8db2aa4023945d25/eval/LANE_evaluation/lane3d/eval_3D_lane.py#L305-L307
• R_vg: LaneNet to Waymo
  https://github.com/OpenPerceptionX/OpenLane/blob/1e8e61ccb456232955cf999b8db2aa4023945d25/eval/LANE_evaluation/lane3d/eval_3D_lane.py#L302-L304
• original cam_extrinsics in JSON: Waymo camera to Waymo vehicle
• inv(R_vg): Waymo to LaneNet

Note that the 3D lane GT are annotated in the Waymo camera coord sys. With matrices above, the final cam_extrinsics can transform GT to road coord (which is right-down of the camera and actually has the same z height with Waymo vehicle coord rather than on the road, with setting x/y=0 in the last line of code).
https://github.com/OpenPerceptionX/OpenLane/blob/1e8e61ccb456232955cf999b8db2aa4023945d25/eval/LANE_evaluation/lane3d/eval_3D_lane.py#L308-L314

In conclusion, there are some redundancies in the process and one could try to use only R_vg and original cam_extrinsics to see if they could get the same results.

[presmann]

@RicardLee

• LaneNet (3D-LaneNet) camera coord sys. x-right, y-front, z-up

How can we hope to achieve the statement In conclusion, "try to use only R_vg and original cam_extrinsics to see if they could get the same results"? The original extrinsic matrix has an offset that is zeroed when creating the virtual ground.

[RicardLee]

@presmann Hi, the conclusion means that zeroed operation would be applied except for R_vg and cam_extrinsics.

[notabigfish]

Hi, after reading the above explanations, i'm still confused, sorry for this...

cam_extrinsics[:3, :3] = np.matmul(np.matmul(np.matmul(np.linalg.inv(R_vg), cam_extrinsics[:3, :3]), R_vg), R_gc)
In the formula above, there are 3 matmuls:

1. np.matmul(np.linalg.inv(R_vg), cam_extrinsics[:3, :3]): this one transforms waymo camera to lanenet, right?
2. np.matmul(xxx, R_vg): Does this one transform waymo camera to waymo vehicle? Why not just use origin cam extrinsics?
3. np.matmul(xxx, R_gc): i think i'm already wrong in step 2, so still confused about this step...

One more question, gt_cam_pitch = 0 means simply no pitch? or is this related to the 3 matmuls?

Thanks!

[RicardLee]

Hi, matrix multiplication should be applied from right to left.
Assume that Point A_1 in waymo camera coord:
1.A_2 = np.matmul( cam_representation, A_1): transform the waymo camera-coord to normal camera-coord.
2.A_3 = np.matmul( R_gc, A_2):transform the normal camera-coord to LaneNet camera-coord.
3.A_4 = np.matmul( R_vg, A_3):transform the LaneNet camera-coord to waymo camera-coord.
4.A_5 = np.matmul( cam_extrinsics[:3, :3], A_4):transform the waymo camera-coord to waymo vehicle-coord.
5.A_6 = np.matmul( np.linalg.inv(R_vg), A_5):transform the waymo vehicle-coord to LaneNet vehicle-coord.
Finally, we get the 3d-lane in LaneNet vehicle-coord (road coord).
As the conclusion says that there are some redundancies in the process.
The cam pitch is not used in the following process and it would be deprecated.

[whoakang]

thanks for your excellent work ! i have a few question about output:

1. Does the output 3d-lane is in LaneNet vehicle-coord (road coord) : x-right,y-front,z-up，and the origin point of z is not at road plane，but as same as Waymo vehicle coord。
2. If want to output 3d-lane in Waymo vehicle-coord at last predict, just use A_5 to calculate。
   waiting for your reply，thanks

[RicardLee]

@whoakang hi，

1. Waymo vehicle coord is as following: x-front,y-left,z-up. It's different from LaneNet vehicle-coord (road coord).
   

2. the 3d-lane in LaneNet vehicle-coord (road coord) is the last output so that the A_6 is necessary.

[whoakang]

thanks for your reply.

you mean the origin point of road coordinate system and Waymo vehicle coord is same?

[RicardLee]

@whoakang hi, after setting x/y=0 , the origin point of the road coordinate and waymo vehicle is different.