Lidar points to image

[Sarxell]

Good night @miguelriemoliveira and @danifpdra ,

I am having a problem understanding how to pass the lidar points to the image. Right now, I found my extrinsic and intrinsic values, the intrinsic from the camera_info and the extrinsic from the urdf.

It gave me this values:

        self.cameraIntrinsic = np.array([[1060.9338813153618, 0.0, 640.5, -74.26537169207533],
                                        [0.0, 1060.9338813153618, 360.5, 0.0],
                                        [0.0, 0.0, 1.0, 0.0]])
        # camera extrinsic from the lidar to the camera (back and front have different extrinsic values )
        self.cameraExtrinsicFront = np.array([[1.0, 0.0, 0.0, -0.073],
                                             [0.0, 1.0, 0.0, 0.011],
                                             [0.0, 0.0, 1.0, -0.084],
                                             [0.0, 0.0, 0.0, 1.0]])

        self.cameraExtrinsicBack = np.array([[-1.0, 0.0, 0.0, 0.2],
                                            [0.0, -1.0, 0.0, 0.011],
                                            [0.0, 0.0, 1.0, -0.084],
                                            [0.0, 0.0, 0.0, 1.0]])

I also found the coordinates from the lidar, the way we made in class 10:

...
        z = 0
        for idx, range in enumerate(msg.ranges):
            theta = msg.angle_min + idx * msg.angle_increment
            x = range * math.cos(theta)
            y = range * math.sin(theta)
            self.points.append([x, y, z, 0])
....

When I multiply these values I get values in the order of the 2000, 2300 most of the time...
(my camera has a width of 1280 and height of 720)
I already tried thinking that it's from the middle and subtracting -1280/2 and -720/2 but I still got no results
I can also have the extrinsic matrix wrong but I don't know

Probably I'm thinking this wrong but I cannot figure it out

[miguelriemoliveira]

Hi @Sarxell ,

the way to project lidar 3d coordinates to a camera is as follows:

First you must transform the points from the lidar coordinate frame to the camera coordinate frame. For that you need to transformation from the camera to the lidar (cameraTlidar). You can get it using a

rosrun tf tf_echo ...

So if you do this:

P[camera] = cameraTlidar x P[lidar]

you get the points in the camera reference frame (notice the camera reference frame is the camera_rgb_optical_frame in ROS).

After that you can project to the image by using the intrinsic matrix K

[u,v,w]T = K x P[camera]

xpix = u/w
ypix = v/w

Some of lidar points may indeed be outside the image max width and height, because they are not visible in the image.

More info:
https://docs.opencv.org/3.4/d9/d0c/group__calib3d.html

and an example:
https://github.com/lardemua/atom/blob/noetic-devel/atom_evaluation/scripts/point_cloud_to_image.py

Hope it helps,
Miguel

[Sarxell]

So, the transformation is to the camera_rgb_optical_frame to the lidar? I was doing the matrix from the camera_link...
And I have a question
In here:
[u,v,w]T = K x P[camera]

This T is the cameraTlidar or is it another transformation?

[miguelriemoliveira]

On Mon, Feb 21, 2022, 15:28 SaraPombinho ***@***.***> wrote: So, the transformation is to the camera_rgb_optical_frame to the lidar? I was doing the matrix from the camera_link...

Yes the optical frame is the one with the z forward. And I have a question

In here: [u,v,w]T = K x P[camera] This T is the cameraTlidar or is it another transformation?

Here T means transposed ...

…

— Reply to this email directly, view it on GitHub <#67 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACWTHVTUXR2DQTHS2N4NIM3U4JK2VANCNFSM5O45NDYQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[Sarxell]

I forgot! I will try it that way

[Sarxell]

I changed the cameraTlidar to the correct frames and it worked! Thank you!

[miguelriemoliveira]

Great!

…

On Mon, Feb 21, 2022, 16:58 SaraPombinho ***@***.***> wrote: I changed the cameraTlidar to the correct frames and it worked! Thank you! — Reply to this email directly, view it on GitHub <#67 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACWTHVV4CEDDDPSSF5NWNJDU4JVLPANCNFSM5O45NDYQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>