Homography between the ground plane and the camera plane

[aloukkal]

First of all thank you for your amazing work. I am looking for a way to have the homography between the ground plane and the camera plane to warp the image from the camera in the bird eye view plan and reversely.
I would like to know if I can have this homography directly or if it is possible to put a calibration pattern like a chessboard on the road ?

Thanks.

[marcgpuig]

Hi @aloukkal, glad you are working with CARLA :)

We don't provide this feature currently.

• If what you want is put another camera from a top view, you can do it, but it will be attached to the car (position and rotation). As you have the camera information, I think you can make certain adjustments to correct the top camera rotation output.

• If what you whant is change the road texture, this can be easly done through the Unreal Engine editor. If so, remember to disable antialiasing so the chessboard patern will be more clear.

• Anyway remember that you have the FOV and the intrinsic and extrinsic matrix for each camera. Maybe you can use only that if what you want is "calibrate" the cameras.

Maybe you can give more information of what are you trying to achieve.

Regards!

[aloukkal]

@marcgpuig Thank you for your quick answer. My objective is to have the camera intrinsic matrix ( As I have access to the camera FOV, this is ok) , the rotation and translation matrix to do the IPM Inverse perspective mapping between the bird eye view and the camera plane:

http://answers.opencv.org/question/174548/inverse-perspective-mapping-ipm-on-the-capture-from-camera-feed/

One way to have the homography between the ground plane and the camera plane is to put a chessboard on the ground and use some opencv feature to extract the matrix. I am not familiar with unreal engine editor but I will give it a try, thanks.

I don't want to have directly the bird eye view image, I need to know the transformation between the bird eye view plane and the camera plane.

Regards.

[marcgpuig]

Thanks @aloukkal now I understand it better!

Think that the road is a perfect plane that extends through the X and Y axis (on Z = 3811.0 cm), and you know the camera position and rotation regarding the car, and the car position and rotation regarding the world on each frame.

Waiting for good news :)

[aloukkal]

Hi @marcgpuig,

I have been trying without success to do the inverse mapping. I will explain in more details maybe you could help.
I am trying to do it by defining myself the rotation and translation matrix:

yaw = 0
pitch = 0
roll = 0
w, h = 600, 300 
A = np.matrix([
    [1,0,-w/2],
    [0,1,-h/2],
    [0,0,0],
    [0,0,1]])

R_y = np.matrix([
    [1,0,0,0],
    [0, np.cos(yaw), -np.sin(yaw), 0],
    [0, np.sin(yaw), np.cos(yaw), 0],
    [0,0,0,1]])

R_p = np.matrix([
    [np.cos(pitch), 0, -np.sin(pitch),0],
    [0,1,0,0],
    [np.sin(pitch), 0, np.cos(pitch),0],
    [0,0,0,1]])
R_r = np.matrix([
    [np.cos(roll), -np.sin(roll),0,0],
    [np.sin(roll), np.cos(roll),0,0],
    [0,0,1,0],
    [0,0,0,1]])

R = R_y * R_p* R_r

T =np.matrix([
    [1,0,0,0],
    [0,1,0,0],
    [0,0,1,-1.50],
    [0,0,0,1]])
f= w/(2 * np.tan(90 * np.pi / 360))
K =np.matrix([
    [f, 0, w/2, 0],
    [0, f, h/2,0],
    [0, 0, 1, 0],])

H = K * (T * (R * A))

orig = cv2.imread ('XXX.png')
warp = cv2.warpPerspective(orig, H, (600, 300))

My camera is defined this way:

 camera0 = Camera('CameraRGB')
 camera0.set_image_size(1024, 512)
 camera0.set_position(100, 0, 150

Your help would be much appreciated, thanks

[marcgpuig]

Hi @aloukkal,

Did you took a look at point_cloud_example.py example?

Outside of your game loop you can do:

# Compute the camera transform matrix
camera_to_car_transform = camera0.get_unreal_transform()

(Use get_unreal_transform() to get the matrix with the Unreal necessary corrections applied)

Now, inside the game loop:

# (Camera) local 3d to world
# Get the transform from the player protobuf transformation
world_transform = Transform(measurements.player_measurements.transform)

# Compute the final transformation matrix
car_to_world_transform = world_transform * camera_to_car_transform

I think that car_to_world_transform contains the transformation matrix that you need.

[aloukkal]

@marcgpuig Hi,

Thank you I have just download the 0.8 version of carla, thanks again for your amazing work (LIDAR !!!). I have tried saving the car_to_world_transform matrix as a numpy file but I can't load it with error "No module named 'carla' ". Can you please tell me how to save and load this transformation matrix ?

I have succeeded in doing the warping by taking 4 points on the edges of the boardwalk, and using the depth of these four points and the fact that one pixel is 16cm in the map I have identified these four points on the map. Then with the findhomography feature of opencv I have obtained the matrix, it works but it's not perfect. Doing it with the unreal transform would be much better.

[marcgpuig]

Hey @aloukkal, that sounds really cool!

It look like you are saving the car_to_world_transform directly.
But this matrix it is actually a Carla module's Transform() instance, so it has dependencies.

You have 2 options here:

• Try saving car_to_world_transform.matrix that is the actual numpy.matrix.
• Include the Transform class from Carla module in your code.

Hope this helps.
I'm excited to see your results!

[gunshi]

Hi @aloukkal , were you able to get the transform and the BEV mapping?
I'm looking to do something similar so just wondering.