Couldn't project cuboids to 2D camera image.

[crackgfw]

Can you give one tutorial?
What is the relation between camera used and front camera, front left camera,..., back camera?

[xpchuan-95]

Have you ever tried this tutorial?
https://github.com/scaleapi/pandaset-devkit/blob/master/tutorials/projection/projection.ipynb

[crackgfw]

My code:
Is there any bug?

import pandaset
from pandaset import geometry
import numpy as np
import cv2

dataset = pandaset.DataSet("/data/PandaSet")

seq002 = dataset["002"]
seq002.load()
camera_keys = seq002.camera.keys()
print("Available cameras: ", camera_keys)

seq_idx = 0
cuboids0 = seq002.cuboids[seq_idx]
box = [cuboids0["position.x"][seq_idx], cuboids0["position.y"][seq_idx], cuboids0["position.z"][seq_idx],
cuboids0["dimensions.x"][seq_idx], cuboids0["dimensions.y"][seq_idx], cuboids0["dimensions.y"][seq_idx],
cuboids0["yaw"][seq_idx]]

corners = geometry.center_box_to_corners(box)

camera_used = cuboids0["camera_used"][seq_idx]
camera_name = "front_right_camera"
choosen_camera = seq002.camera[camera_name]

ori_image = seq002.camera[camera_name][seq_idx]
img = cv2.cvtColor(np.array(ori_image), cv2.COLOR_RGB2BGR)

projected_points2d, camera_points_3d, inner_indices = geometry.projection(lidar_points=corners,
camera_data=choosen_camera[seq_idx],
camera_pose=choosen_camera.poses[seq_idx],
camera_intrinsics=choosen_camera.intrinsics,
filter_outliers=False)

box1 = projected_points2d[(0, 3, 7, 4), :]
box2 = projected_points2d[(1, 2, 6, 5), :]
box1 = np.rint(box1).astype(np.int32)
box2 = np.rint(box2).astype(np.int32)
box1 = box1.reshape((-1, 1, 2))
box2 = box2.reshape((-1, 1, 2))
cv2.polylines(img, [box1], True, (0, 0, 255), 5)
cv2.polylines(img, [box2], True, (255, 0, 0), 5)

cv2.imshow('boxed', img)

cv2.waitKey()
cv2.destroyAllWindows()

[xpchuan-95]

The box you try to show on the image is not in front of the camera, it's out of field of vision, maybe at back of the camera.

[crackgfw]

I have tried every camera. It doesn't work well

[xpchuan-95]

I have tried every camera. It doesn't work well

There is a mistake in visualization code,
cuboids0["dimensions.x"][seq_idx], cuboids0["dimensions.y"][seq_idx], cuboids0["dimensions.y"] ->
cuboids0["dimensions.x"][seq_idx], cuboids0["dimensions.y"][seq_idx], cuboids0["dimensions.z"]

[crackgfw]

Thanks a lot.