How to use data from carla to do object detection

[ahsan155]

I am working on a project where I want to create a bridge between carla and autoware. I already have carla-ros bridge running. My end goal is to have sensor data being passed to autoware from carla and get driving instruction from autoware that will drive vehicle in carla.

I am using the dropdown menu in rviz to select carla/lidar topic in sensing/perception section of autoware. I also made changes in debug section to visualize camera data. I am not sure if using dropdown menu makes autoware subscribe to data being published from carla. As the first step of carla-autoware integration, I just want to do object detection. So what changes do I need to make in autoware packages to do object detection? or, do I need to call any other packages along with autoware_launch to do it?

[maxime-clem]

Rviz is only a visualization tool and it cannot be used to configure which topics are used by Autoware.
The easiest solution to your problem is to fix the topic name of the pointcloud published by your carla bridge to match the topic used by Autoware.

If you want, a new integration of CARLA with Autoware is being developed and should be added to the universe repository soon.
You can already use by checking out the branches of these PRs:

• feat(carla_autoware): add interface to easily use CARLA with Autoware autoware.universe#6859
• feat(e2e_simulator.launch): add argument for running the CARLA interface autoware_launch#924
  If you follow the instructions of the new autoware_carla package, then you should be able to test object detection.

[ahsan155]

so I am running lidar_centerpoint for object detection with lidar. This node is receiving lidar data from topic "/sensing/lidar/pointcloud" and it is publishing detection information to topic /objects. I found topic information in lidar_centerpoint.launch.xml file. I echoed message in /objects topic and output looks like this :

---

header:
stamp:
sec: 1716669738
nanosec: 437980450
frame_id: lidar_front
objects: []

header:
stamp:
sec: 1716669738
nanosec: 537925434
frame_id: lidar_front
objects: []

header:
stamp:
sec: 1716669738
nanosec: 637989740
frame_id: lidar_front
objects: []

Objects array is empty but there are many vehicles in scenario.
This is the output of running lidar_centerpoint:

I am using this following code to publish carla lidar data.
`class CarlaLidarPublisher(Node):
def init(self):
super().init('carla_lidar_publisher')
self.lidar_arr = []

    self.br = tf2_ros.StaticTransformBroadcaster(self)
    static_transform_msg = geometry_msgs.msg.TransformStamped()
    static_transform_msg.header.stamp = self.get_clock().now().to_msg()
    static_transform_msg.header.frame_id = 'base_link'
    static_transform_msg.child_frame_id = 'lidar_front'
    static_transform_msg.transform.translation.x = 0.0  # Replace with the appropriate translation values
    static_transform_msg.transform.translation.y = 0.0
    static_transform_msg.transform.translation.z = 2.5  # Assuming the LiDAR is at 2.5m height
    static_transform_msg.transform.rotation.x = 0.0  # Replace with the appropriate rotation values
    static_transform_msg.transform.rotation.y = 0.0
    static_transform_msg.transform.rotation.z = 0.0
    static_transform_msg.transform.rotation.w = 1.0
    self.br.sendTransform(static_transform_msg)

    # Initialize CARLA client
    self.client = carla.Client('localhost', 2000)
    self.client.set_timeout(10.0)
    self.world = self.client.get_world()

    # Set up the LiDAR sensor
    self.lidar_bp = self.world.get_blueprint_library().find('sensor.lidar.ray_cast')
    self.lidar_bp.set_attribute('range', '20')
    self.lidar_bp.set_attribute('rotation_frequency', '80')
    self.lidar_bp.set_attribute('channels', '32')
    self.lidar_bp.set_attribute('points_per_second', '320000')

    self.vehicle = self.world.get_actors().filter('vehicle.custom.ant')[0]  # Get the first vehicle actor
    transform = carla.Transform(carla.Location(x=0.0, z=2.5))
    self.lidar_sensor = self.world.spawn_actor(self.lidar_bp, transform, attach_to=self.vehicle)
    self.lidar_sensor.listen(self.lidar_callback)

    # Create a publisher for LiDAR data
    self.lidar_pub = self.create_publisher(PointCloud2, '/sensing/lidar/pointcloud', 10)

def lidar_callback(self, point_cloud):
    # Process the point cloud data
    points = np.frombuffer(point_cloud.raw_data, dtype=np.dtype('f4')).reshape(-1, 4)
    # Convert to PointCloud2
    msg = self.create_pointcloud2_msg(points)
    # Publish the PointCloud2 message
    self.lidar_pub.publish(msg)

def create_pointcloud2_msg(self, points):
    msg = PointCloud2()
    msg.header.stamp = self.get_clock().now().to_msg()
    msg.header.frame_id = 'lidar_front'

    fields = [
        PointField(name='x', offset=0, datatype=PointField.FLOAT32, count=1),
        PointField(name='y', offset=4, datatype=PointField.FLOAT32, count=1),
        PointField(name='z', offset=8, datatype=PointField.FLOAT32, count=1),
        PointField(name='intensity', offset=12, datatype=PointField.FLOAT32, count=1)
    ]

    msg.height = 1
    msg.width = len(points)
    msg.fields = fields
    msg.is_bigendian = False
    msg.point_step = 16  # 4 fields * 4 bytes each
    msg.row_step = msg.point_step * len(points)
    msg.is_dense = True
    msg.data = points.tobytes()

    return msg

def main(args=None):
rclpy.init(args=args)
node = CarlaLidarPublisher()
rclpy.spin(node)
node.destroy_node()
rclpy.shutdown()

if name == 'main':
main()`

what can be the reason for object not being detected? Is the lidar data format incorrect. can you suggest a very simple script to publish lidar data correctly.

[maxime-clem]

To see some working pointcloud data, you can check the rosbag from the log replay tutorial or you can run the AWSIM simulator.

[maxime-clem]

@ahsan155 were you able to solve your issue ?

[ahsan155]

Yes I was able to do it. thanks!