Repository navigation
PCL Segmentation
Previous: Data Sources
The team decided that machine learning would be used for the classification of point clusters from Lidar data. In order to perform this classification, the waymo-od dataset needed to be segmented into clusters to make the training data useful. The project required two segmentation algorithms, for two purposes:
- Segment waymo-od dataset based on provided bounding boxes as training data
- Segment data collected from Panasonic lidar as application data.
The team created a createDataset class to iterate through a folder of data downloaded from the Waymo website, segment the pointclouds in each frame by the provided bounding boxes (see createDataset.py). To ease the understanding of the data, the team also created an inherited createDatasetVis class that published and subscribed to ROS topics, providing real-time visualization of the raw data and bounding boxes with rviz as it was parsed. Changing the /visualization rosparam allows introspection of each step of the segmentation process individually.
Waymo-od data is stored in tensorflow files called .tfrecords. The team primarily used Waymo's provided tools to unpack these files. One caveat is that the pointcloud intensity data was stored as range images, sorted independently of the pointclouds. Rather than attempt to remap the pointclouds to the range images, the team opted to create a custom version of Waymo's convert_range_image_to_pcl function, removing the image code and retaining the point-intensity mapping of the data (waymo file, custom file). Separately, the bounding boxes were extracted for future use. An example of the unpacked data visualized in Rviz is shown below.
A standard pre-processing step in autonomous vehicle literature is removing points representing the ground plane from the data. There are several ways to do this, two of which are manual thresholding and histogram analysis. The team opted for manual thresholding, as the urban environment allowed for a clean road surface and the origin of the vehicle's coordinate system was planar with the road. The threshold was set at 0.1m above the road surface to account for noise, and all points below this plane were removed, as seen in the image below.
As shown above, the thresholding removes most groundplane points. However, points that don't represent features but are still above the roadplane, such as the curb or noisy road points, are not removed. In future development, the team plans to revise this step to analyze the point density histogram along the z-axis, and threshold the pointcloud at the height of maximum density, as implemented by Velodyne in a paper titled LIDAR-based 3D Object Perception.
The most significant step in the pointcloud segmentation process was determining which points were within each bounding box. In order to minimize processing time, the operation was done primarily with numpy. An outline of the algorithm is as follows:
bbox = object bounding box
pcl = filtered pointcloud of frame
for each bbox:
project pcl into xy-plane
Create rotation matrix based on bbox heading
Rotate pcl into bbox frame of reference with rotation matrix
Threshold pcl by xy-coordinates of bbox corners
Re-attach z-coordinate and intensity data to xy-data
Return cluster of points
This operation was run for each bounding box in a frame. For frame with 60-80 bounding boxes, the process takes ~1s to complete for a given frame. A set of 5000 clusters can be generated in a couple of hours using an average of 3 processors. An example resulting frame can be seen below.
This image shows the downsampling done with the bounding boxes. As these bounding boxes are generated manually, there are some point-based discrepancies. Notably, the blue bounding box representing the sign has a large number of points in the previous two images, but very few actually contained points in the clustered image. Many of the points that clearly belong to the sign nonetheless lie outside of the bounding box. A future possible fix to this issue would be to define some padding distance around the bounding box within which points would still be counted as part of a cluster.
Another interesting feature of this method of clustering is that points that are directly below a bounding box are counted as part of the cluster. In the image above, the blue sign bounding box also captures several of the points in the signpost, as when the data is transposed into the xy-plane, these points lie within the bounding box.