Adding image segmentation based pointcloud filter for Autoware.Universe

[badai-nguyen]

Issue

Currently, we (TIER IV) and I believe that other teams also struggling with some cases of UNKNOWN detected objects from camera_lidar_fusion which are related to:

• Pointcloud from growing plants, bushes from road side such as described in : Obstacle Detection and Avoidance happening even if obstacle(trees) is not in path or drivable area. autoware.universe#5293 or https://github.com/orgs/autowarefoundation/discussions/3857
• Pointcloud from raindrop, splash from high speed vehicle on a wet road / exhausted gas
• Pointcloud of ground surface when ground_segmentation is fail because of lidar-lidar calibration error, some critical case like speed bump or rough road surface as ground remover package cannot delete ghost points appears front of car when passing speedbump autoware.universe#3673

Filtering such pointcloud based only on pointcloud geometry feature are challenge.

Possible solution

I belieave that understanding the pointcloud context information such as matching with image semantic/instance segmentation can help to understand and remove unnecessary pointcloud for autonomous driving.

So I am going to add an option of pointcloud filter based on image segmentation around camera_lidar_fusion pipeline.

For instance, and filtering based on drivable area and plant segmentation from image semantic segmentation result can refine the pointcloud after ground_segmentation and compare_map_filter:

The filter's accuracy will primarily depend on the perforamce of image segmentation model and the precision of aligning pointcloud with image plane.

In case of the filter is good enough, I think it can potentially replace the ground segmenation or compare map filter.

[cycyc1949]

That's a good idea. My idea is to filter the ground and detect objects, and then cluster the saved point clouds and project them onto the image to capture small image areas for classification.