Metric calculation node development

[mitsudome-r]

Description

(Copied issue from https://gitlab.com/autowarefoundation/autoware.auto/AutowareAuto/-/issues/1446)
To implement and upload to AutowreAuto repository sample ROS2 node for metric calculation. As a sample, the calculation of deviation from planned path should be calculated.

Purpose

To validate specific functionalities of AutowareAuto

Desired Behavior

Metric calculation node subscribes to ROS2 topics published by Autoware or Simulation pipeline and publishes value of calculated metric

Definition of Done

• Sample metric calculation node developed and uploaded to the AutowareAuto repository
• Instruction how to run, and how to develop other nodes for calculation of different metrics

[WJaworskiRobotec]

Similar feature related to trajectory evaluation : autowarefoundation/autoware.universe#288

[djargot]

Currently working on this issue.
I am reusing parts of the planning_evaluator and motion_evaluator and expanding it to calculate various metrics, like:

• average velocity
• calculation of localization errors: lateral, absolute deviations from the trajectory
• heading angle error
• distance to closest obstacle

The final goals are:

• to have a generic metric calculator that more specific metric calculators (planning, localization, object detection, possibly more?) could inherit from.
• to make it easy to add new metrics under different categories (planning, localization, object detection, etc.)

[djargot]

Recently added 2 separate evaluators (localization and kinematic), cleaned the code, and added basic unit tests.

I am currently trying to find the best way to find proper beginning and ending times between which metrics should be collected as it should happen only when a vehicle is driving.
Additionally, working on segmentation evaluator and extracting common parts of all evaluators to parent evaluator.

[djargot]

Added draft of ground detection error calculation from point clouds - calculating confusion matrix.
Also, tested localization evaluator with TierIV SIM.

[WJaworskiRobotec]

@djargot Have you added publishing of gt localization on the seperate topic (/tf_ground_truth or sth like that) to a branch in TierIV simulator ?

[WJaworskiRobotec]

Localization and kinematic evaluators are already merged to autoware.universe

localization and kinematic evaluator video

[djargot]

I created initial version of segmentation metrics (autowarefoundation/autoware.universe#1600) to calculate precision, recall, accuracy, and confusion matrix. Values are calculated both for a single point cloud as well as across whole scenario/ROS bag.

I tested metrics using labeled point cloud provided by MORAI (related issue: autowarefoundation/autoware.universe#669).

Example results for ground segmentation metrics for a single point cloud:

[stale]

This pull request has been automatically marked as stale because it has not had recent activity.

[mitsudome-r]

Close due to stale issue