feat(perception)!: add support of perception FP validation

[ktro2828]

Category

• Perception
  • Detection

What

Related to #76
Add support of False Positive validation.

[New features]

• Add parameter named max_matchable_radii.

evaluation_config_dict = {
            "evaluation_task": "fp_validation",
            ....
            "max_matchable_radii": [5.0, 3.0, 3.0, 3.0],
        }

• Add new evaluation task and label for FP validation

# === Evaluation task ===
class EvaluationTask(Enum):
    ...
    # False Positive validation
    FP_VALIDATION = "fp_validation"
    FP_VALIDATION2D = "fp_validation2d"
    ...

# === Label ===
class AutowareLabel(Enum):
    ...
    # for FP validation
    FP = "false_positive"
    ...

class TrafficLightLabel(Enum):
    ...
    # for FP validation
    FP = "false_positive"
    ...

• MatchingStatus

class MatchingStatus(Enum):
    TP = "TP"
    FP = "FP"
    FN = "FN"
    TN = "TN"
    ...

• GroundTruthStatus.
  • This is class for keeping and calculating status information of each GT.
  • get_object_status(...) -> List[GroundTruthStatus] returns list of status info for each GT.

class GroundTruthStatus:
  """Class for keeping and calculating status information of each matching status for one GT.

  Attributes:
      uuid (str): UUID of ground truth object.
      total_frame_nums (List[int]): List of frame numbers, which GT is evaluated.
      tp_frame_nums (List[int]): List of frame numbers, which GT is evaluated as TP.
      fp_frame_nums (List[int]): List of frame numbers, which GT is evaluated as FP.
      tn_frame_nums (List[int]): List of frame numbers, which GT is evaluated as TN.
      fn_frame_nums (List[int]): List of frame numbers, which GT is evaluated as FN.

  Args:
      uuid (str): object uuid
  """

  def __init__(self, uuid: str) -> None:
      self.uuid: str = uuid

### === How to get TP/FP/TN/FN rates ===
from perception_eval.evaluation import get_status_info
  
  # REQUIRED: 
  #    frame_results (List[PerceptionFrameResult]
  
  status_info: List[GroundTruthStatus] = get_status_info(frame_results)
  
  for status in status_info:
      staus.uuid # uuid of GT
      status.total_frame_nums # List of total frame numbers GT was evaluated
      status.tp_frame_nums # List of TP frame numbers
      status.fp_frame_nums # List of FP frame numbers
      status.tn_frame_nums # List of TN frame numbers
      status.fn_frame_nums # List of FN frame numbers
      
      # Get TP/FP/TN/FN rates per frame
      tp_rate, fp_rate, tn_rate, fn_rate = status.get_status_rates()

Sample output by test/perception_fp_validataion_lsim.py

[2023-07-21 18:29:35,828] [INFO] [perception_fp_validation_lsim.py:93 display] 0 TP objects, 1 FP objects, 1 TN objects, 0 FN objects
[2023-07-21 18:29:35,830] [INFO] [perception_fp_validation_lsim.py:93 display] 0 TP objects, 1 FP objects, 1 TN objects, 0 FN objects
[2023-07-21 18:29:35,832] [INFO] [perception_fp_validation_lsim.py:93 display] 0 TP objects, 1 FP objects, 1 TN objects, 0 FN objects
[2023-07-21 18:29:35,834] [INFO] [perception_fp_validation_lsim.py:93 display] 0 TP objects, 1 FP objects, 1 TN objects, 0 FN objects
[2023-07-21 18:29:35,836] [INFO] [perception_fp_validation_lsim.py:93 display] 0 TP objects, 1 FP objects, 1 TN objects, 0 FN objects
[2023-07-21 18:29:35,839] [INFO] [perception_fp_validation_lsim.py:93 display] 0 TP objects, 2 FP objects, 1 TN objects, 0 FN objects
[2023-07-21 18:29:35,842] [INFO] [perception_fp_validation_lsim.py:93 display] 0 TP objects, 2 FP objects, 1 TN objects, 0 FN objects
[2023-07-21 18:29:35,846] [INFO] [perception_fp_validation_lsim.py:93 display] 0 TP objects, 2 FP objects, 1 TN objects, 0 FN objects
[2023-07-21 18:29:35,850] [INFO] [perception_fp_validation_lsim.py:93 display] 0 TP objects, 2 FP objects, 1 TN objects, 0 FN objects
[2023-07-21 18:29:35,854] [INFO] [perception_fp_validation_lsim.py:93 display] 0 TP objects, 2 FP objects, 1 TN objects, 0 FN objects
[2023-07-21 18:29:35,857] [INFO] [perception_fp_validation_lsim.py:93 display] 0 TP objects, 2 FP objects, 1 TN objects, 0 FN objects
[2023-07-21 18:29:35,857] [INFO] [perception_fp_validation_lsim.py:105 display_status_rates] uuid: cb1040491bbcf642078c00a7a75c95f9, TP: inf, FP: 0.200, TN: 0.800, FN: inf
Total: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19], TP: [], FP: [0, 1, 2, 3], TN: [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19], FN: []
[2023-07-21 18:29:35,857] [INFO] [perception_fp_validation_lsim.py:105 display_status_rates] uuid: 2b3dbc70e876f6829fb27301e27316f0, TP: inf, FP: inf, TN: 1.000, FN: inf
Total: [0, 1, 2, 3], TP: [], FP: [], TN: [0, 1, 2, 3], FN: []
[2023-07-21 18:29:35,857] [INFO] [perception_fp_validation_lsim.py:105 display_status_rates] uuid: 837029711ca9fbfd36f5dd7c59846c37, TP: inf, FP: 1.000, TN: inf, FN: inf
Total: [9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19], TP: [], FP: [9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19], TN: [], FN: []
[2023-07-21 18:29:35,857] [INFO] [perception_fp_validation_lsim.py:105 display_status_rates] uuid: d363a05be6f01b25e4d21794b320fcc0, TP: inf, FP: 1.000, TN: inf, FN: inf
Total: [14, 15, 16, 17, 18, 19], TP: [], FP: [14, 15, 16, 17, 18, 19], TN: [], FN: []

Type of change

• New feature
• Breaking change
• Update test
• Update document

[DEPRECATED]

The old one will be removed in next minor update -> 1.2.0

• evaluation/matching/objects_filter.py

Old	New	Description of new one
divide_tp_fp_objects(...)	get_positive_objects(...)	Returns TP/FP object results
get_fn_objects(...)	get_negative_objects(...)	Returns TN/FN objecrts
_is_fn_objects(...)

Test performed

• test.sensing_lsim

• test.perception_lsim

Reference

Notes for reviewer

[hayato-m126]

estimated_objects is array

[hayato-m126]

Since fp_validation2d is defined, don't we need an implementation example test/perceiotin_fp_validat_lsim2d.py?

[hayato-m126]

This PR works with tier4/driving_log_replayer#200