feat!: add support of multiple object's shapes

[ktro2828]

Signed-off-by: ktro2828 kotaro.uetake@tier4.jp

Category

• Perception
• Sensing

What

Add support of evaluating the objects not only shape with box, but also polygon and cylinder.

After this PR, We need to specify shape paramter to initialize DynamicObject instead of size as below.
Only for POLYGON objects, footprint must be specified.
BOX or CYLINDER objects' footprint can be computed internally as a rectangle.

NOTE
Input footprint is assumed that it is with respect to each object's coordinate system.
Although polygon returned with DynamicObject.state.footprint is with respect to each object's coordinate system, polygon returned with DynamicObject.get_footprint() is with respect to base_link or map coordinate system.

we support following shapes.

class ShapeType(Enum):
    BOUNDING_BOX = "bounding_box"
    # CYLINDER = "cylinder" -> removed at https://github.com/tier4/autoware_perception_evaluation/pull/7/commits/f957d2e5f0b55f280c0365ac3318203264527e65
    POLYGON = "polygon"

from perception_eval.common.object import DynamicObject
from perception_eval.common.shape import Shape, ShapeType

shape_type: Union[str, ShapeType] = ShapeType.BOUNDING_BOX  # or "bounding_box" is also OK.
size: Tuple[float, float, float]
footprint: Optional[Polygon] = None # For BOUNDING_BOX None is OK. For POLYGON it should be specified.
 
shape = Shape(shape_type=shape_type, size=size, footprint=footprint)

dynamic_object = DynamicObject(
    ...
    shape=shape,
    ...
)

# We can access "size" as before
object_size: Tuple[float, float, float] = dynamic_object.state.size

# This is with respect to each object's coordinate system
footprint: Polygon = dynamic_object.state.footprint

# This is with respect to base_link or map coordinate system
footprint: Polygon = dynamic_object.get_footprint()

Type of change

• New feature
• Update document

Test performed

• test.sensing_lsim

• test.perception_lsim

Reference

Notes for reviewer

[miursh]

I have forgotten it because it's been too long, but I was going to leave a comment.

[ktro2828]

@miursh Cylinder has been removed at f957d2e

[ktro2828]

NOTE

In this PR, when estimation has unknown label it will be allowed to be matched

autoware_perception_evaluation/perception_eval/perception_eval/evaluation/result/object_result.py

Lines 419 to 421 in f47b36b

	if (
	estimated_object_.semantic_label == ground_truth_object_.semantic_label
	or estimated_object_.semantic_label.label == AutowareLabel.UNKNOWN

Also, object_result.is_label_correct always returns True if label of estimation is unknown

autoware_perception_evaluation/perception_eval/perception_eval/evaluation/result/object_result.py

Lines 216 to 228 in f47b36b

	def is_label_correct(self) -> bool:
	"""Get whether label is correct.
	Returns:
	bool: Whether label is correct
	"""
	if self.ground_truth_object:
	return (
	self.estimated_object.semantic_label == self.ground_truth_object.semantic_label
	or self.estimated_object.semantic_label.label == AutowareLabel.UNKNOWN
	)
	else:
	return False

[miursh]

@ktro2828

Also, object_result.is_label_correct always returns True if label of estimation is unknown

Is this always used if label is unknown? or is there some config/option to use this?

[ktro2828]

@miursh Sorry, I made a mistake, and I have updated like following.

1. Allows to match when either Est or GT has unknown label
   

   autoware_perception_evaluation/perception_eval/perception_eval/evaluation/result/object_result.py

   Lines 415 to 420 in f01f350

   	if (
   	est_obj.semantic_label == gt_obj.semantic_label
   	or any(
   	label == CommonLabel.UNKNOWN
   	for label in (est_obj.semantic_label.label, gt_obj.semantic_label.label)
   	)

2. DynamicObjectResult.is_result_correct() only checks matching score except of classification task
   

   autoware_perception_evaluation/perception_eval/perception_eval/evaluation/result/object_result.py

   Lines 97 to 123 in f01f350

   	def is_result_correct(
   	self,
   	matching_mode: MatchingMode,
   	matching_threshold: float,
   	) -> bool:
   	"""The function judging whether the result is target or not.
   	Args:
   	matching_mode (MatchingMode):
   	The matching mode to evaluate.
   	matching_threshold (float):
   	The matching threshold to evaluate.
   	For example, if matching_mode = IOU3d and matching_threshold = 0.5,
   	and IoU of the object is higher than "matching_threshold",
   	this function appends to return objects.
   	Returns:
   	bool: If label is correct and satisfy matching threshold, return True
   	"""
   	# Whether is matching to ground truth
   	matching: Optional[MatchingMethod] = self.get_matching(matching_mode)
   	return (
   	self.is_label_correct
   	if matching is None
   	else matching.is_better_than(matching_threshold)
   	)

for classification, returns is_label_correct result. It checks whether label is same, which means it is not allowed to match unknown and the other labels

autoware_perception_evaluation/perception_eval/perception_eval/evaluation/result/object_result.py

Lines 215 to 224 in f01f350

	def is_label_correct(self) -> bool:
	"""Get whether label is correct.
	Returns:
	bool: Whether label is correct
	"""
	if self.ground_truth_object:
	return self.estimated_object.semantic_label == self.ground_truth_object.semantic_label
	else:
	return False

[miursh]

@ktro2828
For current implementation,

autoware_perception_evaluation/perception_eval/perception_eval/evaluation/result/object_result.py

Lines 415 to 420 in f01f350

	if (
	est_obj.semantic_label == gt_obj.semantic_label
	or any(
	label == CommonLabel.UNKNOWN
	for label in (est_obj.semantic_label.label, gt_obj.semantic_label.label)
	)

if the autoware output label is "unknown", it always run matching, isn't it?
I think V&V team requirement is that they want to evaluate both with and without "unknown" class by one scenario.

[ktro2828]

@miursh @hayato-m126 in 6003414

I added a LabelParam and it has a parameter named allow_matching_unknonw. This is a flag whether allow to match unknown and the other labels.

from perception_eval.common.label import LabelParam

# === Perception ===
perception_label_param = LabelParam(
            label_prefix="autoware",  # Prefix of label name ... ("autoware", "traffic_light")
            merge_similar_labels=False,  # A flag if merge similar labels ... e.g. bus -> car
            allow_matching_unknown=True,  # A flag if allow to matching unknown and the other labels
            count_label_number=True,  # A flag if count the number of each label as debug
)

perception_cfg = PerceptionEvaluationConfig(
    ...
    label_param=perception_label_param,
    ...
)

# === Sensing ===
sensing_label_param = LabelParam(
            label_prefix="autoware",  # Prefix of label name ... ("autoware", "traffic_light")
            count_label_number=True,  # A flag if count the number of each label as debug
)

sensing_cfg = SensingEvaluationConfig(
    ...
    label_param=sensing_label_param,
    ...
)

• TODO: update documents and docstrings

[hayato-m126]

The PR for driving_log_replayer corresponding to this PR is made below.
tier4/driving_log_replayer#198

I will review it as soon as I can confirm that it works.

[ktro2828]

@hayato-m126 I'm sorry for that I've removed LabelParam and make it possible to specify label parameters in evaluation_config_dict like following, because I think this is easier for users to initialize PerceptionEvaluationConfig 🙏

evaluation_config_dict = {
            ....
            # label parameters
            "label_prefix": "autoware",
            "merge_similar_labels": False,
            "allow_matching_unknown": True,
           ....
}


evaluation_config = PerceptionEvaluationConfig(
            ...
            evaluation_config_dict=evaluation_config_dict, # <- label_param has been removed
            load_raw_data=True,
)

[ktro2828]

@hayato-m126 I've merged develop, so could you check this with driving_log_replayer?

[hayato-m126]

Thank you for updating this branch. I'm going to check.

[hayato-m126]

LGTM

This PR works fine with tier4/driving_log_replayer#198