Evaluate current planning pipeline

[mitsudome-r]

Checklist

• I've read the contribution guidelines.
• I've searched other issues and no duplicate issues were found.
• I've agreed with the maintainers that I can plan this task.

Description

Investigate performance of planning pipeline.

Purpose

Evaluate current planning pipeline and confirm that it has enough features for Bus ODD.

Possible approaches

Test planning pipeline against scenarios created by ODD WG.

Definition of done

• Get the list of scenario files from ODD WG.
• Run scenario tests manually against the planning pipeline
• List failed tests and investigate the cause of failure.
• Create issues for missing features or bugs found from failed tests.

[mehmetdogru]

Some of the .yaml scenario files are created by ODD wg but testing them using scenario_test_runner with autoware.universe is not possible at the moment because of the current state of autoware.universe.

[mehmetdogru]

Here are the notes about the features I have tested so far using planning_simulator:

• Stop Sign: Tested in both Kashiwa and a custom map. For this feature to work hd-map must have an annotated stop_line and a stop_sign on the lane. Stops at stop_line while the vehicle's front bumper is on the stop_line. Waits as much as the time given as parameter. Then moves on.

• Croswalk: Tested in both Kashiwa and a custom map. For this feature to work hd-map must have an annotated crosswalk on the lane. Then it satisfies the conditions as explained here. I have replicated those conditions using 2D Dummy pedestrian rviz tool while it was standing still and walking around/on the crosswalk. For vehicle to have a specific behavior for the crosswalk, any points of the pedestrian or any points of its predicted path should be in the annotated crosswalk polygon.

• Swerve: Obstacle avoidance feature works quite well. I used 2D dummy cars on several locations of the lanes and observed the ego vehicle behavior. Works as it is proposed.

• Lane Follow: Module works as expected. Tested in both Kashiwa and a custom map. When there is a car(2D dummy car) in front of the ego vehicle, ego vehicle is able to apply ACC successfully.

• Traffic Light: When I test this module in Kashiwa map, ego vehicle stops at the red light and moves on at the green light. However in a custom map expected traffic light behavior has almost never been observed. I will create an issue about it and give more details about this problem. Edit: fixed by fix(behavior_velocity_planner): fix velocity interpolation changing the stop position inappropriately #623

• A bug: behavior_path_planner can't plan a path with the first given goal command #636 Edit: fixed by fix(behavior_path_planner): initialize scene module status #650

• Lane Change: Tried this module on a custom map created wrt this doc and made sure that neighbor lane is available. However lane_change module was never registered even though there is a car in front on the ego vehicle's lane moving forward with a slower speed. Further investigation is needed.

[mehmetdogru]

Here will be the results and findings of simulation results of ODD scenarios defined by ODD WG and will be updated when new yaml files are added:

https://docs.google.com/spreadsheets/d/1uWpaJw68MwdCS_wysQsbG8YyG9UaYK8V00AFTd7HiVs/edit?usp=sharing

Also for the issues created see Issues sheet in the spreadsheet above.

[sglee-morai]

For the scenario UC-PB-005-0001-Hsinchu.yaml, which I originally created, I see the note is written like below,

• Vehicle doesn't stay on the stop line
• Vehicle sometimes can't make the right turn because of the narrow drivable area and gets stuck on the right turn. Fails with timeout"

The problem is that there is NO explicit stop line in the initial lane where the ego-vehicle is positioned initially. So please let me know if the scenario file itself should be modified.

Regardless of a stop line, the ego should stop anyway before entering any lane that is intersecting to the current lane that the ego is driving on, but I wonder what's your intention on the note whether to edit the scenario at the first place, so please let me know :)

[mehmetdogru]

@sglee-morai

The reason why I wrote this was that in ODD Design Document, initial case is defined as "EGO is stopping at the stop line" but the vehicle is not stopped on the stop line and it should be written :) You are right about that if the vehicle is already stopped before the turn, there is no difference between whether there is a stop line or not. It would matter tho if the initial vehicle speed was not zero, however no difference in this case.

There are no stop lines available on the lanelet2_map which would satisfy this (UC-PB-005-0001) scenario design so there is nothing to be done on your end regarding the scenario content. Therefore I edited Related Part column for the scenario file as lanelet2_map.

Thank you for the attention and the fine work.

[sglee-morai]

@mehmetdogru Thanks a lot Mehmet for your confirmation!

@sglee-morai

The reason why I wrote this was that in ODD Design Document, initial case is defined as "EGO is stopping at the stop line" but the vehicle is not stopped on the stop line and it should be written :) You are right about that if the vehicle is already stopped before the turn, there is no difference between whether there is a stop line or not. It would matter tho if the initial vehicle speed was not zero, however no difference in this case.

There are no stop lines available on the lanelet2_map which would satisfy this (UC-PB-005-0001) scenario design so there is nothing to be done on your end regarding the scenario content. Therefore I edited Related Part column for the scenario file as lanelet2_map.

Thank you for the attention and the fine work.

[mitsudome-r]

For UC-PB001-003, I have made an update to the scenario file. https://gitlab.com/autowarefoundation/operational-design-domains/-/merge_requests/26

[xmfcx]

For the initial testing, I think we can call this complete with the results in: #566 (comment)

If we need to evaluate it again, we can recreate another issue.