17.0.1
Description
Abstract
This pull request fixes a randomly occurring
Simulation error [AutowareError]: Simulator waited for the Autoware state to transition to WAITING_FOR_ROUTE, but time is up. The current Autoware state is INITIALIZING.
The root cause is that the spinner thread is created before std::atomic<bool> is_stop_requested = false; is initialized. This can cause is_stop_requested = true, preventing the spin from executing.
To fix this issue, in this PR, change the member order of concealer::AutowareUniverse class with consideration of initialization dependencies.
Background
To improve test reliability, we want to eliminate random failures. We'd like to run with retry 0.
Details
Currently, in the header https://github.com/tier4/scenario_simulator_v2/blob/ce2156518c7cb58374699c1dc9f68539a80c51ab/external/concealer/include/concealer/autoware_universe.hpp#L87 std::atomic<bool> is_stop_requested = false; is initialized.
Then, in the initializer list at https://github.com/tier4/scenario_simulator_v2/blob/ce2156518c7cb58374699c1dc9f68539a80c51ab/external/concealer/src/autoware_universe.cpp#L163, the spinner thread is started:
spinner(std::thread([this]() {
However, at this point, sometimes initialization hasn't finished. If is_stop_requested is true before initilization, spin_some will not execute, preventing the necessary topics from being published. Specifically, without the /localization/kinematic_state topic being published, Autoware cannot initialize (it can't generate a route) and remains stuck in the INITIALIZING state. ( route_state is neccesary for transition of LegacyAutowareState https://github.com/tier4/scenario_simulator_v2/blob/ce2156518c7cb58374699c1dc9f68539a80c51ab/external/concealer/include/concealer/legacy_autoware_state.hpp#L92-L96).
References
Here is debug result.
fail scenanario with error of
Simulation error [AutowareError]: Simulator waited for the Autoware state to transition to WAITING_FOR_ROUTE, but time is up. The current Autoware state is INITIALIZING
[simple_sensor_simulator_node-2] [AutowareUniverse] Initial is_stop_requested = 1
pass scenario
[simple_sensor_simulator_node-2] [AutowareUniverse] Initial is_stop_requested = 0
PR test results.
main
all results have the error
- https://evaluation.tier4.jp/evaluation/reports/54d2c929-6bc8-5053-9616-ed3570d365cb/tests/bf5b8f8f-abfd-50db-bd9b-cdb19525b6ff/6e4d226d-4144-5c57-ae03-907b76039fa3/059726e4-5bab-54d8-9971-2b685ed1f550?failure_cause_labels=SimulationError%3AAutowareError&project_id=prd_jt
- https://evaluation.tier4.jp/evaluation/reports/ea02e32f-0855-5aba-bb91-79abaae36804/tests/592c3cd5-0ad9-59ed-be40-7310899aa099/059156de-ecdd-5eb1-94ef-5806298ab9c8/028dd66f-ac20-5fc4-a749-2c8c0a058e58?failure_cause_labels=SimulationError%3AAutowareError&project_id=prd_jt&state=failed
- https://evaluation.tier4.jp/evaluation/reports/31f5ee53-cc81-5e38-8771-e28503b2b218/tests/f3cf7b46-5800-53c3-ab0d-4508a9b9c778/62c3816b-1acc-558d-86f8-60cf87de2569/040c1fb7-3451-580b-9aa2-b4cf5f0823db?failure_cause_labels=SimulationError%3AAutowareError&project_id=prd_jt&state=failed
PR
no results have the error
- https://evaluation.tier4.jp/evaluation/reports/6e933890-b1f4-5d6b-b63b-6f5d5c2712a4?project_id=prd_jt
- https://evaluation.tier4.jp/evaluation/reports/90bd3f03-fe13-5723-9c2e-fbbada0fd827?project_id=prd_jt