17.2.1
Description
This PR contains a fix for the issue described here
Abstract
SimulatorCore::deactivate function is slow because it waits for the Autoware to shutdown.
StatusMonitor throws the health check error if Autoware takes too long.
The threshold is 10s by default, but there is overrideThreshold function that increases the threshold for one call.
There was one place, where the threshold was not increased for the SimulatorCore::deactivate function. Sometimes, Autoware takes more tn 10s to deactivate and this triggers the error.
Another issue is that the overrideThreshold function overrides the threshold only for the duration of the call. On exit, the original threshold is restored and it might already be exceeded. To mitigate this issue I decided to invoke touch upon entry and exit, which informs the watchdog that the process is responsive and resets the timer.
Details
Threshold for Autoware to shutdown was initialized by initialize_duration parameter, which in my opinion is misleading, so I created a separate constant with an explicit name - simulator_core_shutdown_threshold. I have decided to set its value to 30, because initialize_duration default value was 30s. However, Evaluator receives 90s during runs.
I believe 30s is more than enough, because during shutdown, Autoware receives SIGINT and then SIGTERM and then SIGKILL. SIGINT and SIGTERM have timeouts of 5 or 8 seconds, while SIGKILL should unconditionally terminate Autoware within 3s. Those values, when summed up, should be smaller than the simulator_core_shutdown_threshold.
Suspected cause
In openscenario_interpreter::Interpreter::reset(), function call SimulatorCore::deactivate took some time to complete, and then immediately openscenario_interpreter::record::stop was called, which takes at least 3 seconds. Wrapping SimulatorCore::deactivate (or even also openscenario_interpreter::record::stop) with common::status_monitor.overrideThreshold alone will not solve any issues as the timer between or after the calls might already exceed the threshold. Because of this, the implementation of common::StatusMonitor needs some altering.
References
Regression OK