Bug report
Steps to reproduce
- Start the gateway with
server.executor_threads: 2 (the default).
- Set a short discovery refresh, e.g.
refresh_interval_ms: 1000, as the integration tests do.
- Issue an HTTP request that blocks on a ROS service future, such as cancelling an action execution.
- Observe how long the response takes while the periodic discovery pass is running.
Expected behavior
With two executor threads, a service response can be dispatched while another callback is running. server.executor_threads means what its name says.
Actual behavior
Dispatch is serialised regardless of the setting. Every ROS entity the gateway creates lands in the node default MutuallyExclusive callback group: the generic service clients, the action status subscription, and all five timers. A MutuallyExclusive group serialises its callbacks no matter how many executor threads exist, so the second thread is never usable and server.executor_threads is decorative.
Two consequences follow. An HTTP thread waiting on a service future can only be unblocked by that one group, and the periodic discovery refresh runs in the same group, so a full discovery pass can sit in front of a response somebody is waiting for. It also makes a class of timeouts indistinguishable from remote failures: a response that arrived but could not be dispatched looks exactly like one that never came.
Environment
- ros2_medkit version: 0.6.0 (main)
- ROS 2 distro: jazzy and humble
- OS: Ubuntu 24.04 / 22.04
Additional information
Found while tracing an action-cancel timeout that could not be reproduced under load. The measured cancel round trip is around a millisecond in every configuration tried, including two pinned cores with six competing processes, a 125-node churning graph, and CycloneDDS, while CI showed a five second stall. That gap points at dispatch, not at the wire.
Likely fix: give the blocking client work its own reentrant callback group so responses can be dispatched while other callbacks run.
Related: #576.
Bug report
Steps to reproduce
server.executor_threads: 2(the default).refresh_interval_ms: 1000, as the integration tests do.Expected behavior
With two executor threads, a service response can be dispatched while another callback is running.
server.executor_threadsmeans what its name says.Actual behavior
Dispatch is serialised regardless of the setting. Every ROS entity the gateway creates lands in the node default MutuallyExclusive callback group: the generic service clients, the action status subscription, and all five timers. A MutuallyExclusive group serialises its callbacks no matter how many executor threads exist, so the second thread is never usable and
server.executor_threadsis decorative.Two consequences follow. An HTTP thread waiting on a service future can only be unblocked by that one group, and the periodic discovery refresh runs in the same group, so a full discovery pass can sit in front of a response somebody is waiting for. It also makes a class of timeouts indistinguishable from remote failures: a response that arrived but could not be dispatched looks exactly like one that never came.
Environment
Additional information
Found while tracing an action-cancel timeout that could not be reproduced under load. The measured cancel round trip is around a millisecond in every configuration tried, including two pinned cores with six competing processes, a 125-node churning graph, and CycloneDDS, while CI showed a five second stall. That gap points at dispatch, not at the wire.
Likely fix: give the blocking client work its own reentrant callback group so responses can be dispatched while other callbacks run.
Related: #576.