Flaky services performance

[hidmic]

Bug report

Required Info:

• Operating System:
  • Windows 10, Ubuntu 20.04, OSX
• Installation type:
  • Binaries and form source
• Version or commit hash:
  • Foxy
• DDS implementation:
  • Fast-RTPS most often, but also with CycloneDDS
• Client library (if applicable):
  • rclcpp

Steps to reproduce issue

Any service / client pair will do, e.g.:

ros2 run demo_nodes_cpp add_two_ints_server

and

ros2 run demo_nodes_cpp add_two_ints_client

Expected behavior

Client sends request, servers gets request then sends response, and client gets response.

Actual behavior

The client will hang some times and not get any result. How often that happens seems to vary on a platform and RMW implementation.

Additional information

Issues with services performance has been observed before, as early as ros2/demos#304. More recent occurrences may be found in ros2/rmw_cyclonedds#185, ros2/system_tests#420 and ros2/rmw_cyclonedds#74. Nondeterministic as it is, it has seemed resolved in the past, suggesting race conditions as root cause.

Parameter servers as well as action servers and clients relying on services for their implementation are also likely to be negatively affected by this issue.

[hidmic]

CC @jacobperron

[hidmic]

@MiguelCompany @richiprosima it'd be great if a fix for rmw_fastrtps_cpp landed before Foxy gets released.

[MiguelCompany]

@hidmic I've just created ros2/rmw_fastrtps#390, which should fix the issue of sending a response when the reader has not yet been matched.

[sloretz]

Closing in favor of ros2/rmw_fastrtps#392 and ros2/rmw_cyclonedds#191 since these appear to be duplicates and the other two are on more specific repos.