Flaky: failed to add guard condition to wait set: guard condition implementation is invalid

[moriarty]

Bug report

Internally we've notice an uptick in flaky tests and started keeping track of them.
These tests are in running in large github runners.
I just took a very brief look over the reported flakes to group them into buckets.

I saw that several different test failures all flaked with the same RCLError.

These tests are not all in the same internal packages in a I am opening this public issue and will try to reproduce in public MoveIt Code.

    terminate called after throwing an instance of 'rclcpp::exceptions::RCLError'
      what():  failed to add guard condition to wait set: guard condition implementation is invalid, at ./src/rcl/guard_condition.c:172, at ./src/rcl/wait.c:460

Required Info:

• Operating System:
  • Ubuntu 22.04
• Installation type:
  • Binaries from Snapshot repos
• Version or commit hash:
  • humble from a snapshot "2023-09-25"
• DDS implementation:
  • CycloneDDS
• Client library (if applicable):
  • rclcpp

Steps to reproduce issue

Todo(@moriarty)
Currently it's showing up in tests which have been marked as flaky.
I need to investigate further how flaky/often and try to create a way to reproduce the issue
I need to create a public self contained example to reproduce this issue.

I didn't see any other issues reporting the same error so I'll open this publicly and also assign myself an internal ticket to investigate further. If we're just using something incorrectly I will update and close this but it seems odd that it's flaky.

[moriarty]

FYI/CC: @tylerjw & @sea-bass

[moriarty]

    terminate called after throwing an instance of 'rclcpp::exceptions::RCLError'
      what():  failed to add guard condition to wait set: guard condition implementation is invalid, at ./src/rcl/guard_condition.c:172, at ./src/rcl/wait.c:460
    Stack trace (most recent call last) in thread 2868:
    #17   Object "", at 0xffffffffffffffff, in 
    #16   Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7f532cd91bf3, in __clone
    #15   Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7f532cd00ac2, in 
    #[14](https://github.com/PickNikRobotics/moveit_studio/actions/runs/6960359365/job/18940188796?pr=5438#step:9:15)   Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30", at 0x7f532cf92252, in 
    #13   Object "/opt/overlay_ws/build/moveit_studio_behavior/test/test_get_plan_using_tamp", at 0x55d0e50b3650, in std::thread::_State_impl<std::thread::_Invoker<std::tuple<moveit_studio::test_utils::RosExecutorTest::RosExecutorTest()::{lambda()#1}> > >::_M_run()
    #12   Object "/opt/ros/humble/lib/librclcpp.so", at 0x7f532d2eb771, in rclcpp::Executor::spin_once_impl(std::chrono::duration<long, std::ratio<1l, 1000000000l> >)
    #11   Object "/opt/ros/humble/lib/librclcpp.so", at 0x7f532d2ea242, in rclcpp::Executor::get_next_executable(rclcpp::AnyExecutable&, std::chrono::duration<long, std::ratio<1l, 1000000000l> >)
    #10   Object "/opt/ros/humble/lib/librclcpp.so", at 0x7f532d2e7524, in rclcpp::Executor::wait_for_work(std::chrono::duration<long, std::ratio<1l, 1000000000l> >)
    #9    Object "/opt/ros/humble/lib/librclcpp.so", at 0x7f532d2fa4[16](https://github.com/PickNikRobotics/moveit_studio/actions/runs/6960359365/job/18940188796?pr=5438#step:9:17), in 
    #8    Object "/opt/ros/humble/lib/librclcpp.so", at 0x7f532d2e7d3b, in rclcpp::detail::add_guard_condition_to_rcl_wait_set(rcl_wait_set_s&, rclcpp::GuardCondition const&)
    #7    Object "/opt/ros/humble/lib/librclcpp.so", at 0x7f532d2e3fc8, in rclcpp::exceptions::throw_from_rcl_error(int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rcutils_error_state_s const*, void (*)())
    #6    Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30", at 0x7f532cf641fd, in std::rethrow_exception(std::__exception_ptr::exception_ptr)
    #5    Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30", at 0x7f532cf64276, in std::terminate()
    #4    Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30", at 0x7f532cf64[20](https://github.com/PickNikRobotics/moveit_studio/actions/runs/6960359365/job/18940188796?pr=5438#step:9:21)b, in 
    #3    Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30", at 0x7f532cf58b9d, in 
    #2    Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7f532cc947f2, in abort
    #1    Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7f532ccae475, in raise
    #0    Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7f532cd029fc, in pthread_kill
    Aborted (Signal sent by tkill() 2844 0)

[moriarty]

This might be a duplicate of ros2/rclcpp#2163 ... which I commented on back in August but forgot about

         @sea-bass the latest rolling binaries aren't expected to fix this.

I tried @fmrico steps ros2/rclcpp#2163 (comment) to reproduce with the open PR ros2/rclcpp#2142 from @mjcarroll and this seems to work.

~/ros/r/rclcpp$ vcs status
..
=== ./src/rclcpp (git) ===
On branch mjcarroll/rclcpp_waitset_executor
Your branch is up to date with 'origin/mjcarroll/rclcpp_waitset_executor'.

nothing to commit, working tree clean
=== ./src/vector_transmission (git) ===
On branch main
Your branch is up to date with 'origin/main'.

Originally posted by @moriarty in ros2/rclcpp#2163 (comment)

[moriarty]

This might also be a duplicate moveit/moveit2_tutorials#822

[moriarty]

Close as duplicate of ros2/rclcpp#2163