Address flakiness in rosbag2_play_end_to_end tests

[MichaelOrlov]

• Closes test_rosbag2_play_end_to_end flaky #408
  Make test_rosbag2_play_end_to_end more deterministic
• Change QoS depth in test databases to correspond number of messages
• Change QoS durability to transient local in test DB and mcap file
• Explicitly specify QoS depth = 10 for subscribers
• Explicitly specify QoS reliability to reliable for subscribers
• Explicitly specify QoS durability to transient local for subscribers
• Update metadata in test DB and mcap files to the latest version(7)
• Start player in pause mode and wait on subscribers for matched
  publishers from player then send resume service call to unpause.
• Add spin_and_wait_for_matched(topic_names) for SubscriptionManager
• Remove xfail test_rosbag2_play_end_to_end

Add wait_for_matched for record_end_to_end_exits_gracefully_on_sigterm

• Remove xfail for test_rosbag2_info_end_to_end

Fix for play_filters_by_topic test

• Uncomment play_filters_by_topic test
• Use proper qos settings for subscribers in play_filters_by_topic
  and fix expectations about number of published messages.
• Log warning if SubscriptionManager::continue_spinning(..) finished by
  timeout.
• Enable play_end_to_end_test for windows.

Note: Didn't remove xfail for test_rosbag2_record_end_to_end because tests fails on Windows build due to absence of the console control handling for CTRL_C_EVENT and CTRL_SHUTDOWN_EVENT events.
Will address test_rosbag2_record_end_to_end tests in follow up PRs.

[MichaelOrlov]

Gist: https://gist.githubusercontent.com/MichaelOrlov/001cdcfa299a0cd40badfb30fae6aa8c/raw/c737f0f4f0068f00fccb96bf909caf35cb4fb722/ros2.repos
BUILD args: --packages-above-and-dependencies rosbag2_tests
TEST args: --packages-above rosbag2_tests
ROS Distro: rolling
Job: ci_launcher
ci_launcher ran: https://ci.ros2.org/job/ci_launcher/11915

• Linux
• Linux-aarch64
• Windows

[MichaelOrlov]

Re-run CI with --retest-until-fail 5 after returning xfail for test_rosbag2_record_end_to_end and enforcing QoS settings for subscribers.
Gist: https://gist.githubusercontent.com/MichaelOrlov/001cdcfa299a0cd40badfb30fae6aa8c/raw/c737f0f4f0068f00fccb96bf909caf35cb4fb722/ros2.repos
BUILD args: --packages-above-and-dependencies rosbag2_tests
TEST args: --packages-above rosbag2_tests
ROS Distro: rolling
Job: ci_launcher
ci_launcher ran: https://ci.ros2.org/job/ci_launcher/12008

• Linux
• Linux-aarch64
• Windows

[MichaelOrlov]

@ros-pull-request-builder retest this please

[MichaelOrlov]

Re-run CI with --retest-until-fail 5 after fixing and enabling play_filters_by_topic test.
Gist: https://gist.githubusercontent.com/MichaelOrlov/001cdcfa299a0cd40badfb30fae6aa8c/raw/c737f0f4f0068f00fccb96bf909caf35cb4fb722/ros2.repos
BUILD args: --packages-above-and-dependencies rosbag2_test_common rosbag2_tests
TEST args: --packages-above rosbag2_test_common rosbag2_tests
ROS Distro: rolling
Job: ci_launcher
ci_launcher ran: https://ci.ros2.org/job/ci_launcher/12009

• Linux
• Linux-aarch64
• Windows

[clalancette]

I've left a few pieces of minor feedback.

[MichaelOrlov]

Re-run CI with --retest-until-fail 5 after review.

BUILD args: --packages-above-and-dependencies rosbag2_test_common rosbag2_tests
TEST args: --packages-above rosbag2_test_common rosbag2_tests
ROS Distro: rolling
Job: ci_launcher
ci_launcher ran: https://ci.ros2.org/job/ci_launcher/12033

• Linux
• Linux-aarch64
• Windows

[MichaelOrlov]

@clalancette Fair warning about or CI.
I know that there are race condition in process destruction in this PR and play_filters_by_topic test fails every other time on my local machine. But our CI didn't catch it for Linux versions even with --retest-until-fail 5. And as you can see rpr job also passes green.
This is not a first time I see that our CI not catching pretty obvious failures to witch it was pretty sensitive before (priore to the code freeze for Iron).
It looks like now we have pretty performant machines in CI and they are not very loaded with anything else.
IMO it should be some balanced load on CI infrastructure to catch corner cases and race conditions.

[MichaelOrlov]

Re-run CI with --retest-until-fail 5 after addressing race condition in process destruction.

BUILD args: --packages-above-and-dependencies rosbag2_test_common rosbag2_tests
TEST args: --packages-above rosbag2_test_common rosbag2_tests
ROS Distro: rolling
Job: ci_launcher
ci_launcher ran: https://ci.ros2.org/job/ci_launcher/12034

• Linux
• Linux-aarch64
• Windows

[MichaelOrlov]

@clalancette FYI I have fixed race conditions in process termination routines. Which was causing intermittent test failures due to sending SIGINT signal when the process was already in the destruction phase.
Now we will wait to finish rosbag2 play process instead of trying forcefully stop it.
See my latest commit 3af5b41 for details.

[clalancette]

It looks like this probably needs some PUBLIC annotations for Windows CI to pass.

[MichaelOrlov]

@clalancette Nop. It's a bit more straightforward. The Windows build was failing because I start using wait_until_completion inside stop_execution and the former one was declared after the latter. I made a fix in the new commit.

Re-running CI for Windows.

• Windows

[MichaelOrlov]

@clalancette This PR has ABI and API changes in the test framework, although I want to backport it to the Iron branch if you wouldn't mind. It will help us to have a more stable CI for Iron.

[clalancette]

@clalancette This PR has ABI and API changes in the test framework, although I want to backport it to the Iron branch if you wouldn't mind. It will help us to have a more stable CI for Iron.

Yeah, that should be fine here.

[MichaelOrlov]

Re-run Windows CI after fixing build errors. It was copy-past from Unix version and process_id shall be renamed to the handle

Re-running CI for Windows.

• Windows

[MichaelOrlov]

Re-running CI for Windows after fixing the compilation error from the previous run.

• Windows

[MichaelOrlov]

CI failed on Windows in the test_rosbag2_play_end_to_end with error message

C:\ci\ws\src\ros2\rosbag2\rosbag2_tests\test\rosbag2_tests\test_rosbag2_play_end_to_end.cpp(80): error: Expected equality of these values:
  future.wait_for(service_call_timeout_)
    Which is: 4-byte object <01-00 00-00>
  std::future_status::ready
    Which is: 4-byte object <00-00 00-00>

Line test_rosbag2_play_end_to_end.cpp(80) corresponds to the

rosbag2/rosbag2_tests/test/rosbag2_tests/test_rosbag2_play_end_to_end.cpp

Line 80 in 3966c81

EXPECT_EQ(future.wait_for(service_call_timeout_), std::future_status::ready);

Which means timeout for getting a response from a service call. The timeout is defined there as 4 seconds. It might be not enough for Windows.

[MichaelOrlov]

Re-running CI for Windows after increasing timeout for service call wait.

• Windows

[MichaelOrlov]

https://github.com/Mergifyio backport iron

[mergify]

backport iron

✅ Backports have been created

• #1330 Address flakiness in rosbag2_play_end_to_end tests (backport #1297) has been created for branch iron

[ros-discourse]

This pull request has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/ros-2-tsc-meeting-minutes-2023-05-18/31587/1