Secure publisher_subscriber tests failing with timeouts

[azeey]

A random subset of the various combinations of test_secure_publisher_subscriber_* tests have started failing since https://ci.ros2.org/job/nightly_linux_repeated/1900/ (and maybe even earlier). The failures have all been timeouts.

• test_secure_publisher_subscriber__UnboundedSequences__rmw_fastrtps_cpp__secure_comm_0
• test_secure_publisher_subscriber__Empty__rmw_fastrtps_cpp__secure_comm_3
• test_secure_publisher_subscriber__Empty__rmw_fastrtps_cpp__secure_comm_3
• test_secure_publisher_subscriber__Empty__rmw_fastrtps_cpp__secure_comm_1
• test_secure_publisher_subscriber__UnboundedSequences__rmw_fastrtps_cpp__secure_comm_1
• test_secure_publisher_subscriber__Empty__rmw_fastrtps_cpp__secure_comm_2

[dirk-thomas]

@mikaelarguedas Can you take a look at these test failures?

[mikaelarguedas]

Thanks for the ping

The issue description doesn't give much context, when did these actually start appearing ? do they happen only on FastRTPS ? only on Linux ? If yes, this looks like a potential duplicate of https://github.com/ros2/sros2/issues/221
Other potentially insightful information:

• do the regular test_communication tests (those without security) fail in a similar manner sometimes ?
  • if so it is likely a discovery problem and the tests are not robust to it (and security may exacerbate this underlying issue)
  • if not: as there been any changes to test_communication to improve robustness of discovery that could be integrated to test_security?

I'm not sure when I'd have bandwidth to look at these, assigning @ros-security-wg hoping that someone else from the WG may have time to look into this.

[kyrofa]

@azeey any chance you could provide the information @mikaelarguedas requested so we can better triage?

[ivanpauno]

The issue description doesn't give much context, when did these actually start appearing ?

I'm not quite sure, but they seem to be old (as the tests are too flaky, it's hard to track when they started).

do they happen only on FastRTPS ?

I haven't seen that error in other rmw implementations.

If yes, this looks like a potential duplicate of ros2/sros2#221

👍 I think we can close one.
@mikaelarguedas Let me know which one you prefer opened.

do the regular test_communication tests (those without security) fail in a similar manner sometimes ?

In the last 10 linux repeated builds I have only found this test_communication failure:
https://ci.ros2.org/job/nightly_linux_repeated/1993/testReport/junit/test_communication/TestPublisherSubscriberAfterShutdown/test_processes_finished_gracefully_Strings_/
which doesn't seem to be the same problem.

if so it is likely a discovery problem and the tests are not robust to it (and security may exacerbate this underlying issue)

I haven't look at the test thoroughly, but I think that could be the case.

if not: as there been any changes to test_communication to improve robustness of discovery that could be integrated to test_security?

Both tests look quite similar to me, and I haven't found any improvement in test_communication history.

[chapulina]

Update on the status of each test:

• test_secure_publisher_subscriber__UnboundedSequences__rmw_fastrtps_cpp__secure_comm_0

  • no failure for the past 31 biilds

• test_secure_publisher_subscriber__Empty__rmw_fastrtps_cpp__secure_comm_3

  • 1 failure in the last 24 builds

• test_secure_publisher_subscriber__Empty__rmw_fastrtps_cpp__secure_comm_3

  • 1 failure in the past 24 builds

• test_secure_publisher_subscriber__Empty__rmw_fastrtps_cpp__secure_comm_1

  • no failure for the past 31 builds

• test_secure_publisher_subscriber__UnboundedSequences__rmw_fastrtps_cpp__secure_comm_1

  • 3 failures in the past 24 builds, including last night

• test_secure_publisher_subscriber__Empty__rmw_fastrtps_cpp__secure_comm_2

  • 1 failure in the past 24 builds

[jacobperron]

Another instance of security-related tests failing recently (November 30th): https://ci.ros2.org/view/nightly/job/nightly_linux_repeated/2122/testReport/

[mikaelarguedas]

It also keeps failing on sros2 CI on a regular basis: https://github.com/ros2/sros2/runs/1488014494?check_suite_focus=true

[clalancette]

The last time this seems to have failed was on April 1st in: https://ci.ros2.org/view/nightly/job/nightly_linux_repeated/2244/ . Since then, there have been no instances of this test failure in nightly_linux_repeated. Thus, I'm going to go ahead and close this out; feel free to reopen if we see it again.