Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate Release Builds with Debug Symbols under GDB for potential issues with rclrs_tests usage of DDS #365

Closed
maspe36 opened this issue Mar 12, 2024 · 2 comments

Comments

@maspe36
Copy link
Collaborator

maspe36 commented Mar 12, 2024

As discussed in the March 11th monthly meetup, we have some occasional unit test failures. These failures have seemingly been narrowed down to rclrs_tests and appear to be related to DDS usage. See #364 for more details.

With that in mind, it is imperative that we take multiple approaches to thoroughly investigate and analyze the root causes of these failures.

For this issue, we agreed that it would be beneficial to attach GDB to some release builds that contain debug symbols, in a fresh ros2 workspace. As I understand it, the hope was we could get a nicer stack trace for our intermittent failures than what we saw in #364.

@maspe36
Copy link
Collaborator Author

maspe36 commented Mar 20, 2024

@mxgrey I'm not sure how deep you dug here, but since #372 I haven't been able to reproduce this locally.

Going to close this for now.

@maspe36 maspe36 closed this as completed Mar 20, 2024
@mxgrey
Copy link
Collaborator

mxgrey commented Mar 20, 2024

I actually wasn't able to get gdb to work correctly on any test cases because the rapid creation and destruction of threads was leaving gdb without a stack to latch onto.

But it's reassuring that you're no longer reproducing it, and it's a good indication that this was a drop order issue on our end all along.

I'm skeptical that we're totally out of the woods yet. There may be potential drop orders that just don't happen to show up in our tests. So I'll still plan on auditing the rcl handlers more closely.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants