Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix transient "Area 51" failures #3476

Merged
merged 3 commits into from Jul 3, 2023

Conversation

webbnh
Copy link
Member

@webbnh webbnh commented Jul 3, 2023

This PR addresses an instability in Agent legacy tests test-51 and test-52.

In graceful_shutdown() in tool_meister_stop.py, the code issues a debug log message if it has to wait for the Tool Data Sync to exit. However, if, by the time control reaches this point, the TDS has already exited, then the PID file will have been removed, and the attempt to read it produces an exception which is discarded and the wait is skipped.

This results in a race condition which may or may not produce the log message output. There is no correctness issue associated with the presence or absence of the message in the log, but the legacy unit test system won't tolerate variable output, and, since the "gold" file includes the message, the test is considered to have failed any time the race goes the other way.

This PR modifies the output filtering for this message to, instead of masking just the PID value, remove the line from the log altogether if it is present. And, it updates the two relevant "gold" files. This should enable a deterministic result.

This PR also removes an unused definition which was left over from PR #3427 and reformats a long line which it added.

@webbnh webbnh added Agent Unit tests Tool Meister Of and relating to the Tool Meister sub-system Tests labels Jul 3, 2023
@webbnh webbnh added this to the v0.73 milestone Jul 3, 2023
@webbnh webbnh requested review from ndokos and dbutenhof July 3, 2023 18:16
@webbnh webbnh self-assigned this Jul 3, 2023
exec-tests Outdated Show resolved Hide resolved
Copy link
Member

@ndokos ndokos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (but I haven't run the tests locally - I'll go do that and report back)

Copy link
Member

@dbutenhof dbutenhof left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍🏻 💯 🥇 🚀 🎸 🎆 🥳

@webbnh webbnh merged commit 7d5defa into distributed-system-analysis:main Jul 3, 2023
4 checks passed
@webbnh webbnh deleted the no_area_51 branch July 3, 2023 20:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Agent Tests Tool Meister Of and relating to the Tool Meister sub-system Unit tests
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

None yet

3 participants