Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix flakiness in tests caused by WindowsTime #8087

Merged
merged 1 commit into from Aug 9, 2023
Merged

Fix flakiness in tests caused by WindowsTime #8087

merged 1 commit into from Aug 9, 2023

Conversation

crusaderky
Copy link
Collaborator

@crusaderky crusaderky commented Aug 9, 2023 •
edited

There may be more affected tests but it's hard to find them all in one go

fjetter
fjetter reviewed Aug 9, 2023
View reviewed changes
distributed/utils_test.py
Comment on lines +2601 to +2604
"""Temporarily disable the automatic resync of distributed.metrics._WindowsTime
which, every 10 minutes, can cause time() to go backwards a few milliseconds.

On Linux and MacOSX, this fixture is a no-op.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't the entire point of _WindowsTime to have a monotonic clock?

Copy link
Collaborator Author

@crusaderky crusaderky Aug 9, 2023 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only on distributed.metrics.monotonic. Not on distributed.metrics.time.
time() offers no guarantee of being monotonic, on any OS. However on Linux it's very unlikely to be something that actually happens as you need to actually change the wall clock with NTP.

@github-actions
Copy link
Contributor

github-actions bot commented Aug 9, 2023

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

       20 files  ±0         20 suites  ±0   11h 26m 5s ⏱️ + 18m 31s
  3 754 tests ±0    3 643 ✔️ ±0     106 💤 ±0  5 ❌ +1 
36 314 runs  ±0  34 558 ✔️ +1  1 751 💤 +1  5 ❌  - 1 

For more details on these failures, see this check.

Results for commit 13ce317. ± Comparison against base commit 9b8f3b8.

@crusaderky crusaderky self-assigned this Aug 9, 2023
@fjetter fjetter merged commit 9469b91 into dask:main Aug 9, 2023
20 of 28 checks passed
@crusaderky crusaderky deleted the WindowsTime branch August 9, 2023 15:45
@crusaderky
Copy link
Collaborator Author

This PR was ineffective at fixing the problem: https://github.com/dask/distributed/actions/runs/5854070495/job/15869288671?pr=8101

This was referenced Aug 14, 2023
Merged
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@fjetter fjetter fjetter approved these changes

Assignees

@crusaderky crusaderky

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Flaky test_task_groups_update_start_stop
2 participants
@crusaderky @fjetter