Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flaky test_robust_to_bad_sizeof_estimates #5753

Merged

Conversation

crusaderky
Copy link
Collaborator

Fix occasional flakiness in test:
https://github.com/dask/distributed/runs/5036808926?check_suite_focus=true

The test hung on a 5s timeout. I think it's because the OS could not write to disk 100MB worth of data fast enough.

@crusaderky crusaderky self-assigned this Feb 3, 2022
config={
"distributed.worker.memory.target": False,
"distributed.worker.memory.spill": 0.7,
},
Copy link
Collaborator Author

@crusaderky crusaderky Feb 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no change from default for spill, but I like to spell out explicitly relevant settings - also in the case they change in the future

reported by sizeof(), which may be inaccurate
"""
memory = s.workers[a.address].memory.process
# Reach 'spill' threshold after 400MB of managed data
a.memory_limit = memory / 0.7 + 400e6
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this works just by luck today for the purpose of the spill threshold, and does NOT work for the target threshold. See #5367

@github-actions
Copy link
Contributor

github-actions bot commented Feb 3, 2022

Unit Test Results

       16 files   -        2         16 suites   - 2   8h 11m 39s ⏱️ - 1h 20m 7s
  2 582 tests ±       0    2 502 ✔️ +       2       80 💤 ±    0  0  - 2 
20 398 runs   - 2 706  19 106 ✔️  - 2 382  1 292 💤  - 322  0  - 2 

Results for commit d8ca3a2. ± Comparison against base commit 30ffa9c.

@gjoseph92 gjoseph92 self-requested a review February 3, 2022 17:05
@crusaderky crusaderky merged commit d0a93e9 into dask:main Feb 8, 2022
@crusaderky crusaderky deleted the test_robust_to_bad_sizeof_estimates branch February 8, 2022 09:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants