Improve error on cancelled tasks due to disconnect #8705
Conversation
Unit Test ResultsSee test report for an extended history of previous test failures. This is useful for diagnosing flaky tests. 29 files ± 0 29 suites ±0 11h 31m 28s ⏱️ + 1h 45m 39s For more details on these failures, see this check. Results for commit 192674b. ± Comparison against base commit 03035da. This pull request removes 13 and adds 10 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
distributed/client.py
Outdated
| @@ -5445,7 +5456,7 @@ async def _wait(fs, timeout=None, return_when=ALL_COMPLETED): | |||
| {fu for fu in fs if fu.status != "pending"}, | |||
| {fu for fu in fs if fu.status == "pending"}, | |||
| ) | |||
| cancelled = [f.key for f in done if f.status == "cancelled"] | |||
| cancelled = {f.key: f._state.exception for f in done if f.cancelled()} | |||
There was a problem hiding this comment.
This might become very large. Ideally, we'd group these by the reason for cancellation.
distributed/client.py
Outdated
| bad = {f for f in futures if f.cancelled()} | ||
| if bad: | ||
| raise CancelledError(bad) | ||
| cancelled = {f.key: f._state.exception for f in futures if f.cancelled()} |
There was a problem hiding this comment.
This might become very large. Ideally, we'd group these by the reason for cancellation.
| if key not in self.futures or self.futures[key].status in failed: | ||
| for future in future_set: | ||
| key = future.key | ||
| if key not in self.futures or future.status in failed: |
There was a problem hiding this comment.
As opposed to futures_of and wait, we only raise a single exception here. We should probably align this.
There was a problem hiding this comment.
We should do this in a follow-up PR, this mixes cancellations with actual errors.
|
test failure in |
Should be fixed, it was one of those tests that worked locally but failed on CI |
distributed/tests/test_client.py
Outdated
| # @pytest.mark.slow | ||
| @gen_cluster(client=True, Worker=Nanny, timeout=60) |
Closes #8690
pre-commit run --all-files