Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better logging for anomalous task termination #8082

Merged
merged 2 commits into from Aug 8, 2023

Conversation

crusaderky
Copy link
Collaborator

@crusaderky crusaderky commented Aug 8, 2023

Follow-up to #8006
In #8006, CancelledError not only caused the worker state machine to deadlock, but it also left no trace in the log.
While #8013 prevents this specific issue, it's wise to ensure that CancelledError is never silently ignored.

@crusaderky crusaderky self-assigned this Aug 8, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Aug 8, 2023

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

       20 files  ±0         20 suites  ±0   11h 31m 28s ⏱️ + 35m 38s
  3 752 tests ±0    3 642 ✔️  - 3     104 💤 ±0  6 +4 
36 477 runs  ±0  34 806 ✔️  - 2  1 665 💤 ±0  6 +3 

For more details on these failures, see this check.

Results for commit 640c592. ± Comparison against base commit ef6d4bf.

♻️ This comment has been updated with latest results.

return
except BaseException: # pragma: nocover
logger.exception("async instruction handlers should never raise!")
# This should never happen
logger.exception(f"Unhandled exception in async instruction for {task}")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not intended as blocking comment, but I could see it being useful if we included the type of the exception and possibly the exception message here.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

logger.exception automatically does that

@crusaderky
Copy link
Collaborator Author

As test_quiet_client_close highlights, this change can cause quite a bit of noise if someone shuts down a LocalCluster while tasks are still running on it. I believe this is a good thing.

@crusaderky crusaderky merged commit de86523 into dask:main Aug 8, 2023
21 of 27 checks passed
@crusaderky crusaderky deleted the log-anomalous-task-end branch August 8, 2023 17:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants