Fix P2P shuffle with LocalCluster(..., processes=False)
#8125
Conversation
Unit Test ResultsSee test report for an extended history of previous test failures. This is useful for diagnosing flaky tests. 21 files ± 0 21 suites ±0 11h 47m 43s ⏱️ + 20m 50s For more details on these failures, see this check. Results for commit 2a05649. ± Comparison against base commit acb2809. ♻️ This comment has been updated with latest results. |
| @@ -119,6 +119,9 @@ def get_or_create( | |||
| key: str, | |||
There was a problem hiding this comment.
is the spec: type annotation incorrect? Should it be ToPickle[ShuffleSpec] | ShuffleSpec ?
There was a problem hiding this comment.
I don't mind adjusting this, I honestly don't care too much about typing for a case that shouldn't exist in the first place.
There was a problem hiding this comment.
I've adjusted typing.
| @@ -119,6 +119,9 @@ def get_or_create( | |||
| key: str, | |||
| worker: str, | |||
| ) -> ToPickle[ShuffleRunSpec]: | |||
| # FIXME: Sometimes, this doesn't actually get pickled | |||
There was a problem hiding this comment.
which times are sometimes? Can create an issue about this and link it?
There was a problem hiding this comment.
My suspicion is that it's related to inproc://, but I've also seen this mentioned in other parts of the codebase, so I'm not sure about that. I was planning to investigate this soon(ish), but there are more pressing issues for now.
Closes dask/dask#10455
pre-commit run --all-files