Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cloudpickle's downstream distributed tests failing #6403

Closed
jakirkham opened this issue May 20, 2022 · 2 comments · Fixed by cloudpipe/cloudpickle#472
Closed

cloudpickle's downstream distributed tests failing #6403

jakirkham opened this issue May 20, 2022 · 2 comments · Fixed by cloudpipe/cloudpickle#472

Comments

@jakirkham
Copy link
Member

As part of cloudpickle's downstream testing support (to test with different libraries using cloudpickle), the distributed portion of these tests fails. In particular all the failures are from distributed itself. Relevant CI log. More details in PR ( cloudpipe/cloudpickle#471 ).

cc @ogrisel @jrbourbeau @mrocklin

@ogrisel
Copy link
Contributor

ogrisel commented May 20, 2022

Maybe the CI config in cloucpickle is lacking some mandatory test dependencies but it's not obvious for instance the p2p shuffle error does not raise any error message related to a missing package.

@jrbourbeau
Copy link
Member

We recently added a new, but still unreleased p2p shuffle algorithm for dask.dataframe which uses distributed under the hood. The NotImplementedError: Unknown shuffle method p2p failure in the distributed tests is because (I think) CI is using the latest main branch of distributed, but the latest release of dask, which doesn't yet have the p2p shuffle option added. We're releasing dask and distributed in a couple of hours, so this particular issue should resolve itself once the new dask release is out. That said, in general dask and distributed are co-developed and released together, so we should probably update cloudpickle's CI to install the main branch of both dask and distributed in the downstream distributed CI build

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants