Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Assertion for SpillBuffer breaks usage of custom Worker(data=...) types #5909

Closed
pentschev opened this issue Mar 7, 2022 · 2 comments · Fixed by #5904
Closed

Assertion for SpillBuffer breaks usage of custom Worker(data=...) types #5909

pentschev opened this issue Mar 7, 2022 · 2 comments · Fixed by #5904
Assignees

Comments

@pentschev
Copy link
Member

#5814 introduced https://github.com/dask/distributed/blob/main/distributed/worker.py#L3884 which I believe unintentionally breaks the flexibility of passing a custom type via Worker(data=...). In Dask-CUDA we use the data= kwarg to pass a specialized handler for spilling GPU memory, and it currently breaks one of our tests, see here.

Would it be possible to remove that assertion or make it somehow flexible so that a custom type wouldn't fail?

cc @crusaderky who's the author.

cc @quasiben @madsbk

@crusaderky
Copy link
Collaborator

Fixed in #5904. I would be grateful if you could review it and see if it is fit for purpose for you.

@pentschev
Copy link
Member Author

Thanks @crusaderky for the reply, I responded in #5904 (comment) .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants