Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Poor memory management with dask 2021.04.0 #359

Closed
hakonanes opened this issue May 25, 2021 · 4 comments
Closed

Poor memory management with dask 2021.04.0 #359

hakonanes opened this issue May 25, 2021 · 4 comments

Comments

@hakonanes
Copy link
Member

All EBSD pattern processing methods, like remove_static_background(), didn't increase memory by much with dask 2021.03.0 and earlier, while with 2021.04.0, the memory usage increases drastically.

I believe it is related to dask/dask#7583. Apparently, from the wording in the second comment in that issue, the current implementation of dask array chunking before calls to map_blocks() and finally store() is made by a "newbie", meaning me...

@magnunor, do you experience any memory issues with pyxem using dask 2021.04.0?

@hakonanes hakonanes added the bug Something isn't working label May 25, 2021
@hakonanes hakonanes added this to the v0.4.0 milestone May 25, 2021
@hakonanes hakonanes pinned this issue May 25, 2021
@hakonanes hakonanes modified the milestones: v0.4.0, v0.3.4 May 25, 2021
@hakonanes
Copy link
Member Author

I leave this issue open until we remove the restriction on the newest dask version in setup.py, i.e. when the above referenced issue is fixed.

@hakonanes
Copy link
Member Author

Patch release v0.3.4 is now released, where dask is pinned to <=2021.03.1

@hakonanes
Copy link
Member Author

hakonanes commented Aug 18, 2021

Have to check, but this should be fixed in dask/dask#8040! That fix will be included in version 2021.8.1, I think, so we should be fine by updating the dependencies to allow from that and above.

@hakonanes hakonanes removed the bug Something isn't working label Aug 19, 2021
@hakonanes hakonanes unpinned this issue Aug 19, 2021
@hakonanes
Copy link
Member Author

Fixed in #418.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant