-
Notifications
You must be signed in to change notification settings - Fork 29
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Poor memory management with dask 2021.04.0 #359
Comments
I leave this issue open until we remove the restriction on the newest dask version in setup.py, i.e. when the above referenced issue is fixed. |
Patch release v0.3.4 is now released, where dask is pinned to <=2021.03.1 |
Have to check, but this should be fixed in dask/dask#8040! That fix will be included in version |
Fixed in #418. |
All EBSD pattern processing methods, like remove_static_background(), didn't increase memory by much with dask 2021.03.0 and earlier, while with 2021.04.0, the memory usage increases drastically.
I believe it is related to dask/dask#7583. Apparently, from the wording in the second comment in that issue, the current implementation of dask array chunking before calls to
map_blocks()
and finallystore()
is made by a "newbie", meaning me...@magnunor, do you experience any memory issues with pyxem using dask 2021.04.0?
The text was updated successfully, but these errors were encountered: