Skip to content

Add dask.blockwise.fuse_roots optimization#5451

Merged
mrocklin merged 8 commits intodask:masterfrom
mrocklin:blockwise-root-fuse
Nov 12, 2019
Merged

Add dask.blockwise.fuse_roots optimization#5451
mrocklin merged 8 commits intodask:masterfrom
mrocklin:blockwise-root-fuse

Conversation

@mrocklin
Copy link
Copy Markdown
Member

@mrocklin mrocklin commented Oct 2, 2019

This fuses data generation or access graphs into blockwise after
converting everything to low level task graphs

cc @TomAugspurger we were just talking about this

This fuses data generation or access graphs into blockwise after
converting everything to low level task graphs
@mrocklin mrocklin force-pushed the blockwise-root-fuse branch from eece3b6 to b8a5d97 Compare October 2, 2019 19:39
@mrocklin
Copy link
Copy Markdown
Member Author

mrocklin commented Nov 9, 2019

Tests pass. I'm going to wait until next week to make sure that the 2.7.0 release is smooth, and then I'll merge this in if there are no objections.

@mrocklin mrocklin merged commit 1f87ae6 into dask:master Nov 12, 2019
@mrocklin mrocklin deleted the blockwise-root-fuse branch November 12, 2019 16:53
@mrocklin
Copy link
Copy Markdown
Member Author

@TomAugspurger @rabernat this is in. It should help avoid communication in situations where multiple arrays are loaded and then combined together as in z = x + y. I believe that this was causing you some heartache with excessive memory use. Some of your workloads may now be smoother with this in.

@dcherian
Copy link
Copy Markdown
Collaborator

Exciting! Thanks @mrocklin

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants