Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fuse connected blockwise subgraphs #136

Closed
Tracked by #339
tomwhite opened this issue Oct 3, 2022 · 1 comment · Fixed by #346
Closed
Tracked by #339

Fuse connected blockwise subgraphs #136

tomwhite opened this issue Oct 3, 2022 · 1 comment · Fixed by #346

Comments

@tomwhite
Copy link
Member

tomwhite commented Oct 3, 2022

It should be possible to fuse any connected subgraph of blockwise operations into a single blockwise operation.

For the single output case, it is possible to trace each block of the output to its (transitive) input blocks. The blockwise functions would be composed appropriately.

Handling memory requirements would need some care. It might be useful to break up the required memory for an operation into compressed_in, in, out, compressed_out (where the compressed parts are the buffers used by Zarr). Then it would be possible to drop the compressed parts when combining operations (since they no longer write to Zarr).

But subgraphs with large numbers of inputs might still hit memory limits, so it would not always be possible to combine the whole subgraph into one. This suggests that when checking for if fusion can take place, memory limits should be taken into account.

This could be generalised for multiple outputs (#69), but the two issues are somewhat independent.

@tomwhite
Copy link
Member Author

This is an example of how the subgraph could be fused (nodes in the red circle):

cubed-geo

Here's a simpler case, where all the pink boxes could be fused into one:

cubed-fusion-multiple-inputs-unoptimized

Tests for these cases are here: 26cb6cb

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant