optimise linear layer chains

[martindurant]

No description provided.

[martindurant]

I suspect this fails when reducing in any layer (i.e,. number of indices decreases), but for the purposes of dask-awkward we might be fine, where there is always one index of iteration: the partition.

[codecov-commenter]

Codecov Report

Merging #210 (c546cda) into main (4db98e1) will increase coverage by 0.14%.
The diff coverage is 98.75%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@            Coverage Diff             @@
##             main     #210      +/-   ##
==========================================
+ Coverage   94.74%   94.88%   +0.14%     
==========================================
  Files          19       19              
  Lines        2111     2190      +79     
==========================================
+ Hits         2000     2078      +78     
- Misses        111      112       +1     

Impacted Files	Coverage Δ
src/dask_awkward/lib/optimize.py	96.55% <98.75%> (+1.12%)	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[douglasdavis]

related: dask/dask#9795

[martindurant]

@rjzamora , you might be interested to look at this. It implements a linear-time "fuse" operation on chains of Blockwise layers which is much faster that standard dask's layer fuse for very large numbers of layers. As it stands, the algorithm passes the specific two tests written for it, but causes many other dask-awkward tests to fail. I believe that there are two most likely causes:

• we should be excluding reductive layers from consideration, but aren't. It would be even better if we could cope with these, but the number of indices changes and that would need special attention.
• something is not quite right with the keys included in the subgraph versus the keys mentioned in the various attributes of a fused layer.

Any thoughts you might have on this would be appreciated!

[martindurant]

Good to go?

[lgray]

As far as I can tell from torture tests, yes!