Avoid substituting keys that occur multiple times #10646
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #10645
I haven't run any widespread tests, yet. However, for the cases that are affected, this triggers the computation of a task twice and apparently also prohibits some kind of numpy optimization as reported in #10645
The graph topology here doesn't strike me as rare or very special
so the
add
in this graph is basically fused into a fused into amatmul-transpose
blob / subgraphcallable that accepts two inputs that are identical. I would argue that it is a bug of blockwise_fusion that generates such a blob in the first place since by rewriting that subgraph callable it would be possible to accept just a single dependency, restoring this fusion again.I'm not very familiar with the blockwise fusion code and was able to find this first.
With this PR the above graph optimizes to
This change effectively stops fusing because it detects this situation. If the blockwise fusion bug is fixed, it would fuse into one task (again) but without recomputing the intermediate result twice
cc @rjzamora