Consider reactivating low-level DataFrame optimization when not all layers are Blockwise

[gjoseph92]

Since #7620, we've seen a few instances where users have gotten burned by root-task overproduction (see dask/distributed#5555, dask/distributed#5223 for background) because certain DataFrame optimizations still use low-level graphs, and therefore aren't getting fused anymore. Examples:

• Suboptimal graph structure when read-writing a parquet #8445
• partition_info in map_partitions materializes the graph unnecessarily #8309
• Use Blockwise/map_partitions in various DataFrame join methods #8306

We do want to get everything to Blockwise eventually, but our bandwith to track these down and fix them is limited. In the interim, I propose that by default, we still do low-level fusion when any of the layers in the graph are materialized.

cc @rjzamora @ian-r-rose @jrbourbeau

[rjzamora]

We do want to get everything to Blockwise eventually, but our bandwith to track these down and fix them is limited. In the interim, I propose that by default, we still do low-level fusion when any of the layers in the graph are materialized.

My motivation to finally refocus my efforts on HLG cleanup is certainly reaching a tipping point! Sorry for being somewhat MIA in these topics lately.

I agree that we probably want the middle ground you are suggesting. As I type this, I see that you submitted a PR - So, perhaps we can discuss the specific details there :)