Integrate `BatchCoalescer` into `HashJoinExec` and remove from `CoalesceBatches` optimization rule

[alamb]

Is your feature request related to a problem or challenge?

• Part of [EPIC] Remove CoalesceBatchesExec operator #18779

The LimitedBatchCoalescer can now be used to integrate batch coalescing directly in ExecutionPlan implementations, which is better than using an external operator for the reasons described on #18779

Currently HashJoinExec needs a CoalesceBatches operator after for performance, see

datafusion/datafusion/physical-optimizer/src/coalesce_batches.rs

Line 61 in bd30fe2

let wrap_in_coalesce = plan_any.downcast_ref::<HashJoinExec>().is_some()

We should integrate coalescing directly in the operator

Describe the solution you'd like

1. Add batch coalescing into HashJoinExec
2. Remove HashJoinExec from the CoalesceBatches optimizer rule:

   datafusion/datafusion/physical-optimizer/src/coalesce_batches.rs

   Line 61 in bd30fe2

   let wrap_in_coalesce = plan_any.downcast_ref::<HashJoinExec>().is_some()

Describe alternatives you've considered

@Dandandan did this for FilterExec in these two PRs, so we can probably follow a similar model

• Coalesce batches inside FilterExec #18604
• Remove FilterExec from CoalesceBatches optimization rule, add fetch support #18630

Additional context

No response

[alamb]

FYI @gene-bordegaray this is one of the tickets I mentioned to you today

[gene-bordegaray]

FYI @gene-bordegaray this is one of the tickets I mentioned to you today

Ok great, I will spend some time reading up on the epic and see if I can pick up some of this work

[jonathanc-n]

@gene-bordegaray Is it fine if I take over this one? I'm quite interested in the hash join stuff. I believe the repartition operator should be somewhat similar to this if you are interested

[gene-bordegaray]

@gene-bordegaray Is it fine if I take over this one? I'm quite interested in the hash join stuff. I believe the repartition operator should be somewhat similar to this if you are interested

Yes yes, of course. I am just trying to learn so will be snooping 😅

[jonathanc-n]

take