Ballista shuffle is finally working as intended, providing scalable distributed joins

[andygrove]

Which issue does this PR close?

Builds on #738 Closes #707.

With this PR we finally have scalable distributed joins.

Query 12 performance at SF=100

Executor concurrent tasks	Time (ms)
2	32592.94
4	17865.03
8	11641.62
16	9296.60

+------------+-----------------+----------------+
| l_shipmode | high_line_count | low_line_count |
+------------+-----------------+----------------+
| MAIL       | 623097          | 934694         |
| SHIP       | 622959          | 934510         |
+------------+-----------------+----------------+
Query 12 avg time: 9896.37 ms

Integration tests pass.

Rationale for this change

This is making Ballista work as it was intended to work.

What changes are included in this PR?

Tons of bug fixes around shuffles.

Are there any user-facing changes?

No

[andygrove]

@houqp @Dandandan @edrevo @alamb @jorgecarleitao Ballista is finally working with scalable distributed joins, at least it is for TPC-H. I plan on following up with some further smaller code cleanup PRs now that the functionality is working.

[alamb]

I reviewed the code -- while I am not a ballista expert it seems reasonable to me.

One thing I did notice was that there don't appear to be any new / updated tests in this PR.

[alamb]

is this something that you want to finish up in this PR?

[andygrove]

I've filed https://github.com/apache/arrow-datafusion/issues/758 as a follow-up for implementing this since it involves more serde work.

[andygrove]

One thing I did notice was that there don't appear to be any new / updated tests in this PR.

I've added an additional test to check that TPC-H query 12 gets planned with correct partitioning information in the shuffle readers.

[alamb]

🎉