Arrow Row Format in SortExec #5230

[jaylmiller]

Which issue does this PR close?

issue #5230

Rationale for this change

Modifying SortExec to use arrow row format should increase performance when sorting by multiple columns. Row encoding is preserved so that it can be used by parent nodes (right now that is just SortPreservingMergeExec).

What changes are included in this PR?

Both iterations of:

A first iteration could simply modify sort_batch to use the row format for multi-column sorts, as demonstrated here, falling back to sort_to_indices if only a single column.

A second iteration could then look to find a way to convert to the row format once, and preserve this encoding when feeding sorted batches into SortPreservingMerge.

Are these changes tested?

Mostly covered by existing unit tests. Added unit tests for the new publicly exported structs.

Are there any user-facing changes?

Some public exports had to be added to be able to be used in the integration test core/tests/sort_key_cursor.

[alamb]

Thanks @jaylmiller -- this is looking good. Very nice first contribution

I think this code could almost go in as it -- the only thing that I think we need to see is some sort of benchmark improvement.

I looked around in our existing benchmarks to find a benchmark for our sort operator, but was not able to. Maybe @tustvold or @Dandandan or @andygrove know of a preexisting benchmark

If not, perhaps we can adapt some of the code in https://github.com/apache/arrow-datafusion/blob/master/datafusion/core/benches/merge.rs to make a SortExec benchmark

[jaylmiller]

Thanks @jaylmiller -- this is looking good. Very nice first contribution

I think this code could almost go in as it -- the only thing that I think we need to see is some sort of benchmark improvement.

I looked around in our existing benchmarks to find a benchmark for our sort operator, but was not able to. Maybe @tustvold or @Dandandan or @andygrove know of a preexisting benchmark

If not, perhaps we can adapt some of the code in https://github.com/apache/arrow-datafusion/blob/master/datafusion/core/benches/merge.rs to make a SortExec benchmark

Thanks! Should I add said benchmark to this PR then?

[jaylmiller]

@alamb Does this SortExec benchmark seem good?

[alamb]

@alamb Does this SortExec benchmark seem good?

Yes, I think it looks like a good start. Thank you @jaylmiller . Could you please create a new PR with just the sort benchmark (so we can merge that and use it as a way to compare the code on master and this PR)?

Thanks again!