Implement a sort operator

[dominiklohmann]

This is a very basic implementation of a sort operator that comes with a few fundamental limitations:

• All data is kept in memory
• There is no windowing support
• Extension types are not supported for sorting (ip, subnet, enum).
• Sorting only works by one column at a time
• The resulting slices of the operator have a size of 1, which leads to very slow data access in following operators

But other than that it works as you'd expect.

Tasks

Preview Give feedback

1. Implement in-memory sorting
2. Write follow-up stories for the above-mentioned limitations
3. Write tests
4. Write documentation (including known limitations)
5. Add a changelog entry

[mavam]

@dominiklohmann I pushed a commit with an updated documentation.

[mavam]

One thing I noticed during testing: just sort by itself doesn't work, even if the input has just a single field. I would argue we note this down and tackle it during the other follow-ups.

[mavam]

Added another commit to test nulls-first. (Currently failing)

[dominiklohmann]

One thing I noticed during testing: just sort by itself doesn't work, even if the input has just a single field. I would argue we note this down and tackle it during the other follow-ups.

The sensible default if the field is omitted is likely sort . asc nulls-last, but we can't express that in the expression language yet. I've added a story for this.

[mavam]

Overall, this works as expected. I added tests and they now work.

Unfortunately I lack the Arrow depth to review what the code does in detail. If you need a second pair of eyes from @tenzir/pipelines-squad, I'd ask there. Otherwise my approval is only based on high-level skimming and testing.