[BEAM-7268] make sorter extension Hadoop-free

[nevillelyh]

Right now the Java sorter extension depends on Hadoop SequenceFile for external sort. It'll be nice to re-implement it without the dependency to avoid conflicts.

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

Post-Commit Tests Status (on master branch)

Lang	SDK	Apex	Dataflow	Flink	Gearpump	Samza	Spark
Go		---	---	---	---	---	---
Java
Python		---			---	---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website
Non-portable
Portable	---		---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

[nevillelyh]

R: @kanterov

[kanterov]

Great to reduce dependencies. If dependency conflicts are the only concern, an alternative could be shading Hadoop dependencies. It's hard to say without having benchmarks if this code is going to perform better or not. I would suggest having it as an alternative implementation for sorting and keep the previous one.

If I'm not mistaken, your work for SortValues is needed to express GroupByKeyAndSortValues as GroupByKey+SortValues. There is a thread on a mailing list on adding GroupByKeyAndSortValues as a built-in transform in Beam gbk-sort-values. I would rather go that way, runners such as Dataflow, Spark, and Flink would override it with efficient implementation because they have a concept of secondary sorting, that would be way more efficient than anything else.

What do you think about rather investing in having GroupByKeyAndSortValues, because, in the end, that's what we need? It might be better to move this discussion into the mailing list.

cc @kennknowles @reuvenlax

[kennknowles]

Seems like a fine idea to me. It is a good point that you could just add it as a parallel extension. Or do fancy build tricks. But probably it is just easy and clean to have a separate library. It will be a little while before I can give this a good review, and I think @reuvenlax is the better choice anyhow.

[nevillelyh]

Agree with both. It's still worth having a "lean" option though. I'll work on some benchmark & adding it as a parallel implement, e.g. {Native,Hadoop}ExternalSorter?

[nevillelyh]

Split ExternalSorter into Native & Hadoop. Also did some basic benchmark, with 5 million UUID KV pairs (~360MB raw bytes), 64MB memory, Hadoop vs native impl is ~51.4s vs ~17.9s.

[nevillelyh]

I've made the change backwards compatible and added the native impl as an option. @kanterov @reuvenlax PTAL?

[kanterov]

I don't have time to review before the 10th of June.

[lukecwik]

@kanterov Were you planning to take a look at this?

[kanterov]

Sorry for the delay, I was looking into a couple of things:

• license for the original code
• performance
• compatibility

The code is public domain and can to be used without attribution, in fact, it's already used in Apache, so I assume that this is fine. https://github.com/apache/jackrabbit-oak/blob/trunk/oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/sort/ExternalSort.java#L44-L46

For performance, I'm running a couple of Dataflow jobs that sort 1 billion integers, and so far it looks good.

For compatibility, I checked the source code, and it is source compatible, the default behavior is Hadoop, as it was before, with the possibility to switch to native sorting.

@nevillelyh @clairemcginty @lukecwik let me know if you want to add anything, and I can merge this tomorrow

[kanterov]

Run Java_Examples_Dataflow PreCommit

[kanterov]

сс @lemire

[lemire]

@kanterov Thanks for pinging me. How can I help?

[kanterov]

@lemire thanks, no help is needed, just thought that you might want to know

[lemire]

Indeed. Glad to see our code being used.