Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-7078] [SPARK-7079] Binary processing sort for Spark SQL #6444

Closed
wants to merge 67 commits into from
Closed

[SPARK-7078] [SPARK-7079] Binary processing sort for Spark SQL #6444

wants to merge 67 commits into from

Commits on Jul 6, 2015

  1. WIP towards external sorter for Spark SQL. 

    This is based on an early version of my shuffle sort patch; the
implementation will undergo significant refactoring based on
improvements made as part of the shuffle patch. Stay tuned.
    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    d5d3106 View commit details
    Browse the repository at this point in the history

  2. Import my original tests and get them to pass.

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    2bd8c9a View commit details
    Browse the repository at this point in the history

  3. Merge in a sketch of a unit test for the new sorter (now failing).

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    58f36d0 View commit details
    Browse the repository at this point in the history

  4. Commit some missing code from an old git stash.

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    dda6752 View commit details
    Browse the repository at this point in the history

  5. Remove some debug logging

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    c8792de View commit details
    Browse the repository at this point in the history

  6. Move more code to Java; fix bugs in UnsafeRowConverter length type. 

    The length type is an int, not long, but the code was inconsistent about
this.  I also now use byte arrays instead of long arrays in some places
in order to avoid off-by-factor-of-8 errors.
    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    9cc98f5 View commit details
    Browse the repository at this point in the history

  7. Fix whitespace

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    73cc761 View commit details
    Browse the repository at this point in the history

  8. SparkFunSuite change

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    dfdb93f View commit details
    Browse the repository at this point in the history

  9. Move most of the existing SMJ code into Java.

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    b420a71 View commit details
    Browse the repository at this point in the history

  10. WIP towards copying

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    1b841ca View commit details
    Browse the repository at this point in the history

  11. Back out SMJ operator change; isolate changes to selection of sort op. 

    I'll consider fusing the sort and merge steps in a followup PR.
    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    269cf86 View commit details
    Browse the repository at this point in the history

  12. Update for InternalRow refactoring

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    d468a88 View commit details
    Browse the repository at this point in the history

  13. Port test to SparkPlanTest

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    7eafecf View commit details
    Browse the repository at this point in the history

  14. Back out of BlockObjectWriter change

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    21d7d93 View commit details
    Browse the repository at this point in the history

  15. Update to reflect SparkPlanTest changes

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    62f0bb8 View commit details
    Browse the repository at this point in the history

  16. Back out some Hive changes that aren't needed anymore

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    26c8931 View commit details
    Browse the repository at this point in the history

  17. Add some missing newlines at the ends of files

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    206bfa2 View commit details
    Browse the repository at this point in the history

  18. Harmonization with shuffle's unsafe sorter

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    ebf9eea View commit details
    Browse the repository at this point in the history

  19. Many more changes to harmonize with shuffle sorter

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    1db845a View commit details
    Browse the repository at this point in the history

  20. Fix IntelliJ complaint due to negated if condition

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    82bb0ec View commit details
    Browse the repository at this point in the history

  21. Clean up Exchange code a bit

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    9869ec2 View commit details
    Browse the repository at this point in the history

  22. Centralize logic for picking sort operator implementations

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    6d6a1e6 View commit details
    Browse the repository at this point in the history

  23. Update test name

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    90c2b6a View commit details
    Browse the repository at this point in the history

  24. Get UnsafeInMemorySorterSuite to pass (WIP)

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    41b8881 View commit details
    Browse the repository at this point in the history

  25. Commit failing test demonstrating bug in handling objects in spills

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    7f875f9 View commit details
    Browse the repository at this point in the history

  26. Some WIP work on prefix comparison.

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    6b156fb View commit details
    Browse the repository at this point in the history

  27. Fix consideration of column types when choosing sort implementation.

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    d246e29 View commit details
    Browse the repository at this point in the history

  28. Fix memory leak on empty inputs.

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    6890863 View commit details
    Browse the repository at this point in the history

  29. Add tests for sorting on all primitive types.

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    4c37ba6 View commit details
    Browse the repository at this point in the history

  30. Add missing SortPrefixUtils file

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    95058d9 View commit details
    Browse the repository at this point in the history

  31. Integrate prefix comparators for Int and Long (others coming soon)

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    b310c88 View commit details
    Browse the repository at this point in the history

  32. Prefix comparators for float and double

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    66a813e View commit details
    Browse the repository at this point in the history

  33. Implement prefix sort for strings (albeit inefficiently).

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    0dfe919 View commit details
    Browse the repository at this point in the history

  34. Remove code gen experiment.

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    939f824 View commit details
    Browse the repository at this point in the history

  35. Fix test compilation issue

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    5822e6f View commit details
    Browse the repository at this point in the history

  36. Merge remote-tracking branch 'origin/master' into sql-external-sort

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    9969c14 View commit details
    Browse the repository at this point in the history

  37. Undo part of a SparkPlanTest change in apache#7162 that broke my test.

    @JoshRosen
    JoshRosen committed Jul 6, 2015
    Configuration menu
    Copy the full SHA
    7c3c864 View commit details
    Browse the repository at this point in the history

Commits on Jul 7, 2015

  1. Revert "Undo part of a SparkPlanTest change in apache#7162 that broke… 

    … my test."

This reverts commit 7c3c864.
    @JoshRosen
    JoshRosen committed Jul 7, 2015
    Configuration menu
    Copy the full SHA
    0a79d39 View commit details
    Browse the repository at this point in the history

  2. Fix tests by binding attributes.

    @JoshRosen
    JoshRosen committed Jul 7, 2015
    Configuration menu
    Copy the full SHA
    f27be09 View commit details
    Browse the repository at this point in the history

  3. Test ascending and descending sort orders.

    @JoshRosen
    JoshRosen committed Jul 7, 2015
    Configuration menu
    Copy the full SHA
    88b72db View commit details
    Browse the repository at this point in the history

  4. Force spilling in UnsafeExternalSortSuite.

    @JoshRosen
    JoshRosen committed Jul 7, 2015
    Configuration menu
    Copy the full SHA
    82e21c1 View commit details
    Browse the repository at this point in the history

  5. Fixes to multiple spilling-related bugs.

    @JoshRosen
    JoshRosen committed Jul 7, 2015
    Configuration menu
    Copy the full SHA
    8d7fbe7 View commit details
    Browse the repository at this point in the history

  6. Fix critical issues in test which led to false negatives.

    @JoshRosen
    JoshRosen committed Jul 7, 2015
    Configuration menu
    Copy the full SHA
    87b6ed9 View commit details
    Browse the repository at this point in the history

  7. Fix inconsistent handling / encoding of record lengths.

    @JoshRosen
    JoshRosen committed Jul 7, 2015
    Configuration menu
    Copy the full SHA
    5d6109d View commit details
    Browse the repository at this point in the history

  8. Temporarily enable only the passing sort tests

    @JoshRosen
    JoshRosen committed Jul 7, 2015
    Configuration menu
    Copy the full SHA
    b81a920 View commit details
    Browse the repository at this point in the history

  9. Make sorting of answers explicit in SparkPlanTest.checkAnswer().

    @JoshRosen
    JoshRosen committed Jul 7, 2015
    Configuration menu
    Copy the full SHA
    1c7bad8 View commit details
    Browse the repository at this point in the history

  10. Set global = true in UnsafeExternalSortSuite.

    @JoshRosen
    JoshRosen committed Jul 7, 2015
    Configuration menu
    Copy the full SHA
    b86e684 View commit details
    Browse the repository at this point in the history

  11. Fix prefix comparison of null primitives.

    @JoshRosen
    JoshRosen committed Jul 7, 2015
    Configuration menu
    Copy the full SHA
    08701e7 View commit details
    Browse the repository at this point in the history

  12. Somewhat hacky fix for descending sorts

    @JoshRosen
    JoshRosen committed Jul 7, 2015
    Configuration menu
    Copy the full SHA
    1d7ffaa View commit details
    Browse the repository at this point in the history

  13. Test with larger data.

    @JoshRosen
    JoshRosen committed Jul 7, 2015
    Configuration menu
    Copy the full SHA
    613e16f View commit details
    Browse the repository at this point in the history

  14. NULL_PREFIX has to be negative infinity for floating point types

    @JoshRosen
    JoshRosen committed Jul 7, 2015
    Configuration menu
    Copy the full SHA
    88aff18 View commit details
    Browse the repository at this point in the history

  15. Clean up prefix comparators for integral types

    @JoshRosen
    JoshRosen committed Jul 7, 2015
    Configuration menu
    Copy the full SHA
    9d00afc View commit details
    Browse the repository at this point in the history

  16. Fix bugs in string prefix comparison.

    @JoshRosen
    JoshRosen committed Jul 7, 2015
    Configuration menu
    Copy the full SHA
    f99a612 View commit details
    Browse the repository at this point in the history

  17. Add missing license header.

    @JoshRosen
    JoshRosen committed Jul 7, 2015
    Configuration menu
    Copy the full SHA
    293f109 View commit details
    Browse the repository at this point in the history

Commits on Jul 8, 2015

  1. Merge remote-tracking branch 'origin/master' into sql-external-sort

    @JoshRosen
    JoshRosen committed Jul 8, 2015
    Configuration menu
    Copy the full SHA
    844f4ca View commit details
    Browse the repository at this point in the history

  2. Re-enable NullType sorting test now that SPARK-8868 is fixed

    @JoshRosen
    JoshRosen committed Jul 8, 2015
    Configuration menu
    Copy the full SHA
    d31f180 View commit details
    Browse the repository at this point in the history

  3. Clean up final row copying code.

    @JoshRosen
    JoshRosen committed Jul 8, 2015
    Configuration menu
    Copy the full SHA
    c56ec18 View commit details
    Browse the repository at this point in the history

  4. Remove unnecessary zeroing of row conversion buffer

    @JoshRosen
    JoshRosen committed Jul 8, 2015
    Configuration menu
    Copy the full SHA
    845bea3 View commit details
    Browse the repository at this point in the history

  5. Hacky approach to copying of UnsafeRows for sort followed by limit.

    @JoshRosen
    JoshRosen committed Jul 8, 2015
    Configuration menu
    Copy the full SHA
    d13ac55 View commit details
    Browse the repository at this point in the history

Commits on Jul 9, 2015

  1. Merge remote-tracking branch 'origin/master' into sql-external-sort

    @JoshRosen
    JoshRosen committed Jul 9, 2015
    Configuration menu
    Copy the full SHA
    3947fc1 View commit details
    Browse the repository at this point in the history

  2. Fix scalastyle

    @JoshRosen
    JoshRosen committed Jul 9, 2015
    Configuration menu
    Copy the full SHA
    cd05866 View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2015

  1. Merge remote-tracking branch 'origin/master' into sql-external-sort

    @JoshRosen
    JoshRosen committed Jul 10, 2015
    Configuration menu
    Copy the full SHA
    d1e28bc View commit details
    Browse the repository at this point in the history

  2. Add test and fix bug for sorting empty arrays

    @JoshRosen
    JoshRosen committed Jul 10, 2015
    Configuration menu
    Copy the full SHA
    2f48777 View commit details
    Browse the repository at this point in the history

  3. Fix spill reading for large rows; add test

    @JoshRosen
    JoshRosen committed Jul 10, 2015
    Configuration menu
    Copy the full SHA
    5135200 View commit details
    Browse the repository at this point in the history

  4. Make sortAnswers = false the default in SparkPlanTest

    @JoshRosen
    JoshRosen committed Jul 10, 2015
    Configuration menu
    Copy the full SHA
    35dad9f View commit details
    Browse the repository at this point in the history

  5. Merge remote-tracking branch 'origin/master' into sql-external-sort

    @JoshRosen
    JoshRosen committed Jul 10, 2015
    Configuration menu
    Copy the full SHA
    2bbac9c View commit details
    Browse the repository at this point in the history

  6. Remove a bunch of overloaded methods to avoid default args. issue

    @JoshRosen
    JoshRosen committed Jul 10, 2015
    Configuration menu
    Copy the full SHA
    6beb467 View commit details
    Browse the repository at this point in the history