Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-10371] [SQL] Implement subexpr elimination for UnsafeProjections #9480

Closed
wants to merge 5 commits into from
Closed

[SPARK-10371] [SQL] Implement subexpr elimination for UnsafeProjections #9480

wants to merge 5 commits into from

Commits on Nov 9, 2015

  1. [SPARK-10371] [SQL] Implement subexpr elimination for UnsafeProjections 

    This patch adds the building blocks for codegening subexpr elimination and implements
it end to end for UnsafeProjection. The building blocks can be used to do the same thing
for other operators.

It introduces some utilities to compute common sub expressions. Expressions can be added to
this data structure. The expr and its children will be recursively matched against existing
expressions (ones previously added) and grouped into common groups. This is built using
the existing `semanticEquals`. It does not understand things like commutative or associative
expressions. This can be done as future work.

After building this data structure, the codegen process takes advantage of it by:
  1. Generating a helper function in the generated class that computes the common
     subexpression. This is done for all common subexpressions that have at least
     two occurrences and the expression tree is sufficiently complex.
  2. When generating the apply() function, if the helper function exists, call that
     instead of regenerating the expression tree. Repeated calls to the helper function
     shortcircuit the evaluation logic.
    @nongli
    nongli committed Nov 9, 2015
    Configuration menu
    Copy the full SHA
    f3ab2bc View commit details
    Browse the repository at this point in the history

  2. Code review comments from davies.

    @nongli
    nongli committed Nov 9, 2015
    Configuration menu
    Copy the full SHA
    05b59e6 View commit details
    Browse the repository at this point in the history

  3. Added a few more test cases.

    @nongli
    nongli committed Nov 9, 2015
    Configuration menu
    Copy the full SHA
    fa7aa9d View commit details
    Browse the repository at this point in the history

  4. Clean up from CR.

    @nongli
    nongli committed Nov 9, 2015
    Configuration menu
    Copy the full SHA
    a9bcdb0 View commit details
    Browse the repository at this point in the history

  5. Add flag to disable.

    @nongli
    nongli committed Nov 9, 2015
    Configuration menu
    Copy the full SHA
    6cf0186 View commit details
    Browse the repository at this point in the history