[SPARK-22591][SQL] GenerateOrdering shouldn't change CodegenContext.INPUT_ROW #19800

viirya · 2017-11-23T12:14:09Z

What changes were proposed in this pull request?

When I played with codegen in developing another PR, I found the value of CodegenContext.INPUT_ROW is not reliable. Under wholestage codegen, it is assigned to null first and then suddenly changed to i.

The reason is GenerateOrdering changes CodegenContext.INPUT_ROW but doesn't restore it back.

How was this patch tested?

Added test.

viirya · 2017-11-23T12:17:06Z

cc @cloud-fan @kiszk

cloud-fan · 2017-11-23T12:43:37Z

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/OrderingSuite.scala

@@ -156,4 +156,13 @@ class OrderingSuite extends SparkFunSuite with ExpressionEvalHelper {
      assert(genOrdering.compare(rowB1, rowB2) < 0)
    }
  }
+
+  test("SPARK-22591: GenerateOrdering shouldn't change ctx.INPUT_ROW") {


does it cause any bugs?

I'm working to support wholestage codegen when generating expression codes safe from 64k limit. When there is not INPUT_ROW in context but we wrongly set a INPUT_ROW value, a non-existing InternalRow i will be added into function parameters.

As we use INPUT_ROW a lot in codegen framework, it is risky to change its value without restoring it.

cloud-fan · 2017-11-23T13:19:22Z

...lyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateOrdering.scala

@@ -72,6 +72,7 @@ object GenerateOrdering extends CodeGenerator[Seq[SortOrder], Ordering[InternalR
   * Generates the code for ordering based on the given order.
   */
  def genComparisons(ctx: CodegenContext, ordering: Seq[SortOrder]): String = {
+    val oldInputRow = ctx.INPUT_ROW
    val comparisons = ordering.map { order =>
      val oldCurrentVars = ctx.currentVars


seems we can also move this our of the loop.

how about

val oldInputRow = ... val oldCurrentVars = ... val inputRow = "i" ctx.INPUT_ROW = inputRow ctx.currentVars = null val comparisons = ... ... ctx.INPUT_ROW = oldInputRow ctx.currentVars = oldCurrentVars ...

Ok. Looks good.

SparkQA · 2017-11-23T14:34:53Z

Test build #84128 has finished for PR 19800 at commit 121d309.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2017-11-23T14:39:49Z

This test org.apache.spark.sql.catalyst.expressions.CastSuite.SPARK-22500: cast for struct should not generate codes beyond 64KB seems flaky? I saw it fails with java.lang.OutOfMemoryError: GC overhead limit exceeded in other PR too.

SparkQA · 2017-11-23T16:17:22Z

Test build #84131 has finished for PR 19800 at commit 27145ae.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2017-11-23T16:51:42Z

...lyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateOrdering.scala

+    // Restore original currentVars and INPUT_ROW.
+    ctx.currentVars = oldCurrentVars
+    ctx.INPUT_ROW = oldInputRow
+    finalCode


ctx.currentVars = oldCurrentVars ctx.INPUT_ROW = oldInputRow s""" |InternalRow $inputRow = null; |$code """.stripMargin

Yes, thanks.

cloud-fan · 2017-11-23T16:53:07Z

I created https://issues.apache.org/jira/browse/SPARK-22595 to track the flaky test

SparkQA · 2017-11-24T02:54:25Z

Test build #84144 has finished for PR 19800 at commit 7462a55.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2017-11-24T10:47:29Z

thanks, merging to master/2.2!

…NPUT_ROW ## What changes were proposed in this pull request? When I played with codegen in developing another PR, I found the value of `CodegenContext.INPUT_ROW` is not reliable. Under wholestage codegen, it is assigned to null first and then suddenly changed to `i`. The reason is `GenerateOrdering` changes `CodegenContext.INPUT_ROW` but doesn't restore it back. ## How was this patch tested? Added test. Author: Liang-Chi Hsieh <viirya@gmail.com> Closes #19800 from viirya/SPARK-22591. (cherry picked from commit 62a826f) Signed-off-by: Wenchen Fan <wenchen@databricks.com>

…ruct should not generate codes beyond 64KB This PR reduces the number of fields in the test case of `CastSuite` to fix an issue that is pointed at [here](#19800 (comment)). ``` java.lang.OutOfMemoryError: GC overhead limit exceeded java.lang.OutOfMemoryError: GC overhead limit exceeded at org.codehaus.janino.UnitCompiler.findClass(UnitCompiler.java:10971) at org.codehaus.janino.UnitCompiler.findTypeByName(UnitCompiler.java:7607) at org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:5758) at org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:5732) at org.codehaus.janino.UnitCompiler.access$13200(UnitCompiler.java:206) at org.codehaus.janino.UnitCompiler$18.visitReferenceType(UnitCompiler.java:5668) at org.codehaus.janino.UnitCompiler$18.visitReferenceType(UnitCompiler.java:5660) at org.codehaus.janino.Java$ReferenceType.accept(Java.java:3356) at org.codehaus.janino.UnitCompiler.getType(UnitCompiler.java:5660) at org.codehaus.janino.UnitCompiler.buildLocalVariableMap(UnitCompiler.java:2892) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:2764) at org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1262) at org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1234) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:538) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:890) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:894) at org.codehaus.janino.UnitCompiler.access$600(UnitCompiler.java:206) at org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:377) at org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:369) at org.codehaus.janino.Java$MemberClassDeclaration.accept(Java.java:1128) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369) at org.codehaus.janino.UnitCompiler.compileDeclaredMemberTypes(UnitCompiler.java:1209) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:564) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:890) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:894) at org.codehaus.janino.UnitCompiler.access$600(UnitCompiler.java:206) at org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:377) at org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:369) at org.codehaus.janino.Java$MemberClassDeclaration.accept(Java.java:1128) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369) at org.codehaus.janino.UnitCompiler.compileDeclaredMemberTypes(UnitCompiler.java:1209) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:564) ... ``` Used existing test case Author: Kazuaki Ishizaki <ishizaki@jp.ibm.com> Closes #19806 from kiszk/SPARK-22595. (cherry picked from commit 554adc7) Signed-off-by: Wenchen Fan <wenchen@databricks.com>

…ruct should not generate codes beyond 64KB ## What changes were proposed in this pull request? This PR reduces the number of fields in the test case of `CastSuite` to fix an issue that is pointed at [here](#19800 (comment)). ``` java.lang.OutOfMemoryError: GC overhead limit exceeded java.lang.OutOfMemoryError: GC overhead limit exceeded at org.codehaus.janino.UnitCompiler.findClass(UnitCompiler.java:10971) at org.codehaus.janino.UnitCompiler.findTypeByName(UnitCompiler.java:7607) at org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:5758) at org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:5732) at org.codehaus.janino.UnitCompiler.access$13200(UnitCompiler.java:206) at org.codehaus.janino.UnitCompiler$18.visitReferenceType(UnitCompiler.java:5668) at org.codehaus.janino.UnitCompiler$18.visitReferenceType(UnitCompiler.java:5660) at org.codehaus.janino.Java$ReferenceType.accept(Java.java:3356) at org.codehaus.janino.UnitCompiler.getType(UnitCompiler.java:5660) at org.codehaus.janino.UnitCompiler.buildLocalVariableMap(UnitCompiler.java:2892) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:2764) at org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1262) at org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1234) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:538) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:890) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:894) at org.codehaus.janino.UnitCompiler.access$600(UnitCompiler.java:206) at org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:377) at org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:369) at org.codehaus.janino.Java$MemberClassDeclaration.accept(Java.java:1128) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369) at org.codehaus.janino.UnitCompiler.compileDeclaredMemberTypes(UnitCompiler.java:1209) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:564) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:890) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:894) at org.codehaus.janino.UnitCompiler.access$600(UnitCompiler.java:206) at org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:377) at org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:369) at org.codehaus.janino.Java$MemberClassDeclaration.accept(Java.java:1128) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369) at org.codehaus.janino.UnitCompiler.compileDeclaredMemberTypes(UnitCompiler.java:1209) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:564) ... ``` ## How was this patch tested? Used existing test case Author: Kazuaki Ishizaki <ishizaki@jp.ibm.com> Closes #19806 from kiszk/SPARK-22595.

…NPUT_ROW ## What changes were proposed in this pull request? When I played with codegen in developing another PR, I found the value of `CodegenContext.INPUT_ROW` is not reliable. Under wholestage codegen, it is assigned to null first and then suddenly changed to `i`. The reason is `GenerateOrdering` changes `CodegenContext.INPUT_ROW` but doesn't restore it back. ## How was this patch tested? Added test. Author: Liang-Chi Hsieh <viirya@gmail.com> Closes apache#19800 from viirya/SPARK-22591. (cherry picked from commit 62a826f) Signed-off-by: Wenchen Fan <wenchen@databricks.com>

…ruct should not generate codes beyond 64KB This PR reduces the number of fields in the test case of `CastSuite` to fix an issue that is pointed at [here](apache#19800 (comment)). ``` java.lang.OutOfMemoryError: GC overhead limit exceeded java.lang.OutOfMemoryError: GC overhead limit exceeded at org.codehaus.janino.UnitCompiler.findClass(UnitCompiler.java:10971) at org.codehaus.janino.UnitCompiler.findTypeByName(UnitCompiler.java:7607) at org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:5758) at org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:5732) at org.codehaus.janino.UnitCompiler.access$13200(UnitCompiler.java:206) at org.codehaus.janino.UnitCompiler$18.visitReferenceType(UnitCompiler.java:5668) at org.codehaus.janino.UnitCompiler$18.visitReferenceType(UnitCompiler.java:5660) at org.codehaus.janino.Java$ReferenceType.accept(Java.java:3356) at org.codehaus.janino.UnitCompiler.getType(UnitCompiler.java:5660) at org.codehaus.janino.UnitCompiler.buildLocalVariableMap(UnitCompiler.java:2892) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:2764) at org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1262) at org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1234) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:538) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:890) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:894) at org.codehaus.janino.UnitCompiler.access$600(UnitCompiler.java:206) at org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:377) at org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:369) at org.codehaus.janino.Java$MemberClassDeclaration.accept(Java.java:1128) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369) at org.codehaus.janino.UnitCompiler.compileDeclaredMemberTypes(UnitCompiler.java:1209) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:564) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:890) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:894) at org.codehaus.janino.UnitCompiler.access$600(UnitCompiler.java:206) at org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:377) at org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:369) at org.codehaus.janino.Java$MemberClassDeclaration.accept(Java.java:1128) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369) at org.codehaus.janino.UnitCompiler.compileDeclaredMemberTypes(UnitCompiler.java:1209) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:564) ... ``` Used existing test case Author: Kazuaki Ishizaki <ishizaki@jp.ibm.com> Closes apache#19806 from kiszk/SPARK-22595. (cherry picked from commit 554adc7) Signed-off-by: Wenchen Fan <wenchen@databricks.com>

GenerateOrdering shouldn't change ctx.INPUT_ROW.

121d309

cloud-fan reviewed Nov 23, 2017

View reviewed changes

Address comment.

27145ae

cloud-fan reviewed Nov 23, 2017

View reviewed changes

kiszk mentioned this pull request Nov 23, 2017

[SPARK-22595][SQL] fix flaky test: CastSuite.SPARK-22500: cast for struct should not generate codes beyond 64KB #19806

Closed

Address comment.

7462a55

viirya mentioned this pull request Nov 24, 2017

[SPARK-22600][SQL] Fix 64kb limit for deeply nested expressions under wholestage codegen #19813

Closed

asfgit closed this in 62a826f Nov 24, 2017

viirya deleted the SPARK-22591 branch December 27, 2023 18:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-22591][SQL] GenerateOrdering shouldn't change CodegenContext.INPUT_ROW #19800

[SPARK-22591][SQL] GenerateOrdering shouldn't change CodegenContext.INPUT_ROW #19800

viirya commented Nov 23, 2017

viirya commented Nov 23, 2017

cloud-fan Nov 23, 2017

viirya Nov 23, 2017

viirya Nov 23, 2017

cloud-fan Nov 23, 2017

viirya Nov 23, 2017

cloud-fan Nov 23, 2017 •

edited

Loading

viirya Nov 23, 2017

SparkQA commented Nov 23, 2017

viirya commented Nov 23, 2017

SparkQA commented Nov 23, 2017

cloud-fan Nov 23, 2017

viirya Nov 23, 2017

cloud-fan commented Nov 23, 2017

SparkQA commented Nov 24, 2017

cloud-fan commented Nov 24, 2017

[SPARK-22591][SQL] GenerateOrdering shouldn't change CodegenContext.INPUT_ROW #19800

[SPARK-22591][SQL] GenerateOrdering shouldn't change CodegenContext.INPUT_ROW #19800

Conversation

viirya commented Nov 23, 2017

What changes were proposed in this pull request?

How was this patch tested?

viirya commented Nov 23, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cloud-fan Nov 23, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Nov 23, 2017

viirya commented Nov 23, 2017

SparkQA commented Nov 23, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cloud-fan commented Nov 23, 2017

SparkQA commented Nov 24, 2017

cloud-fan commented Nov 24, 2017

cloud-fan Nov 23, 2017 •

edited

Loading