[SPARK-15825][SQL] Fix SMJ invalid results

[hvanhovell]

What changes were proposed in this pull request?

Code generated SortMergeJoin failed with wrong results when using structs as keys. This could (eventually) be traced back to the use of a wrong row reference when comparing structs.

How was this patch tested?

TBD

[SparkQA]

Test build #60268 has finished for PR 13589 at commit 995c86a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

cc @davies

mystery of INPUT_ROW...

[kiszk]

Is it better to use InternalRow $INPUT_ROW = null; in an assignment for funcCode, to clearly show intention of this change?

[hvanhovell]

@kiszk i is set to null here: https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala#L498

I'd rather not change this to INPUT_ROW since this could potentially make things even more confusing.

[a-roberts]

Still getting seg faults on Power and with Intel on OpenJDK with this change when we use spark-submit with two worker instances.

Git diff

--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
@@ -490,6 +490,7 @@ class CodegenContext {
addNewFunction(compareFunc, funcCode)
s"this.$compareFunc($c1, $c2)"
case schema: StructType =>

•  INPUT_ROW = "i"
  

  val comparisons = GenerateOrdering.genComparisons(this, schema)
  val compareFunc = freshName("compareStruct")
  val funcCode: String =

segv occurs four times

cat work/app-20160610093117-0000/*/stderr | grep -C 2 "Segmentation error"
16/06/10 09:34:27 INFO CodeGenerator: Code generated in 115.554506 ms
Unhandled exception
Type=Segmentation error vmState=0x00000000
J9Generic_Signal_Number=00000004 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
Handler1=00003FFF7B487AA0 Handler2=00003FFF7B312AC0

spark-env.sh has
export SPARK_WORKER_CORES=2
export SPARK_WORKER_INSTANCES=2
no settings for going off-heap

[robbinspg]

As Adam says I still get the segv with OpenJDK on linux amd64 running our app. This fix does appear to fix the issue reported in https://issues.apache.org/jira/browse/SPARK-15825

[SparkQA]

Test build #60302 has finished for PR 13589 at commit 08cc611.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[davies]

This line

InternalRow i = null;

requires that INPUT_ROW should be i (could be changed to a refresh name). The fix looks reasonable to me.

[davies]

Merging this into master and 2.0, thanks!