Skip to content

Commit

Permalink
[SPARK-23208][SQL] Fix code generation for complex create array (rela…
Browse files Browse the repository at this point in the history
…ted) expressions

## What changes were proposed in this pull request?
The `GenArrayData.genCodeToCreateArrayData` produces illegal java code when code splitting is enabled. This is used in `CreateArray` and `CreateMap` expressions for complex object arrays.

This issue is caused by a typo.

## How was this patch tested?
Added a regression test in `complexTypesSuite`.

Author: Herman van Hovell <hvanhovell@databricks.com>

Closes #20391 from hvanhovell/SPARK-23208.
  • Loading branch information
hvanhovell authored and cloud-fan committed Jan 25, 2018
1 parent 45b4bbf commit e29b08a
Show file tree
Hide file tree
Showing 2 changed files with 8 additions and 2 deletions.
Expand Up @@ -111,7 +111,7 @@ private [sql] object GenArrayData {
val assignmentString = ctx.splitExpressionsWithCurrentInputs(
expressions = assignments,
funcName = "apply",
extraArguments = ("Object[]", arrayDataName) :: Nil)
extraArguments = ("Object[]", arrayName) :: Nil)

(s"Object[] $arrayName = new Object[$numElements];",
assignmentString,
Expand Down
Expand Up @@ -24,14 +24,15 @@ import org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext
import org.apache.spark.sql.catalyst.plans.PlanTest
import org.apache.spark.sql.catalyst.plans.logical.{LocalRelation, LogicalPlan, Range}
import org.apache.spark.sql.catalyst.rules.RuleExecutor
import org.apache.spark.sql.catalyst.util.GenericArrayData
import org.apache.spark.sql.types._

/**
* SPARK-18601 discusses simplification direct access to complex types creators.
* i.e. {{{create_named_struct(square, `x` * `x`).square}}} can be simplified to {{{`x` * `x`}}}.
* sam applies to create_array and create_map
*/
class ComplexTypesSuite extends PlanTest{
class ComplexTypesSuite extends PlanTest with ExpressionEvalHelper {

object Optimizer extends RuleExecutor[LogicalPlan] {
val batches =
Expand Down Expand Up @@ -171,6 +172,11 @@ class ComplexTypesSuite extends PlanTest{
assert(ctx.inlinedMutableStates.length == 0)
}

test("SPARK-23208: Test code splitting for create array related methods") {
val inputs = (1 to 2500).map(x => Literal(s"l_$x"))
checkEvaluation(CreateArray(inputs), new GenericArrayData(inputs.map(_.eval())))
}

test("simplify map ops") {
val rel = relation
.select(
Expand Down

0 comments on commit e29b08a

Please sign in to comment.