Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-23208][SQL] Fix code generation for complex create array (related) expressions #20391

Closed
wants to merge 1 commit into from
Closed

[SPARK-23208][SQL] Fix code generation for complex create array (related) expressions #20391

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion ...atalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
View file
Open in desktop
Expand Up @@ -111,7 +111,7 @@ private [sql] object GenArrayData {
val assignmentString = ctx.splitExpressionsWithCurrentInputs(
expressions = assignments,
funcName = "apply",
extraArguments = ("Object[]", arrayDataName) :: Nil)
extraArguments = ("Object[]", arrayName) :: Nil)

(s"Object[] $arrayName = new Object[$numElements];",
assignmentString,
Expand Down
8 changes: 7 additions & 1 deletion sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/complexTypesSuite.scala
View file
Open in desktop
Expand Up @@ -24,14 +24,15 @@ import org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext
import org.apache.spark.sql.catalyst.plans.PlanTest
import org.apache.spark.sql.catalyst.plans.logical.{LocalRelation, LogicalPlan, Range}
import org.apache.spark.sql.catalyst.rules.RuleExecutor
import org.apache.spark.sql.catalyst.util.GenericArrayData
import org.apache.spark.sql.types._

/**
* SPARK-18601 discusses simplification direct access to complex types creators.
* i.e. {{{create_named_struct(square, `x` * `x`).square}}} can be simplified to {{{`x` * `x`}}}.
* sam applies to create_array and create_map
*/
class ComplexTypesSuite extends PlanTest{
class ComplexTypesSuite extends PlanTest with ExpressionEvalHelper {

object Optimizer extends RuleExecutor[LogicalPlan] {
val batches =
Expand Down Expand Up @@ -171,6 +172,11 @@ class ComplexTypesSuite extends PlanTest{
assert(ctx.inlinedMutableStates.length == 0)
}

test("SPARK-23208: Test code splitting for create array related methods") {
val inputs = (1 to 2500).map(x => Literal(s"l_$x"))
checkEvaluation(CreateArray(inputs), new GenericArrayData(inputs.map(_.eval())))
}

test("simplify map ops") {
val rel = relation
.select(
Expand Down