Skip to content
Permalink
Browse files

[SPARK-25371][SQL][BACKPORT-2.3] struct() should allow being called w…

…ith 0 args

## What changes were proposed in this pull request?

SPARK-21281 introduced a check for the inputs of `CreateStructLike` to be non-empty. This means that `struct()`, which was previously considered valid, now throws an Exception.  This behavior change was introduced in 2.3.0. The change may break users' application on upgrade and it causes `VectorAssembler` to fail when an empty `inputCols` is defined.

The PR removes the added check making `struct()` valid again.

## How was this patch tested?

added UT

Closes #22391 from mgaido91/SPARK-25371_2.3.

Authored-by: Marco Gaido <marcogaido91@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
  • Loading branch information...
mgaido91 authored and cloud-fan committed Sep 12, 2018
1 parent 18688d3 commit d8ec5ff7502451f61f2be3871d9bd1461b26accd
@@ -147,4 +147,10 @@ class VectorAssemblerSuite
.filter(vectorUDF($"features") > 1)
.count() == 1)
}

test("SPARK-25371: VectorAssembler with empty inputCols") {
val vectorAssembler = new VectorAssembler().setInputCols(Array()).setOutputCol("a")
val output = vectorAssembler.transform(Seq(1).toDF("x"))
assert(output.select("a").limit(1).collect().head == Row(Vectors.sparse(0, Seq.empty)))
}
}
@@ -305,10 +305,7 @@ trait CreateNamedStructLike extends Expression {
}

override def checkInputDataTypes(): TypeCheckResult = {
if (children.length < 1) {
TypeCheckResult.TypeCheckFailure(
s"input to function $prettyName requires at least one argument")
} else if (children.size % 2 != 0) {
if (children.size % 2 != 0) {
TypeCheckResult.TypeCheckFailure(s"$prettyName expects an even number of arguments.")
} else {
val invalidNames = nameExprs.filterNot(e => e.foldable && e.dataType == StringType)
@@ -465,8 +465,6 @@ class DataFrameFunctionsSuite extends QueryTest with SharedSQLContext {
val funcsMustHaveAtLeastOneArg =
("coalesce", (df: DataFrame) => df.select(coalesce())) ::
("coalesce", (df: DataFrame) => df.selectExpr("coalesce()")) ::
("named_struct", (df: DataFrame) => df.select(struct())) ::
("named_struct", (df: DataFrame) => df.selectExpr("named_struct()")) ::
("hash", (df: DataFrame) => df.select(hash())) ::
("hash", (df: DataFrame) => df.selectExpr("hash()")) :: Nil
funcsMustHaveAtLeastOneArg.foreach { case (name, func) =>

0 comments on commit d8ec5ff

Please sign in to comment.
You can’t perform that action at this time.