Skip to content

Commit

Permalink
[SPARK-40615][SQL][TESTS][FOLLOW-UP] Make the test pass with ANSI ena…
Browse files Browse the repository at this point in the history
…bled

### What changes were proposed in this pull request?

This PR proposes the make the tests added in apache#38050 pass with ANSI mode enabled by avoiding string binary operations.

### Why are the changes needed?

To make the tests pass with ANSI enabled on. Currently, it fails as below (https://github.com/apache/spark/actions/runs/3286184541/jobs/5414029918):

```
[info] - SPARK-40615: Check unsupported data type when decorrelating subqueries *** FAILED *** (118 milliseconds)
[info]   "[DATATYPE_MISMATCH.BINARY_OP_WRONG_TYPE] Cannot resolve "(a + a)" due to data type mismatch: the binary operator requires the input type ("NUMERIC" or "INTERVAL DAY TO SECOND" or "INTERVAL YEAR TO MONTH" or "INTERVAL"), not "STRING".; line 1 pos 15;
[info]   'Project [unresolvedalias(scalar-subquery#426412 [], None)]
[info]   :  +- 'Project [unresolvedalias((a#426411 + a#426411), None)]
[info]   :     +- SubqueryAlias __auto_generated_subquery_name
[info]   :        +- Project [upper(cast(outer(x#426413)[a] as string)) AS a#426411]
[info]   :           +- OneRowRelation
[info]   +- SubqueryAlias v1
[info]      +- View (`v1`, [x#426413])
[info]         +- Project [cast(x#426414 as map<string,int>) AS x#426413]
[info]            +- SubqueryAlias t
[info]               +- LocalRelation [x#426414]
[info]   " did not contain "Correlated column reference 'v1.x' cannot be map type" (SubquerySuite.scala:2480)
[info]   org.scalatest.exceptions.TestFailedException:
[info]   at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
[info]   at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
[info]   at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)
[info]   at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295)
[info]   at org.apache.spark.sql.SubquerySuite.$anonfun$new$320(SubquerySuite.scala:2480)
[info]   at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[info]   at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1491)
[info]   at org.apache.spark.sql.test.SQLTestUtilsBase.withTempView(SQLTestUtils.scala:276)
[info]   at org.apache.spark.sql.test.SQLTestUtilsBase.withTempView$(SQLTestUtils.scala:274)
[info]   at org.apache.spark.sql.SubquerySuite.withTempView(SubquerySuite.scala:32)
[info]   at org.apache.spark.sql.SubquerySuite.$anonfun$new$319(SubquerySuite.scala:2459)
[info]   at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
```

### Does this PR introduce _any_ user-facing change?

No, test-only.

### How was this patch tested?

Manually ran the tests and verified that it passes.

Closes apache#38325 from HyukjinKwon/SPARK-40615-followup.

Authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
  • Loading branch information
HyukjinKwon authored and SandishKumarHN committed Dec 12, 2022
1 parent 27fb670 commit 6bb69a5
Showing 1 changed file with 1 addition and 6 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -2462,11 +2462,6 @@ class SubquerySuite extends QueryTest
|create temp view v1(x) as values
|from_json('{"a":1, "b":2}', 'map<string,int>') t(x)
|""".stripMargin)
sql(
"""
|create temp view v2(x) as values
|from_json('{"b":0, "c":2}', 'map<string,int>') t(x)
|""".stripMargin)

// Can use non-orderable data type in one row subquery that can be collapsed.
checkAnswer(
Expand All @@ -2475,7 +2470,7 @@ class SubquerySuite extends QueryTest

// Cannot use non-orderable data type in one row subquery that cannot be collapsed.
val error = intercept[AnalysisException] {
sql("select (select a + a from (select upper(x['a']) as a)) from v1").collect()
sql("select (select concat(a, a) from (select upper(x['a']) as a)) from v1").collect()
}
assert(error.getMessage.contains("Correlated column reference 'v1.x' cannot be map type"))
}
Expand Down

0 comments on commit 6bb69a5

Please sign in to comment.