Skip to content

[SPARK-42044][SQL] Fix incorrect error message for MUST_AGGREGATE_CORRELATED_SCALAR_SUBQUERY #39543

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
itholic wants to merge 5 commits into from
Closed

[SPARK-42044][SQL] Fix incorrect error message for MUST_AGGREGATE_CORRELATED_SCALAR_SUBQUERY #39543

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9b83393
[SPARK-42044][SQL] Fix wrong error message for `MUST_AGGREGATE_CORREL…
itholic Jan 13, 2023
4c98b74
Merge branch 'master' of https://github.com/apache/spark into MUST_AG…
itholic Jan 16, 2023
c823651
fix test
itholic Jan 16, 2023
1668340
resolve
itholic Jan 23, 2023
39cad50
Merge branch 'master' of https://github.com/apache/spark into MUST_AG…
itholic Jan 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion core/src/main/resources/error/error-classes.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1667,7 +1667,7 @@
},
"MUST_AGGREGATE_CORRELATED_SCALAR_SUBQUERY" : {
"message" : [
"Correlated scalar subqueries in the GROUP BY clause must also be in the aggregate expressions<treeNode>."
"Correlated scalar subqueries must be aggregated to return at most one row."
Copy link
Contributor

@cloud-fan cloud-fan Jan 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not related to this PR, but how is this different from MUST_AGGREGATE_CORRELATED_SCALAR_SUBQUERY_OUTPUT?

Copy link
Contributor Author

@itholic itholic Jan 21, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like we can integrate it into MUST_AGGREGATE_CORRELATED_SCALAR_SUBQUERY.

I don't see any difference between two errors.

Let me handle it in separate ticket just in case if someone could argue.

Copy link
Contributor

@cloud-fan cloud-fan Jan 30, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SGTM. Is the ticket created?

Copy link
Contributor Author

@itholic itholic Jan 30, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is the ticket just created: https://issues.apache.org/jira/browse/SPARK-42239

Let me address this soon.

Copy link
Contributor Author

@itholic itholic Jan 30, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just opened the PR: #39806

]
},
"MUST_AGGREGATE_CORRELATED_SCALAR_SUBQUERY_OUTPUT" : {
Expand Down
4 changes: 2 additions & 2 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -947,7 +947,7 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog with QueryErrorsB
expr.failAnalysis(
errorClass = "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
"MUST_AGGREGATE_CORRELATED_SCALAR_SUBQUERY",
messageParameters = Map("treeNode" -> planToString(other)))
messageParameters = Map.empty)
}

// Only certain operators are allowed to host subquery expression containing
Expand All @@ -962,7 +962,7 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog with QueryErrorsB
a.failAnalysis(
errorClass = "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
"MUST_AGGREGATE_CORRELATED_SCALAR_SUBQUERY",
messageParameters = Map("treeNode" -> planToString(a)))
messageParameters = Map.empty)
}
case other =>
other.failAnalysis(
Expand Down
7 changes: 5 additions & 2 deletions sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisErrorSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -934,8 +934,11 @@ class AnalysisErrorSuite extends AnalysisTest {
t.as("t2")))
) :: Nil,
sum($"c2").as("sum") :: Nil, t.as("t1"))
assertAnalysisError(plan, "Correlated scalar subqueries in the group by clause must also be " +
"in the aggregate expressions" :: Nil)
assertAnalysisErrorClass(
plan,
expectedErrorClass =
"UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY.MUST_AGGREGATE_CORRELATED_SCALAR_SUBQUERY",
expectedMessageParameters = Map.empty)
}

test("SPARK-34946: correlated scalar subquery in aggregate expressions only") {
Expand Down
5 changes: 2 additions & 3 deletions sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -545,12 +545,11 @@ class SubquerySuite extends QueryTest
val exception1 = intercept[AnalysisException] {
sql("select a, (select b from l l2 where l2.a = l1.a) sum_b from l l1")
}
checkErrorMatchPVals(
checkError(
exception1,
errorClass = "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
"MUST_AGGREGATE_CORRELATED_SCALAR_SUBQUERY",
parameters = Map("treeNode" -> "(?s)Filter .*"),
sqlState = None,
parameters = Map.empty,
context = ExpectedContext(
fragment = "(select b from l l2 where l2.a = l1.a)", start = 10, stop = 47))
val exception2 = intercept[AnalysisException] {
Expand Down