Skip to content

Commit

Permalink
[SPARK-34012][SQL][3.0] Keep behavior consistent when conf `spark.sql…
Browse files Browse the repository at this point in the history
…legacy.parser.havingWithoutGroupByAsWhere` is true with migration guide

### What changes were proposed in this pull request?
In #22696 we support HAVING without GROUP BY means global aggregate
But since we treat having as Filter before, in this way will cause a lot of analyze error, after #28294 we use `UnresolvedHaving` to instead `Filter` to solve such problem, but break origin logical about treat `SELECT 1 FROM range(10) HAVING true` as `SELECT 1 FROM range(10) WHERE true`   .
This PR fix this issue and add UT.

NOTE: This backport comes from #31039

### Why are the changes needed?
Keep consistent behavior of migration guide.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
added UT

Closes #31049 from AngersZhuuuu/SPARK-34012-3.0.

Authored-by: angerszhu <angers.zhu@gmail.com>
Signed-off-by: Takeshi Yamamuro <yamamuro@apache.org>
  • Loading branch information
AngersZhuuuu authored and maropu committed Jan 6, 2021
1 parent 403bca4 commit aaa3dcc
Show file tree
Hide file tree
Showing 3 changed files with 77 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -723,7 +723,11 @@ class AstBuilder(conf: SQLConf) extends SqlBaseBaseVisitor[AnyRef] with Logging
val withProject = if (aggregationClause == null && havingClause != null) {
if (conf.getConf(SQLConf.LEGACY_HAVING_WITHOUT_GROUP_BY_AS_WHERE)) {
// If the legacy conf is set, treat HAVING without GROUP BY as WHERE.
withHavingClause(havingClause, createProject())
val predicate = expression(havingClause.booleanExpression) match {
case p: Predicate => p
case e => Cast(e, BooleanType)
}
Filter(predicate, createProject())
} else {
// According to SQL standard, HAVING without GROUP BY means global aggregate.
withHavingClause(havingClause, Aggregate(Nil, namedExpressions, withFilter))
Expand Down
10 changes: 10 additions & 0 deletions sql/core/src/test/resources/sql-tests/inputs/group-by.sql
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,16 @@ SELECT 1 FROM range(10) HAVING MAX(id) > 0;

SELECT id FROM range(10) HAVING id > 0;

SET spark.sql.legacy.parser.havingWithoutGroupByAsWhere=true;

SELECT 1 FROM range(10) HAVING true;

SELECT 1 FROM range(10) HAVING MAX(id) > 0;

SELECT id FROM range(10) HAVING id > 0;

SET spark.sql.legacy.parser.havingWithoutGroupByAsWhere=false;

-- Test data
CREATE OR REPLACE TEMPORARY VIEW test_agg AS SELECT * FROM VALUES
(1, true), (1, false),
Expand Down
63 changes: 62 additions & 1 deletion sql/core/src/test/resources/sql-tests/results/group-by.sql.out
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
-- Automatically generated by SQLQueryTestSuite
-- Number of queries: 56
-- Number of queries: 61


-- !query
Expand Down Expand Up @@ -277,6 +277,67 @@ org.apache.spark.sql.AnalysisException
grouping expressions sequence is empty, and '`id`' is not an aggregate function. Wrap '()' in windowing function(s) or wrap '`id`' in first() (or first_value) if you don't care which value you get.;


-- !query
SET spark.sql.legacy.parser.havingWithoutGroupByAsWhere=true
-- !query schema
struct<key:string,value:string>
-- !query output
spark.sql.legacy.parser.havingWithoutGroupByAsWhere true


-- !query
SELECT 1 FROM range(10) HAVING true
-- !query schema
struct<1:int>
-- !query output
1
1
1
1
1
1
1
1
1
1


-- !query
SELECT 1 FROM range(10) HAVING MAX(id) > 0
-- !query schema
struct<>
-- !query output
org.apache.spark.sql.AnalysisException

Aggregate/Window/Generate expressions are not valid in where clause of the query.
Expression in where clause: [(max(`id`) > CAST(0 AS BIGINT))]
Invalid expressions: [max(`id`)];


-- !query
SELECT id FROM range(10) HAVING id > 0
-- !query schema
struct<id:bigint>
-- !query output
1
2
3
4
5
6
7
8
9


-- !query
SET spark.sql.legacy.parser.havingWithoutGroupByAsWhere=false
-- !query schema
struct<key:string,value:string>
-- !query output
spark.sql.legacy.parser.havingWithoutGroupByAsWhere false


-- !query
CREATE OR REPLACE TEMPORARY VIEW test_agg AS SELECT * FROM VALUES
(1, true), (1, false),
Expand Down

0 comments on commit aaa3dcc

Please sign in to comment.