[SPARK-27883][SQL] Port AGGREGATES.sql [Part 2]

[wangyum]

What changes were proposed in this pull request?

This PR is to port AGGREGATES.sql from PostgreSQL regression tests. https://github.com/postgres/postgres/blob/REL_12_BETA1/src/test/regress/sql/aggregates.sql#L145-L350

The expected results can be found in the link: https://github.com/postgres/postgres/blob/REL_12_BETA1/src/test/regress/expected/aggregates.out#L499-L984

When porting the test cases, found four PostgreSQL specific features that do not exist in Spark SQL:

• SPARK-27877: Implement SQL-standard LATERAL subqueries
• SPARK-27878: Support ARRAY(sub-SELECT) expressions
• SPARK-27879: Implement bitwise integer aggregates(BIT_AND and BIT_OR)
• SPARK-27880: Implement boolean aggregates(BOOL_AND, BOOL_OR and EVERY)

How was this patch tested?

N/A

[wangyum]

Do we need to deal with booland_statefunc and boolor_statefunc? These 2 functions are not exist in doc: https://www.postgresql.org/docs/12/functions-aggregate.html

[gatorsmile]

  288  * Function for standard EVERY aggregate conforming to SQL 2003.
  289  * The aggregate is also named bool_and for consistency.
  290  *
  291  * Note: this is only used in plain aggregate mode, not moving-aggregate mode.

See: https://doxygen.postgresql.org/bool_8c_source.html

[gatorsmile]

BooleanType and BooleanType is equivalent to bool bool_and bool?

[wangyum]

Yes. it's equivalent. BOOL_AND and booland_statefunc, BOOL_OR and boolor_statefunc.

[gatorsmile]

Can we rewrite the queries by using AND on the columns with boolean data types?

[wangyum]

Do you mean:

select NULL and NULL is null AS "t"

?

[gatorsmile]

yes.

[wangyum]

The behaviour different:

1. Spark SQL:

SELECT
  -- boolean and transitions
  -- null because strict
  NULL and NULL IS NULL AS `t`,
  TRUE and NULL IS NULL AS `t`,
  FALSE and NULL IS NULL AS `t`,
  NULL and TRUE IS NULL AS `t`,
  NULL and FALSE IS NULL AS `t`,
  -- and actual computations
  TRUE and TRUE AS `t`,
  NOT TRUE and FALSE AS `t`,
  NOT FALSE and TRUE AS `t`,
  NOT FALSE and FALSE AS `t`
-- !query 1 schema
struct<t:boolean,t:boolean,t:boolean,t:boolean,t:boolean,t:boolean,t:boolean,t:boolean,t:boolean>
-- !query 1 output
NULL	true	false	false	false	true	false	true	false

PostgreSQL:

SELECT
  -- boolean and transitions
  -- null because strict
  booland_statefunc(NULL, NULL)  IS NULL AS "t",
  booland_statefunc(TRUE, NULL)  IS NULL AS "t",
  booland_statefunc(FALSE, NULL) IS NULL AS "t",
  booland_statefunc(NULL, TRUE)  IS NULL AS "t",
  booland_statefunc(NULL, FALSE) IS NULL AS "t",
  -- and actual computations
  booland_statefunc(TRUE, TRUE) AS "t",
  NOT booland_statefunc(TRUE, FALSE) AS "t",
  NOT booland_statefunc(FALSE, TRUE) AS "t",
  NOT booland_statefunc(FALSE, FALSE) AS "t";
 t | t | t | t | t | t | t | t | t 
---+---+---+---+---+---+---+---+---
 t | t | t | t | t | t | t | t | t
(1 row)

[gatorsmile]

You need to add parenthesis.

%sql
SELECT
  -- boolean and transitions
  -- null because strict
  (NULL and NULL) IS NULL AS `t`,
  (TRUE and NULL) IS NULL AS `t`,
  (FALSE and NULL) IS NULL AS `t`,
  (NULL and TRUE) IS NULL AS `t`,
  (NULL and FALSE) IS NULL AS `t`,
  -- and actual computations
  TRUE and TRUE AS `t`,
  NOT (TRUE and FALSE) AS `t`,
  NOT (FALSE and TRUE) AS `t`,
  NOT (FALSE and FALSE) AS `t`

You will still see the difference. That is because of the strict mode.

Based on the doc of postgreSQL https://www.postgresql.org/docs/9.5/sql-createfunction.html
, for strict mode,

STRICT indicates that the function always returns null whenever any of its arguments are null. If this parameter is specified, the function is not executed when there are null arguments; instead a null result is assumed automatically.

Our AND does not have a strict mode. It is fine to have a behavior difference, but you need to update the comment to explain it.

[wangyum]

Thank you, I see.

[SparkQA]

Test build #105964 has finished for PR 24743 at commit b585463.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Retest this please.

[SparkQA]

Test build #105979 has finished for PR 24743 at commit b585463.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[wangyum]

@dongjoon-hyun The test will not pass because of #24743 (comment)

[dongjoon-hyun]

Got it, @wangyum !

[SparkQA]

Test build #106052 has finished for PR 24743 at commit ee63c37.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #106231 has finished for PR 24743 at commit 41d4e0a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gatorsmile]

LGTM

Thanks! Merged to master