[SPARK-37641][SQL] Support ANSI Aggregate Function: regr_r2 #34894

beliefer · 2021-12-14T08:47:33Z

What changes were proposed in this pull request?

This PR used to support ANSI aggregate Function: regr_r2

Syntax: REGR_R2(y, x)
Arguments:

y:The dependent variable. This must be an expression that can be evaluated to a numeric type.
x:The independent variable. This must be an expression that can be evaluated to a numeric type.

Examples:
select k, regr_r2(v, v2) from aggr group by k;

k	regr_r2(v, v2)
1	[NULL]
2	0.9976905312

The mainstream database supports regr_r2 show below:
Teradata
https://docs.teradata.com/r/756LNiPSFdY~4JcCCcR5Cw/exhFe2f_YyGqKFakYYUn2A
Snowflake
https://docs.snowflake.com/en/sql-reference/functions/regr_r2.html
Oracle
https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlrf/REGR_-Linear-Regression-Functions.html#GUID-A675B68F-2A88-4843-BE2C-FCDE9C65F9A9
DB2
https://www.ibm.com/docs/en/db2/11.5?topic=af-regression-functions-regr-avgx-regr-avgy-regr-count
H2
http://www.h2database.com/html/functions-aggregate.html#regr_r2
Postgresql
https://www.postgresql.org/docs/8.4/functions-aggregate.html
Sybase
https://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.help.sqlanywhere.12.0.0/dbreference/regr-r2-function.html
Exasol
https://docs.exasol.com/sql_references/functions/alphabeticallistfunctions/regr_function.htm

Why are the changes needed?

regr_r2 is very useful.

Does this PR introduce any user-facing change?

'Yes'. New feature.

How was this patch tested?

New tests.

SparkQA · 2021-12-14T09:46:11Z

Test build #146176 has finished for PR 34894 at commit 267e94c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class RegrR2(x: Expression, y: Expression)

SparkQA · 2021-12-14T10:06:32Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50649/

SparkQA · 2021-12-14T10:52:20Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50649/

SparkQA · 2021-12-14T11:09:58Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50652/

SparkQA · 2021-12-14T12:13:20Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50652/

SparkQA · 2021-12-14T14:43:19Z

Test build #146179 has finished for PR 34894 at commit afe3707.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

beliefer · 2021-12-14T15:22:13Z

retest this please

SparkQA · 2021-12-14T16:20:08Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50661/

SparkQA · 2021-12-14T17:08:12Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50661/

SparkQA · 2021-12-14T20:20:37Z

Test build #146187 has finished for PR 34894 at commit afe3707.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-20T08:24:52Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50866/

SparkQA · 2021-12-20T09:35:46Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50866/

SparkQA · 2021-12-20T12:35:04Z

Test build #146391 has finished for PR 34894 at commit dab7c93.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

beliefer · 2022-02-27T01:51:46Z

ping @MaxGekk @gengliangwang cc @cloud-fan

cloud-fan · 2022-03-08T13:51:59Z

...st/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/linearRegression.scala

+case class RegrR2(x: Expression, y: Expression) extends PearsonCorrelation(x, y, true) {
+  override def prettyName: String = "regr_r2"
+  override val evaluateExpression: Expression = {
+    lazy val corr = ck / sqrt(xMk * yMk)


We are building expressions here, not calculating the result. I don't think this lazy val is necessary.

I actually think it should be good to make sure we won't calculate the corr value if xMk or yMk is 0.

nvm it's just Expression.

cloud-fan · 2022-03-08T13:53:57Z

...st/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/linearRegression.scala

+  override val evaluateExpression: Expression = {
+    lazy val corr = ck / sqrt(xMk * yMk)
+    If(n === 0.0, Literal.create(null, DoubleType),
+      If(n === 1.0, divideByZeroEvalResult, corr * corr))


divideByZeroEvalResult is for legacy behaviors, which we don't need to follow in new functions.

BTW, is there any reference implementation or algorithm introduction to explain this corr * corr?

We know how to evaluate corr, reference

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Corr.scala

Line 126 in 52e7602

If(n === 1.0, divideByZeroEvalResult, ck / sqrt(xMk * yMk)))

jiangxb1987 · 2022-03-25T03:12:49Z

...st/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/linearRegression.scala

+
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+  usage = "_FUNC_(y, x) - Returns the number of non-null number pairs in a group, where `y` is the dependent variable and `x` is the independent variable.",


The function description is not correct, please update.

jiangxb1987 · 2022-03-25T03:27:46Z

...st/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/linearRegression.scala

+  override val evaluateExpression: Expression = {
+    val corr = ck / sqrt(xMk * yMk)
+    If(xMk === 0.0, Literal.create(null, DoubleType),
+      If(yMk === 0.0, Literal.create(1.0, DoubleType), corr * corr))


Should be better to compute the corr here. So you don't compute the value when xMk or yMk is 0.

nvm it's just Expression.

Yes. there are expressions here.

jiangxb1987 · 2022-03-28T18:05:38Z

The changes LGTM, cc @cloud-fan can you confirm?

cloud-fan · 2022-03-29T06:58:12Z

thanks, merging to master/3.3!

### What changes were proposed in this pull request? This PR used to support ANSI aggregate Function: `regr_r2` **Syntax**: REGR_R2(y, x) **Arguments**: - **y**:The dependent variable. This must be an expression that can be evaluated to a numeric type. - **x**:The independent variable. This must be an expression that can be evaluated to a numeric type. **Examples**: `select k, regr_r2(v, v2) from aggr group by k;` | k | regr_r2(v, v2) | |--|---------------| | 1 | [NULL] | | 2 | 0.9976905312 | The mainstream database supports `regr_r2` show below: **Teradata** https://docs.teradata.com/r/756LNiPSFdY~4JcCCcR5Cw/exhFe2f_YyGqKFakYYUn2A **Snowflake** https://docs.snowflake.com/en/sql-reference/functions/regr_r2.html **Oracle** https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlrf/REGR_-Linear-Regression-Functions.html#GUID-A675B68F-2A88-4843-BE2C-FCDE9C65F9A9 **DB2** https://www.ibm.com/docs/en/db2/11.5?topic=af-regression-functions-regr-avgx-regr-avgy-regr-count **H2** http://www.h2database.com/html/functions-aggregate.html#regr_r2 **Postgresql** https://www.postgresql.org/docs/8.4/functions-aggregate.html **Sybase** https://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.help.sqlanywhere.12.0.0/dbreference/regr-r2-function.html **Exasol** https://docs.exasol.com/sql_references/functions/alphabeticallistfunctions/regr_function.htm ### Why are the changes needed? `regr_r2` is very useful. ### Does this PR introduce _any_ user-facing change? 'Yes'. New feature. ### How was this patch tested? New tests. Closes #34894 from beliefer/SPARK-37641. Authored-by: Jiaan Geng <beliefer@163.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit b01d81e) Signed-off-by: Wenchen Fan <wenchen@databricks.com>

beliefer · 2022-03-29T08:07:01Z

@jiangxb1987 Thank you for the review. @cloud-fan Thank you too.

github-actions bot added the SQL label Dec 14, 2021

beliefer force-pushed the SPARK-37641 branch from dab7c93 to c1b22f7 Compare February 26, 2022 12:20

beliefer changed the title ~~[WIP][SPARK-37641][SQL] Support ANSI Aggregate Function: regr_r2~~ [SPARK-37641][SQL] Support ANSI Aggregate Function: regr_r2 Feb 27, 2022

cloud-fan reviewed Mar 8, 2022

View reviewed changes

beliefer force-pushed the SPARK-37641 branch 2 times, most recently from a8fc177 to 4925a76 Compare March 13, 2022 02:38

beliefer force-pushed the SPARK-37641 branch 3 times, most recently from 96023b4 to dd399a6 Compare March 24, 2022 01:40

jiangxb1987 reviewed Mar 25, 2022

View reviewed changes

beliefer added 5 commits March 26, 2022 13:48

[SPARK-37641][SQL] Support ANSI Aggregate Function: regr_r2

4174a17

Update code

c3aa990

Update code

51b67da

Update code

35cc9e4

Update code

823b5c5

beliefer added 3 commits March 26, 2022 13:48

Update code

d8a924b

Update code

457d098

Update code

b76ae64

beliefer force-pushed the SPARK-37641 branch from f441b44 to b76ae64 Compare March 26, 2022 05:54

cloud-fan approved these changes Mar 29, 2022

View reviewed changes

cloud-fan closed this in b01d81e Mar 29, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-37641][SQL] Support ANSI Aggregate Function: regr_r2 #34894

[SPARK-37641][SQL] Support ANSI Aggregate Function: regr_r2 #34894

beliefer commented Dec 14, 2021 •

edited

SparkQA commented Dec 14, 2021

SparkQA commented Dec 14, 2021

SparkQA commented Dec 14, 2021

SparkQA commented Dec 14, 2021

SparkQA commented Dec 14, 2021

SparkQA commented Dec 14, 2021

beliefer commented Dec 14, 2021

SparkQA commented Dec 14, 2021

SparkQA commented Dec 14, 2021

SparkQA commented Dec 14, 2021

SparkQA commented Dec 20, 2021

SparkQA commented Dec 20, 2021

SparkQA commented Dec 20, 2021

beliefer commented Feb 27, 2022

cloud-fan Mar 8, 2022 •

edited

beliefer Mar 9, 2022

jiangxb1987 Mar 25, 2022

jiangxb1987 Mar 25, 2022

cloud-fan Mar 8, 2022

cloud-fan Mar 8, 2022

beliefer Mar 9, 2022

jiangxb1987 Mar 25, 2022

jiangxb1987 Mar 25, 2022

jiangxb1987 Mar 25, 2022

beliefer Mar 25, 2022

jiangxb1987 commented Mar 28, 2022

cloud-fan commented Mar 29, 2022

beliefer commented Mar 29, 2022

[SPARK-37641][SQL] Support ANSI Aggregate Function: regr_r2 #34894

[SPARK-37641][SQL] Support ANSI Aggregate Function: regr_r2 #34894

Conversation

beliefer commented Dec 14, 2021 • edited

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented Dec 14, 2021

SparkQA commented Dec 14, 2021

SparkQA commented Dec 14, 2021

SparkQA commented Dec 14, 2021

SparkQA commented Dec 14, 2021

SparkQA commented Dec 14, 2021

beliefer commented Dec 14, 2021

SparkQA commented Dec 14, 2021

SparkQA commented Dec 14, 2021

SparkQA commented Dec 14, 2021

SparkQA commented Dec 20, 2021

SparkQA commented Dec 20, 2021

SparkQA commented Dec 20, 2021

beliefer commented Feb 27, 2022

cloud-fan Mar 8, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jiangxb1987 commented Mar 28, 2022

cloud-fan commented Mar 29, 2022

beliefer commented Mar 29, 2022

beliefer commented Dec 14, 2021 •

edited

cloud-fan Mar 8, 2022 •

edited