[SPARK-15776][SQL] Divide Expression inside Aggregation function is casted to wrong type #13651

clockfly · 2016-06-13T22:18:44Z

What changes were proposed in this pull request?

This PR fixes the problem that Divide Expression inside Aggregation function is casted to wrong type, which cause select 1/2 and select sum(1/2)returning different result.

Before the change:

scala> sql("select 1/2 as a").show()
+---+
|  a|
+---+
|0.5|
+---+

scala> sql("select sum(1/2) as a").show()
+---+
|  a|
+---+
|0  |
+---+

scala> sql("select sum(1 / 2) as a").schema
res4: org.apache.spark.sql.types.StructType = StructType(StructField(a,LongType,true))

After the change:

scala> sql("select 1/2 as a").show()
+---+
|  a|
+---+
|0.5|
+---+

scala> sql("select sum(1/2) as a").show()
+---+
|  a|
+---+
|0.5|
+---+

scala> sql("select sum(1/2) as a").schema
res4: org.apache.spark.sql.types.StructType = StructType(StructField(a,DoubleType,true))

How was this patch tested?

Unit test.

This PR is based on #13524 by @Sephiroth-Lin

…d to wrong type

cloud-fan · 2016-06-13T22:23:10Z

sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala

@@ -2847,4 +2847,15 @@ class SQLQuerySuite extends QueryTest with SharedSQLContext {
  test("SPARK-15887: hive-site.xml should be loaded") {
    assert(spark.sessionState.newHadoopConf().get("hive.in.test") == "true")
  }
+
+  test("SPARK-15776 Divide expression inside an Aggregation function should not " +


I think we need some low level unit test instead of end-to-end test

@cloud-fan

It involves Analyzer, low level unit test is hard to cover that?

TypeCoercionSuite

SparkQA · 2016-06-14T00:01:46Z

Test build #60439 has finished for PR 13651 at commit df08eea.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

yhuai · 2016-06-14T16:41:45Z

I think we need to explain the inconsistent behavior in the PR description, which is the main reason of making this change. (right now, if we run select 1/2, the result is 0.5. However, select sum(1/2) will return 0).

cloud-fan · 2016-06-14T20:39:28Z

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionSuite.scala

+
+    // Cast integer to double
+    ruleTest(analyzer, sum(Divide(4, 3)), sum(Divide(Cast(4, DoubleType), Cast(3, DoubleType))))
+    // left expression is already Double, skip


what if left expression is int and right expression is double?

SparkQA · 2016-06-14T22:02:27Z

Test build #60515 has finished for PR 13651 at commit bce5ea7.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2016-06-14T23:41:58Z

cc @hvanhovell

SparkQA · 2016-06-15T00:12:49Z

Test build #60527 has finished for PR 13651 at commit f539392.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-06-15T00:13:36Z

Test build #60524 has finished for PR 13651 at commit d8f3263.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-06-15T00:27:14Z

Test build #60528 has finished for PR 13651 at commit ef83429.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-06-15T00:34:01Z

Test build #60529 has finished for PR 13651 at commit 3dd03b7.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-06-15T00:47:58Z

Test build #60532 has finished for PR 13651 at commit 659f17c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-06-15T00:54:15Z

Test build #60533 has finished for PR 13651 at commit 1787c05.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-06-15T00:59:19Z

Test build #60535 has finished for PR 13651 at commit c7e0b33.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-06-15T01:07:37Z

Test build #60536 has finished for PR 13651 at commit ff3aa3b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-06-15T06:20:21Z

Test build #60551 has finished for PR 13651 at commit 3735411.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Sephiroth-Lin · 2016-06-15T12:54:23Z

LGTM thank you

cloud-fan · 2016-06-15T17:11:17Z

...yst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ArithmeticExpressionSuite.scala

      checkConsistencyBetweenInterpretedAndCodegen(Divide, tpe, tpe)
    }
  }

-  test("/ (Divide) for integral type") {
+  test("SPARK-15776: Divide report unresolved when children's data type is " +


This suite is to test the correctness, not resolution.
BTW, the resolution logic is defined by ExpectedInputTypes, I think we don't need to test it again for divide.

clockfly · 2016-06-15T19:21:43Z

@hvanhovell @cloud-fan Thanks! Updated.

cloud-fan · 2016-06-15T20:09:53Z

LGTM, pending jenkins

SparkQA · 2016-06-15T20:46:08Z

Test build #60586 has finished for PR 13651 at commit 2589b57.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-06-15T21:12:53Z

Test build #60590 has finished for PR 13651 at commit cbbc67c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

hvanhovell · 2016-06-15T21:33:42Z

LGTM, pending jenkins

SparkQA · 2016-06-15T21:34:59Z

Test build #60591 has finished for PR 13651 at commit da25d7a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2016-06-15T21:35:10Z

merging to master/2.0!

rxin · 2016-06-15T21:35:10Z

@clockfly can you add "Closes # #13524 " to the pr description before we merge.

…asted to wrong type ## What changes were proposed in this pull request? This PR fixes the problem that Divide Expression inside Aggregation function is casted to wrong type, which cause `select 1/2` and `select sum(1/2)`returning different result. **Before the change:** ``` scala> sql("select 1/2 as a").show() +---+ | a| +---+ |0.5| +---+ scala> sql("select sum(1/2) as a").show() +---+ | a| +---+ |0 | +---+ scala> sql("select sum(1 / 2) as a").schema res4: org.apache.spark.sql.types.StructType = StructType(StructField(a,LongType,true)) ``` **After the change:** ``` scala> sql("select 1/2 as a").show() +---+ | a| +---+ |0.5| +---+ scala> sql("select sum(1/2) as a").show() +---+ | a| +---+ |0.5| +---+ scala> sql("select sum(1/2) as a").schema res4: org.apache.spark.sql.types.StructType = StructType(StructField(a,DoubleType,true)) ``` ## How was this patch tested? Unit test. This PR is based on #13524 by Sephiroth-Lin Author: Sean Zhong <seanzhong@databricks.com> Closes #13651 from clockfly/SPARK-15776. (cherry picked from commit 9bd80ad) Signed-off-by: Wenchen Fan <wenchen@databricks.com>

SparkQA · 2016-06-15T21:39:05Z

Test build #60592 has finished for PR 13651 at commit 8fc6ba7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

### What changes were proposed in this pull request? The pr aims to upgrade `Netty` from `4.1.100.Final` to `4.1.106.Final`. ### Why are the changes needed? - To bring the latest bug fixes Automatically close Http2StreamChannel when Http2FrameStreamExceptionreaches end ofChannelPipeline ([#13651](netty/netty#13651)) Symbol not found: _netty_jni_util_JNI_OnLoad ([#13695](netty/netty#13728)) - 4.1.106.Final release note: https://netty.io/news/2024/01/19/4-1-106-Final.html - 4.1.105.Final release note: https://netty.io/news/2024/01/16/4-1-105-Final.html - 4.1.104.Final release note: https://netty.io/news/2023/12/15/4-1-104-Final.html - 4.1.103.Final release note: https://netty.io/news/2023/12/13/4-1-103-Final.html - 4.1.101.Final release note: https://netty.io/news/2023/11/09/4-1-101-Final.html ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #44384 from panbingkun/SPARK-46432. Lead-authored-by: panbingkun <panbingkun@baidu.com> Co-authored-by: panbingkun <pbk1982@gmail.com> Signed-off-by: yangjie01 <yangjie01@baidu.com>

SPARK-15776 Divide Expression inside an Aggregation function is caste…

df08eea

…d to wrong type

cloud-fan reviewed Jun 13, 2016
View reviewed changes

change the UT, fix comments

bce5ea7

cloud-fan reviewed Jun 14, 2016
View reviewed changes

clockfly force-pushed the SPARK-15776 branch from ac77820 to da0eb48 Compare June 14, 2016 22:45

clockfly changed the title ~~[SPARK-15776][SQL] Divide Expression inside Aggregation function is casted to wrong type~~ [SPARK-15776][SQL][WIP] Divide Expression inside Aggregation function is casted to wrong type Jun 14, 2016

clockfly force-pushed the SPARK-15776 branch 6 times, most recently from 3dd03b7 to ab4ada0 Compare June 14, 2016 23:07

clockfly changed the title ~~[SPARK-15776][SQL][WIP] Divide Expression inside Aggregation function is casted to wrong type~~ [SPARK-15776][SQL] Divide Expression inside Aggregation function is casted to wrong type Jun 14, 2016

fix UT

659f17c

clockfly force-pushed the SPARK-15776 branch 2 times, most recently from 1787c05 to c7e0b33 Compare June 14, 2016 23:29

on wenchen's comment

ff3aa3b

clockfly force-pushed the SPARK-15776 branch from c7e0b33 to ff3aa3b Compare June 14, 2016 23:32

clockfly mentioned this pull request Jun 14, 2016

[SPARK-15776][SQL] Type coercion incorrect #13524

Closed

fix UT failure

3735411

cloud-fan reviewed Jun 15, 2016
View reviewed changes

clockfly force-pushed the SPARK-15776 branch 4 times, most recently from 70d4cc6 to da25d7a Compare June 15, 2016 19:20

add more UT

8fc6ba7

clockfly force-pushed the SPARK-15776 branch from da25d7a to 8fc6ba7 Compare June 15, 2016 19:23

asfgit closed this in 9bd80ad Jun 15, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-15776][SQL] Divide Expression inside Aggregation function is casted to wrong type #13651

[SPARK-15776][SQL] Divide Expression inside Aggregation function is casted to wrong type #13651

clockfly commented Jun 13, 2016 •

edited

cloud-fan Jun 13, 2016

clockfly Jun 13, 2016

yhuai Jun 14, 2016

SparkQA commented Jun 14, 2016

yhuai commented Jun 14, 2016

cloud-fan Jun 14, 2016

SparkQA commented Jun 14, 2016

rxin commented Jun 14, 2016

SparkQA commented Jun 15, 2016

SparkQA commented Jun 15, 2016

SparkQA commented Jun 15, 2016

SparkQA commented Jun 15, 2016

SparkQA commented Jun 15, 2016

SparkQA commented Jun 15, 2016

SparkQA commented Jun 15, 2016

SparkQA commented Jun 15, 2016

SparkQA commented Jun 15, 2016

Sephiroth-Lin commented Jun 15, 2016

cloud-fan Jun 15, 2016

clockfly commented Jun 15, 2016

cloud-fan commented Jun 15, 2016

SparkQA commented Jun 15, 2016

SparkQA commented Jun 15, 2016

hvanhovell commented Jun 15, 2016

SparkQA commented Jun 15, 2016

cloud-fan commented Jun 15, 2016

rxin commented Jun 15, 2016

SparkQA commented Jun 15, 2016

[SPARK-15776][SQL] Divide Expression inside Aggregation function is casted to wrong type #13651

[SPARK-15776][SQL] Divide Expression inside Aggregation function is casted to wrong type #13651

Conversation

clockfly commented Jun 13, 2016 • edited

What changes were proposed in this pull request?

How was this patch tested?

cloud-fan Jun 13, 2016

Choose a reason for hiding this comment

clockfly Jun 13, 2016

Choose a reason for hiding this comment

yhuai Jun 14, 2016

Choose a reason for hiding this comment

SparkQA commented Jun 14, 2016

yhuai commented Jun 14, 2016

cloud-fan Jun 14, 2016

Choose a reason for hiding this comment

SparkQA commented Jun 14, 2016

rxin commented Jun 14, 2016

SparkQA commented Jun 15, 2016

SparkQA commented Jun 15, 2016

SparkQA commented Jun 15, 2016

SparkQA commented Jun 15, 2016

SparkQA commented Jun 15, 2016

SparkQA commented Jun 15, 2016

SparkQA commented Jun 15, 2016

SparkQA commented Jun 15, 2016

SparkQA commented Jun 15, 2016

Sephiroth-Lin commented Jun 15, 2016

cloud-fan Jun 15, 2016

Choose a reason for hiding this comment

clockfly commented Jun 15, 2016

cloud-fan commented Jun 15, 2016

SparkQA commented Jun 15, 2016

SparkQA commented Jun 15, 2016

hvanhovell commented Jun 15, 2016

SparkQA commented Jun 15, 2016

cloud-fan commented Jun 15, 2016

rxin commented Jun 15, 2016

SparkQA commented Jun 15, 2016

clockfly commented Jun 13, 2016 •

edited