Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-28271][SQL][PYTHON][TESTS] Convert and port 'pgSQL/aggregates_part2.sql' into UDF test base #25086

Closed
wants to merge 2 commits into from

Conversation

imback82
Copy link
Contributor

@imback82 imback82 commented Jul 9, 2019

What changes were proposed in this pull request?

This PR adds some tests converted from pgSQL/aggregates_part2.sql' to test UDFs. Please see contribution guide of this umbrella ticket - SPARK-27921.

Diff comparing to 'pgSQL/aggregates_part2.sql'

diff --git a/sql/core/src/test/resources/sql-tests/results/pgSQL/aggregates_part2.sql.out b/sql/core/src/test/resources/sql-tests/results/udf/pgSQL/udf-aggregates_part2.sql.out
index 2606d2eba7..00c06f94b5 100644
--- a/sql/core/src/test/resources/sql-tests/results/pgSQL/aggregates_part2.sql.out
+++ b/sql/core/src/test/resources/sql-tests/results/udf/pgSQL/udf-aggregates_part2.sql.out
@@ -57,23 +57,23 @@ true        false   true    false   true    true    true    true    true


 -- !query 3
-select min(unique1) from tenk1
+select min(udf(unique1)) from tenk1
 -- !query 3 schema
-struct<min(unique1):int>
+struct<min(udf(unique1)):string>
 -- !query 3 output
 0


 -- !query 4
-select max(unique1) from tenk1
+select udf(max(unique1)) from tenk1
 -- !query 4 schema
-struct<max(unique1):int>
+struct<udf(max(unique1)):string>
 -- !query 4 output
 9999


 -- !query 5
-select max(unique1) from tenk1 where unique1 < 42
+select max(unique1) from tenk1 where udf(unique1) < 42
 -- !query 5 schema
 struct<max(unique1):int>
 -- !query 5 output
@@ -81,7 +81,7 @@ struct<max(unique1):int>


 -- !query 6
-select max(unique1) from tenk1 where unique1 > 42
+select max(unique1) from tenk1 where unique1 > udf(42)
 -- !query 6 schema
 struct<max(unique1):int>
 -- !query 6 output
@@ -89,7 +89,7 @@ struct<max(unique1):int>


 -- !query 7
-select max(unique1) from tenk1 where unique1 > 42000
+select max(unique1) from tenk1 where udf(unique1) > 42000
 -- !query 7 schema
 struct<max(unique1):int>
 -- !query 7 output
@@ -97,7 +97,7 @@ NULL


 -- !query 8
-select max(tenthous) from tenk1 where thousand = 33
+select max(tenthous) from tenk1 where udf(thousand) = 33
 -- !query 8 schema
 struct<max(tenthous):int>
 -- !query 8 output
@@ -105,7 +105,7 @@ struct<max(tenthous):int>


 -- !query 9
-select min(tenthous) from tenk1 where thousand = 33
+select min(tenthous) from tenk1 where udf(thousand) = 33
 -- !query 9 schema
 struct<min(tenthous):int>
 -- !query 9 output
@@ -113,15 +113,15 @@ struct<min(tenthous):int>


 -- !query 10
-select distinct max(unique2) from tenk1
+select distinct max(udf(unique2)) from tenk1
 -- !query 10 schema
-struct<max(unique2):int>
+struct<max(udf(unique2)):string>
 -- !query 10 output
 9999


 -- !query 11
-select max(unique2) from tenk1 order by 1
+select max(unique2) from tenk1 order by udf(1)
 -- !query 11 schema
 struct<max(unique2):int>
 -- !query 11 output
@@ -129,7 +129,7 @@ struct<max(unique2):int>

 -- !query 12
-select max(unique2) from tenk1 order by max(unique2)
+select max(unique2) from tenk1 order by max(udf(unique2))
 -- !query 12 schema
 struct<max(unique2):int>
 -- !query 12 output
@@ -137,7 +137,7 @@ struct<max(unique2):int>


 -- !query 13
-select max(unique2) from tenk1 order by max(unique2)+1
+select udf(max(udf(unique2))) from tenk1 order by udf(max(unique2))+1
 -- !query 13 schema
-struct<max(unique2):int>
+struct<udf(max(udf(unique2))):string>
 -- !query 13 output
 9999


 -- !query 14
-select t1.max_unique2, g from (select max(unique2) as max_unique2 FROM tenk1) t1 LATERAL VIEW explode(array(1,2,3)) t2 AS g order by g desc
+select t1.max_unique2, udf(g) from (select max(udf(unique2)) as max_unique2 FROM tenk1) t1 LATERAL VIEW explode(array(1,2,3)) t2 AS g order by g desc
 -- !query 14 schema
-struct<max_unique2:int,g:int>
+struct<max_unique2:string,udf(g):string>
 -- !query 14 output
 9999   3
 9999   2
@@ -155,8 +155,8 @@ struct<max_unique2:int,g:int>


 -- !query 15
-select max(100) from tenk1
+select udf(max(100)) from tenk1
 -- !query 15 schema
-struct<max(100):int>
+struct<udf(max(100)):string>
 -- !query 15 output
 100

How was this patch tested?

Tested as guided in SPARK-27921.

@imback82
Copy link
Contributor Author

imback82 commented Jul 9, 2019

cc: @HyukjinKwon

@dongjoon-hyun dongjoon-hyun changed the title [SPARK-28271][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part2.sql' into UDF test base [SPARK-28271][SQL][TEST] Convert and port 'pgSQL/aggregates_part2.sql' into UDF test base Jul 9, 2019
@dongjoon-hyun
Copy link
Member

ok to test

@SparkQA
Copy link

SparkQA commented Jul 9, 2019

Test build #107416 has finished for PR 25086 at commit 861f02a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon HyukjinKwon changed the title [SPARK-28271][SQL][TEST] Convert and port 'pgSQL/aggregates_part2.sql' into UDF test base [SPARK-28271][SQL][PYTHON][TESTS] Convert and port 'pgSQL/aggregates_part2.sql' into UDF test base Jul 10, 2019
@HyukjinKwon
Copy link
Member

Looks good otherwise.

@SparkQA
Copy link

SparkQA commented Jul 10, 2019

Test build #107422 has finished for PR 25086 at commit 0550038.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

--
--
-- AGGREGATES [Part 2]
-- https://github.com/postgres/postgres/blob/REL_12_BETA1/src/test/regress/sql/aggregates.sql#L145-L350
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use REL_12_BETA2 here because BETA2 is already out and more closer to REL_12. I guess REL_12_BETA2 will be the same with REL_12_BETA1.
If not, please make this up-to-date.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like there's one more case. Let me make a small PR separately to address this comment for aggregates_part2.sql as well.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1.

@HyukjinKwon
Copy link
Member

Merged to master.

Will make a follow right away, @dongjoon-hyun.

@dongjoon-hyun
Copy link
Member

Thank you, @HyukjinKwon ~

@imback82 imback82 deleted the udf_test branch July 13, 2019 01:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants