Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-20311][SQL] Support aliases for table value functions #17928

Closed
wants to merge 4 commits into from
Closed

[SPARK-20311][SQL] Support aliases for table value functions #17928

wants to merge 4 commits into from

Conversation

maropu
Copy link
Member

@maropu maropu commented May 10, 2017

What changes were proposed in this pull request?

This pr added parsing rules to support aliases in table value functions.
The previous pr (#17666) has been reverted because of the regression. This new pr fixed the regression and add tests in SQLQueryTestSuite.

How was this patch tested?

Added tests in PlanParserSuite and SQLQueryTestSuite.

cloud-fan
cloud-fan reviewed May 10, 2017
View reviewed changes
sql/core/src/test/resources/sql-tests/results/subquery/exists-subquery/exists-cte.sql.out Outdated
emp 5 1000.0
emp 6 - no dept 500.0
org.apache.spark.SparkException
Exception thrown in awaitResult:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are we breaking existing queries?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, my bad. I'll check again.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

@SparkQA
Copy link

SparkQA commented May 10, 2017

Test build #76712 has finished for PR 17928 at commit 47b923b.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • case class UnresolvedTableValuedFunction(

@SparkQA
Copy link

SparkQA commented May 10, 2017

Test build #76716 has finished for PR 17928 at commit 0904fc9.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • case class UnresolvedTableValuedFunction(

@maropu
Copy link
Member Author

maropu commented May 10, 2017

@cloud-fan ok, could you check again? Thanks!

@cloud-fan
Copy link
Contributor

LGTM, cc @gatorsmile to take another look

gatorsmile
gatorsmile reviewed May 10, 2017
View reviewed changes
sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4
;

tableAlias
: (AS? strictIdentifier identifierList?)?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also hits another bug in inline tables. Maybe you also can include the following query in the test case inline-table.sql?

sql("SELECT * FROM VALUES (\"one\", 1), (\"three\", null) CROSS JOIN VALUES (\"one\", 1), (\"three\", null)")

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added.

gatorsmile
gatorsmile reviewed May 10, 2017
View reviewed changes
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala Outdated

test("SPARK-20311 range(N) as alias") {
assertEqual(
"select * from range(10) AS t",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: SELECT * FROM range(10) AS t
BTW, we prefer to use upper case for the SQL keywords.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You also can update the similar issues in your test cases.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

@gatorsmile
Copy link
Member

LGTM

@SparkQA
Copy link

SparkQA commented May 10, 2017

Test build #76737 has finished for PR 17928 at commit 29281b1.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

gatorsmile
gatorsmile reviewed May 10, 2017
View reviewed changes
sql/core/src/test/resources/sql-tests/inputs/table-valued-functions.sql Outdated
@@ -24,3 +24,7 @@ select * from RaNgE(2);

-- Explain
EXPLAIN select * from RaNgE(2);

-- cross-join table valued functions
SET spark.sql.crossJoin.enabled=true;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you remove this line? If we specify CROSS JOIN in the query, no need to set this parm.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

@SparkQA
Copy link

SparkQA commented May 11, 2017

Test build #76770 has finished for PR 17928 at commit 54a05da.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

cloud-fan
cloud-fan reviewed May 11, 2017
View reviewed changes
sql/core/src/test/resources/sql-tests/inputs/inline-table.sql Outdated
@@ -49,3 +49,6 @@ select * from values ("one", count(1)), ("two", 2) as data(a, b);

-- string to timestamp
select * from values (timestamp('1991-12-06 00:00:00.0'), array(timestamp('1991-12-06 01:00:00.0'), timestamp('1991-12-06 12:00:00.0'))) as data(a, b);

-- cross-join inline tables
SELECT * FROM VALUES ('one', 1), ('three', null) CROSS JOIN VALUES ('one', 1), ('three', null);
Copy link
Contributor

@cloud-fan cloud-fan May 11, 2017 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this expose the bug? If we treat CROSS as an alias, we still get the same result. how about we run EXPLAIN?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aha, ok

@SparkQA
Copy link

SparkQA commented May 11, 2017

Test build #76784 has started for PR 17928 at commit a7a732b.

@cloud-fan
Copy link
Contributor

retest this please

@SparkQA
Copy link

SparkQA commented May 11, 2017

Test build #76791 has finished for PR 17928 at commit a7a732b.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@asfgit asfgit closed this in 8c67aa7 May 11, 2017
@gatorsmile
Copy link
Member

Thanks! Merging to master.

liyichao pushed a commit to liyichao/spark that referenced this pull request May 24, 2017
@maropu @liyichao
[SPARK-20311][SQL] Support aliases for table value functions
10e528a 
## What changes were proposed in this pull request?
This pr added parsing rules to support aliases in table value functions.
The previous pr (apache#17666) has been reverted because of the regression. This new pr fixed the regression and add tests in `SQLQueryTestSuite`.

## How was this patch tested?
Added tests in `PlanParserSuite` and `SQLQueryTestSuite`.

Author: Takeshi Yamamuro <yamamuro@apache.org>

Closes apache#17928 from maropu/SPARK-20311-3.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cloud-fan cloud-fan cloud-fan left review comments

@gatorsmile gatorsmile gatorsmile left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
4 participants
@maropu @SparkQA @cloud-fan @gatorsmile