Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP][SPARK-34082][SQL] Window expressions with alias inside WHERE and HAVING clauses fail with explicit exceptions #31146

Closed
wants to merge 1 commit into from

Conversation

LantaoJin
Copy link
Contributor

@LantaoJin LantaoJin commented Jan 12, 2021

What changes were proposed in this pull request?

#21580 prohibits window expressions inside WHERE and HAVING clauses. But if the window expressions with alias inside WHERE and HAVING clauses, Spark does not handle this explicitly and will fail with non-descriptive exceptions.

SELECT a, RANK() OVER(ORDER BY b) AS s FROM testData2 WHERE b = 2 AND s = 1

cannot resolve 's' given input columns: [testdata2.a, testdata2.b]

SELECT a, MAX(b), RANK() OVER(ORDER BY a) AS s
FROM testData2
GROUP BY a
HAVING SUM(b) = 5 AND s = 1

cannot resolve 'b' given input columns: [testdata2.a, max(b)]

Why are the changes needed?

Correct the exception message to It is not allowed to use window functions inside $clauseName clause

Does this PR introduce any user-facing change?

No

How was this patch tested?

Add unit test cases

@github-actions github-actions bot added the SQL label Jan 12, 2021
@SparkQA
Copy link

SparkQA commented Jan 12, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38543/

@LantaoJin LantaoJin changed the title [SPARK-34082][SQL] Window expressions with alias inside WHERE and HAVING clauses fail with explicit exceptions [WIP][SPARK-34082][SQL] Window expressions with alias inside WHERE and HAVING clauses fail with explicit exceptions Jan 12, 2021
@SparkQA
Copy link

SparkQA commented Jan 12, 2021

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38543/

@SparkQA
Copy link

SparkQA commented Jan 12, 2021

Test build #133956 has finished for PR 31146 at commit 2f7df5b.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@LantaoJin
Copy link
Contributor Author

Close it due to cannot resolve 'b' given input columns seems a correct error message. Filter should be resolved before Projection. I was confused with QUALIFY syntax in our internal Spark version.

@LantaoJin LantaoJin closed this Jan 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
2 participants