Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-24575][SQL] Prohibit window expressions inside WHERE and HAVING clauses #21580

Closed
wants to merge 1 commit into from

Conversation

aokolnychyi
Copy link
Contributor

What changes were proposed in this pull request?

As discussed before, this PR prohibits window expressions inside WHERE and HAVING clauses.

How was this patch tested?

This PR comes with a dedicated unit test.

@SparkQA
Copy link

SparkQA commented Jun 17, 2018

Test build #92003 has finished for PR 21580 at commit 9a07ea3.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@maropu
Copy link
Member

maropu commented Jun 18, 2018

retest this please

@SparkQA
Copy link

SparkQA commented Jun 18, 2018

Test build #92010 has finished for PR 21580 at commit 9a07ea3.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@hvanhovell
Copy link
Contributor

hvanhovell commented Jun 18, 2018

LGTM. It is better UX to have a more descriptive error messages.

However I do like the idea of being able to use window functions in filters. I often use the following pattern:

val df: Dataframe = ...
df.select(row_number().over(Window.partitionBy($"key").orderBy($"seq")).as("rn"))
  .filter($"rn" === 1)
  .drop("rn")

Teradata, for example, has the qualify filter clause for these cases.

@hvanhovell
Copy link
Contributor

Merging to master. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants