Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEA] Support "WindowGroupLimit" optimization on GPU for Databricks 13.3 ML LTS+ #10531

Closed
sameerz opened this issue Mar 1, 2024 · 1 comment · Fixed by #10550
Closed

[FEA] Support "WindowGroupLimit" optimization on GPU for Databricks 13.3 ML LTS+ #10531

sameerz opened this issue Mar 1, 2024 · 1 comment · Fixed by #10550
Assignees
@mythrocks
Labels
performance A performance related task/issue

Comments

@sameerz
Copy link
Collaborator

sameerz commented Mar 1, 2024

Is your feature request related to a problem? Please describe.
Issue #8208 enabled WindowGroupLimit in the Spark 3.5.0 and 3.5.1 shims. It did not enable for Databricks 13.3 ML LTS.

Describe the solution you'd like
Enable WindowGroupLimit in the Databricks 13.3 ML LTS shim.

Describe alternatives you've considered

Additional context
https://docs.databricks.com/en/release-notes/runtime/13.3lts.html
@sameerz sameerz added feature request New feature or request ? - Needs Triage Need team to review and classify labels Mar 1, 2024
@sameerz sameerz changed the title [FEA] Support "WindowGroupLimit" optimization on GPU [FEA] Support "WindowGroupLimit" optimization on GPU for Databricks 13.3 ML LTS+ Mar 1, 2024
@mythrocks mythrocks self-assigned this Mar 1, 2024
@mythrocks
Copy link
Collaborator

Investigating now.

@sameerz sameerz removed the ? - Needs Triage Need team to review and classify label Mar 1, 2024
@mythrocks mythrocks mentioned this issue Mar 5, 2024
Merged
@sameerz sameerz added performance A performance related task/issue and removed feature request New feature or request labels Mar 6, 2024
mythrocks added a commit that referenced this issue Mar 11, 2024
@mythrocks
Enable window-group-limit optimization on [databricks] (#10550)
9cf2acb 
* WindowGroupLimit support for [databricks].

Fixes #10531.

This is a followup to #10500, which added support to push down window-group-limit filters before the shuffle phase.

#10500 inadvertently neglected to ensure that the optimization works on Databricks. (It turns out that window-group-limit was cherry-picked into Databricks 13.3, despite the nominal Spark version being `3.4.1`.)

This change ensures that the same optimization is available on Databricks 13.3 (and beyond).

---------

Signed-off-by: MithunR <mythrocks@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mythrocks mythrocks

Labels
performance A performance related task/issue
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Enable window-group-limit optimization on [databricks] mythrocks/spark-rapids
2 participants
@mythrocks @sameerz