-
Notifications
You must be signed in to change notification settings - Fork 28k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-47085][SQL][3.4] reduce the complexity of toTRowSet from n^2 to n #45164
Conversation
can you retrigger the CI? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi, @igreenfield and @yaooqinn .
This could be a good improvement. However, Apache Spark community has a backporting policy which allows a bug-fix only.
If you really want to backport this, please change the issue type to BUG
and provide some background why this is a regression at Apache Spark 3.4.0.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1, LGTM (Pending CIs).
Could you re-trigger the CI once more, @igreenfield ?
### What changes were proposed in this pull request? reduce the complexity of RowSetUtils.toTRowSet from n^2 to n ### Why are the changes needed? This causes performance issues. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Tests + test manually on AWS EMR ### Was this patch authored or co-authored using generative AI tooling? No Closes #45164 from igreenfield/branch-3.4. Authored-by: Izek Greenfield <izek.greenfield@adenza.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
Merged to branch-3.4. |
### What changes were proposed in this pull request? reduce the complexity of RowSetUtils.toTRowSet from n^2 to n ### Why are the changes needed? This causes performance issues. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Tests + test manually on AWS EMR ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#45164 from igreenfield/branch-3.4. Authored-by: Izek Greenfield <izek.greenfield@adenza.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
What changes were proposed in this pull request?
reduce the complexity of RowSetUtils.toTRowSet from n^2 to n
Why are the changes needed?
This causes performance issues.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Tests + test manually on AWS EMR
Was this patch authored or co-authored using generative AI tooling?
No