[SPARK-47085][SQL][3.5] reduce the complexity of toTRowSet from n^2 to n #45165
Conversation
LuciferYang
changed the title
There was a problem hiding this comment.
Hi, @igreenfield and @yaooqinn .
This could be a good improvement. However, Apache Spark community has a backporting policy which allows a bug-fix only.
If you really want to backport this, please change the issue type to BUG and provide some background why this is a regression at Apache Spark 3.4.0.
|
Hi @dongjoon-hyun |
Hi, @igreenfield . This looks like a pure performance improvement. As I mentioned before, Apache Spark community has a backporting policy which allows a bug-fix only. |
There was a problem hiding this comment.
Thank you for the detail. According to the updated info, SPARK-39041 causes this regression at Apache Spark 3.4.0.
+1, LGTM.
### What changes were proposed in this pull request? reduce the complexity of RowSetUtils.toTRowSet from n^2 to n ### Why are the changes needed? This causes performance issues. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Tests + test manually on AWS EMR ### Was this patch authored or co-authored using generative AI tooling? No Closes #45165 from igreenfield/branch-3.5. Authored-by: Izek Greenfield <izek.greenfield@adenza.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
|
Merged to branch-3.5. |
What changes were proposed in this pull request?
reduce the complexity of RowSetUtils.toTRowSet from n^2 to n
Why are the changes needed?
This causes performance issues.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Tests + test manually on AWS EMR
Was this patch authored or co-authored using generative AI tooling?
No