-
Notifications
You must be signed in to change notification settings - Fork 141
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[#635] feat(client): enable LOCAL_ORDER by default for Spark AQE #644
Conversation
PTAL @jerqi |
Codecov Report
@@ Coverage Diff @@
## master #644 +/- ##
============================================
+ Coverage 60.61% 62.91% +2.29%
- Complexity 1798 1800 +2
============================================
Files 216 202 -14
Lines 12462 10491 -1971
Branches 1052 1052
============================================
- Hits 7554 6600 -954
+ Misses 4500 3545 -955
+ Partials 408 346 -62 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
common/src/main/java/org/apache/uniffle/common/config/RssClientConf.java
Outdated
Show resolved
Hide resolved
@VisibleForTesting | ||
protected static ShuffleDataDistributionType getDataDistributionType(SparkConf sparkConf) { | ||
RssConf rssConf = RssSparkConfig.toRssConf(sparkConf); | ||
if ((boolean) sparkConf.get(SQLConf.ADAPTIVE_EXECUTION_ENABLED()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sparkConf.get(SQLConf.ADAPTIVE_EXECUTION_ENABLED()
should be used to be consistent with spark, because different spark versions have different default value
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cc @jerqi
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is (boolean)
necessary?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
thanks @zuston , merged to master. |
apache#644) ### What changes were proposed in this pull request? enable LOCAL_ORDER by default for Spark AQE ### Why are the changes needed? Currently, the local_order data distribution type should be activated explicitly. It's better to enable it when using AQE by default. Fix: apache#635 ### Does this PR introduce _any_ user-facing change? Yes ### How was this patch tested? 1. Existing UTs
What changes were proposed in this pull request?
enable LOCAL_ORDER by default for Spark AQE
Why are the changes needed?
Currently, the local_order data distribution type should be activated explicitly. It's better to enable it when using AQE by default.
Fix: #635
Does this PR introduce any user-facing change?
Yes
How was this patch tested?