[SPARK-30864][SQL][DOC] add the user guide for Adaptive Query Execution #27616

JkSelf · 2020-02-18T06:01:58Z

What changes were proposed in this pull request?

This PR will add the user guide for AQE and the detailed configurations about the three mainly features in AQE.

Why are the changes needed?

Add the detailed configurations.

Does this PR introduce any user-facing change?

No

How was this patch tested?

only add doc no need ut.

JkSelf · 2020-02-18T06:02:30Z

cc @cloud-fan

cloud-fan · 2020-02-18T06:07:33Z

docs/sql-performance-tuning.md

@@ -186,3 +186,75 @@ The "REPARTITION_BY_RANGE" hint must have column names and a partition number is
    SELECT /*+ REPARTITION(3, c) */ * FROM t
    SELECT /*+ REPARTITION_BY_RANGE(c) */ * FROM t
    SELECT /*+ REPARTITION_BY_RANGE(3, c) */ * FROM t
+
+## Adaptive Query Execution
+Adaptive Query Execution (AQE) is an optimization technique in Spark SQL that make use of the runtime statistics to choose the most efficient query execution plan. AQE is disabled by default. Spark SQL can use the umbrella configuration of `spark.sql.adaptive.enabled` to control whether turn it on/off. There are three mainly feature in AQE, including coalescing post partition number, optimizing local shuffle reader and optimizing skewed join.


As of Spark 3.0, there are three major features ...

post-shuffle partition number

cloud-fan · 2020-02-18T06:09:36Z

docs/sql-performance-tuning.md

+
+## Adaptive Query Execution
+Adaptive Query Execution (AQE) is an optimization technique in Spark SQL that make use of the runtime statistics to choose the most efficient query execution plan. AQE is disabled by default. Spark SQL can use the umbrella configuration of `spark.sql.adaptive.enabled` to control whether turn it on/off. There are three mainly feature in AQE, including coalescing post partition number, optimizing local shuffle reader and optimizing skewed join.
+ ### Coalescing Post Shuffle Partition Num


cloud-fan · 2020-02-18T06:12:17Z

docs/sql-performance-tuning.md

+## Adaptive Query Execution
+Adaptive Query Execution (AQE) is an optimization technique in Spark SQL that make use of the runtime statistics to choose the most efficient query execution plan. AQE is disabled by default. Spark SQL can use the umbrella configuration of `spark.sql.adaptive.enabled` to control whether turn it on/off. There are three mainly feature in AQE, including coalescing post partition number, optimizing local shuffle reader and optimizing skewed join.
+ ### Coalescing Post Shuffle Partition Num
+ This feature coalesces the post shuffle partitions based on the map output statistics when `spark.sql.adaptive.enabled` and `spark.sql.adaptive.shuffle.reducePostShufflePartitions.enabled` configuration properties are both enabled. There are four following sub-configurations in this optimization rule. 


shall we introduce the benefits of this feature?

cloud-fan · 2020-02-18T06:17:15Z

docs/sql-performance-tuning.md

+ </table>
+
+ ### Optimize Local Shuffle Reader
+ This feature optimize the shuffle reader to local shuffle reader when converting the sort merge join to broadcast hash join in runtime and no additional shuffle introduced. It takes effect when `spark.sql.adaptive.enabled` and `spark.sql.adaptive.shuffle.localShuffleReader.enabled` configuration properties are both enabled.


ditto, users care more about the benefit

add the performance data both three features.

SparkQA · 2020-02-18T06:21:00Z

Test build #118613 has finished for PR 27616 at commit 19a381b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-02-18T07:28:26Z

Test build #118618 has finished for PR 27616 at commit 727f57f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-02-19T13:14:57Z

docs/sql-performance-tuning.md

@@ -186,3 +186,75 @@ The "REPARTITION_BY_RANGE" hint must have column names and a partition number is
    SELECT /*+ REPARTITION(3, c) */ * FROM t
    SELECT /*+ REPARTITION_BY_RANGE(c) */ * FROM t
    SELECT /*+ REPARTITION_BY_RANGE(3, c) */ * FROM t
+
+## Adaptive Query Execution
+Adaptive Query Execution (AQE) is an optimization technique in Spark SQL that makes use of the runtime statistics to choose the most efficient query execution plan. AQE is disabled by default. Spark SQL can use the umbrella configuration of `spark.sql.adaptive.enabled` to control whether turn it on/off. As of Spark 3.0, there are three major features in AQE, including coalescing post-shuffle partition number, optimizing local shuffle reader and optimizing skewed join.


nit: coalescing post-shuffle partitions, local shuffle reader optimization and skewed join optimization.

cloud-fan · 2020-02-19T13:15:20Z

docs/sql-performance-tuning.md

+## Adaptive Query Execution
+Adaptive Query Execution (AQE) is an optimization technique in Spark SQL that makes use of the runtime statistics to choose the most efficient query execution plan. AQE is disabled by default. Spark SQL can use the umbrella configuration of `spark.sql.adaptive.enabled` to control whether turn it on/off. As of Spark 3.0, there are three major features in AQE, including coalescing post-shuffle partition number, optimizing local shuffle reader and optimizing skewed join.
+ ### Coalescing Post Shuffle Partition Number
+ This feature coalesces the post shuffle partitions based on the map output statistics when `spark.sql.adaptive.enabled` and `spark.sql.adaptive.shuffle.reducePostShufflePartitions.enabled` configuration properties are both enabled. There are four following sub-configurations in this optimization rule. And this feature can bring about 1.28x performance gain with query 38 in 3TB TPC-DS.


let's update to the latest config names.

And this feature can bring about 1.28x performance gain with query 38 in 3TB TPC-DS. This is not useful... how about something like

This feature simplifies the tuning of shuffle partitions number when running queries. You don't need to set a proper shuffle partition number that just fits your data. You just need to set a large enough number and Spark can pick the proper shuffle partition number at runtime.

cloud-fan · 2020-02-19T13:23:53Z

docs/sql-performance-tuning.md

+ </table>
+
+ ### Optimize Local Shuffle Reader
+ This feature optimize the shuffle reader to local shuffle reader when converting the sort merge join to broadcast hash join in runtime and no additional shuffle introduced. It takes effect when `spark.sql.adaptive.enabled` and `spark.sql.adaptive.shuffle.localShuffleReader.enabled` configuration properties are both enabled. This feature and coalescing post shuffle partition number feature can bring about 1.76x performance gain with query 77 in 3TB TPC-DS.  


ditto, don't put perf number in a user guide. Just briefly explain how it affects user queries. E.g. save network traffic

SparkQA · 2020-02-27T06:46:21Z

Test build #119003 has finished for PR 27616 at commit 8ff8b71.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-02-27T06:58:34Z

Test build #119002 has finished for PR 27616 at commit 52fcdf3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-03-06T13:14:39Z

Test build #119455 has finished for PR 27616 at commit 4fdbe10.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-03-10T08:35:22Z

cc @maryannxue

HyukjinKwon · 2020-03-10T08:37:07Z

docs/sql-performance-tuning.md

+
+## Adaptive Query Execution
+Adaptive Query Execution (AQE) is an optimization technique in Spark SQL that makes use of the runtime statistics to choose the most efficient query execution plan. AQE is disabled by default. Spark SQL can use the umbrella configuration of `spark.sql.adaptive.enabled` to control whether turn it on/off. As of Spark 3.0, there are three major features in AQE, including coalescing post-shuffle partitions, local shuffle reader optimization and skewed join optimization.
+ ### Coalescing Post Shuffle Partition Number


There's a leading space here which should be removed. Also newline should be inserted prior to '###'.

HyukjinKwon · 2020-03-10T08:37:19Z

docs/sql-performance-tuning.md

+   </tr>
+ </table>
+
+ ### Optimize Local Shuffle Reader


There's another leading space here.

HyukjinKwon · 2020-03-10T08:41:25Z

docs/sql-performance-tuning.md

+
+ ### Optimize Local Shuffle Reader
+ This feature optimize the shuffle reader to local shuffle reader when converting the sort merge join to broadcast hash join in runtime and no additional shuffle introduced. It takes effect when `spark.sql.adaptive.enabled` and `spark.sql.adaptive.localShuffleReader.enabled` configuration properties are both enabled. This feature can improve the performance by saving the network overhead of shuffle process.
+ ### Optimize Skewed Join


HyukjinKwon · 2020-03-10T08:41:44Z

docs/sql-performance-tuning.md

+ ### Optimize Local Shuffle Reader
+ This feature optimize the shuffle reader to local shuffle reader when converting the sort merge join to broadcast hash join in runtime and no additional shuffle introduced. It takes effect when `spark.sql.adaptive.enabled` and `spark.sql.adaptive.localShuffleReader.enabled` configuration properties are both enabled. This feature can improve the performance by saving the network overhead of shuffle process.
+ ### Optimize Skewed Join
+ This feature choose the skewed partition and creates multi tasks to handle the skewed partition when both enable `spark.sql.adaptive.enabled` and `spark.sql.adaptive.skewJoin.enabled`. There are two following sub-configurations in this optimization rule. Data skew can severely downgrade performance of join queries. And this feature can split the skewed partition into multi parallel tasks instead of original 1 task to reduce the overhead of skewed join.


choose -> chooses

HyukjinKwon · 2020-03-10T08:41:53Z

docs/sql-performance-tuning.md

+ ### Optimize Local Shuffle Reader
+ This feature optimize the shuffle reader to local shuffle reader when converting the sort merge join to broadcast hash join in runtime and no additional shuffle introduced. It takes effect when `spark.sql.adaptive.enabled` and `spark.sql.adaptive.localShuffleReader.enabled` configuration properties are both enabled. This feature can improve the performance by saving the network overhead of shuffle process.
+ ### Optimize Skewed Join
+ This feature choose the skewed partition and creates multi tasks to handle the skewed partition when both enable `spark.sql.adaptive.enabled` and `spark.sql.adaptive.skewJoin.enabled`. There are two following sub-configurations in this optimization rule. Data skew can severely downgrade performance of join queries. And this feature can split the skewed partition into multi parallel tasks instead of original 1 task to reduce the overhead of skewed join.


multi -> multiple

HyukjinKwon · 2020-03-10T08:50:32Z

docs/sql-performance-tuning.md

+ </table>
+
+ ### Optimize Local Shuffle Reader
+ This feature optimize the shuffle reader to local shuffle reader when converting the sort merge join to broadcast hash join in runtime and no additional shuffle introduced. It takes effect when `spark.sql.adaptive.enabled` and `spark.sql.adaptive.localShuffleReader.enabled` configuration properties are both enabled. This feature can improve the performance by saving the network overhead of shuffle process.


no additional shuffle introduced -> no additional shuffle is introduced

HyukjinKwon · 2020-03-10T08:51:39Z

docs/sql-performance-tuning.md

+ </table>
+
+ ### Optimize Local Shuffle Reader
+ This feature optimize the shuffle reader to local shuffle reader when converting the sort merge join to broadcast hash join in runtime and no additional shuffle introduced. It takes effect when `spark.sql.adaptive.enabled` and `spark.sql.adaptive.localShuffleReader.enabled` configuration properties are both enabled. This feature can improve the performance by saving the network overhead of shuffle process.


when when spark.sql.adaptive.enabled and spark.sql.adaptive.localShuffleReader.enabled configuration properties are both enabled. ->

when both spark.sql.adaptive.enabled and spark.sql.adaptive.localShuffleReader.enabled configuration are enabled.

HyukjinKwon · 2020-03-10T08:52:13Z

docs/sql-performance-tuning.md

+## Adaptive Query Execution
+Adaptive Query Execution (AQE) is an optimization technique in Spark SQL that makes use of the runtime statistics to choose the most efficient query execution plan. AQE is disabled by default. Spark SQL can use the umbrella configuration of `spark.sql.adaptive.enabled` to control whether turn it on/off. As of Spark 3.0, there are three major features in AQE, including coalescing post-shuffle partitions, local shuffle reader optimization and skewed join optimization.
+ ### Coalescing Post Shuffle Partition Number
+ This feature coalesces the post shuffle partitions based on the map output statistics when `spark.sql.adaptive.enabled` and `spark.sql.adaptive.coalescePartitions.enabled` configuration properties are both enabled. There are four following sub-configurations in this optimization rule. This feature simplifies the tuning of shuffle partitions number when running queries. You don't need to set a proper shuffle partition number to fit your dataset. You just need to set a large enough number and Spark can pick the proper shuffle partition number at runtime.


when spark.sql.adaptive.enabled and spark.sql.adaptive.coalescePartitions.enabled configuration properties are both enabled. ->

when both spark.sql.adaptive.enabled and spark.sql.adaptive.coalescePartitions.enabled configurations are enabled.

You just need to set a large enough number -> You just need to set a large enough initial number of shuffle partitions via xxx config, and ...

HyukjinKwon · 2020-03-10T08:52:32Z

docs/sql-performance-tuning.md

+## Adaptive Query Execution
+Adaptive Query Execution (AQE) is an optimization technique in Spark SQL that makes use of the runtime statistics to choose the most efficient query execution plan. AQE is disabled by default. Spark SQL can use the umbrella configuration of `spark.sql.adaptive.enabled` to control whether turn it on/off. As of Spark 3.0, there are three major features in AQE, including coalescing post-shuffle partitions, local shuffle reader optimization and skewed join optimization.
+ ### Coalescing Post Shuffle Partition Number
+ This feature coalesces the post shuffle partitions based on the map output statistics when `spark.sql.adaptive.enabled` and `spark.sql.adaptive.coalescePartitions.enabled` configuration properties are both enabled. There are four following sub-configurations in this optimization rule. This feature simplifies the tuning of shuffle partitions number when running queries. You don't need to set a proper shuffle partition number to fit your dataset. You just need to set a large enough number and Spark can pick the proper shuffle partition number at runtime.


Please avoid abbreviation in the public documentation. don't -> do not

HyukjinKwon · 2020-03-10T08:54:35Z

docs/sql-performance-tuning.md

+## Adaptive Query Execution
+Adaptive Query Execution (AQE) is an optimization technique in Spark SQL that makes use of the runtime statistics to choose the most efficient query execution plan. AQE is disabled by default. Spark SQL can use the umbrella configuration of `spark.sql.adaptive.enabled` to control whether turn it on/off. As of Spark 3.0, there are three major features in AQE, including coalescing post-shuffle partitions, local shuffle reader optimization and skewed join optimization.
+ ### Coalescing Post Shuffle Partition Number
+ This feature coalesces the post shuffle partitions based on the map output statistics when `spark.sql.adaptive.enabled` and `spark.sql.adaptive.coalescePartitions.enabled` configuration properties are both enabled. There are four following sub-configurations in this optimization rule. This feature simplifies the tuning of shuffle partitions number when running queries. You don't need to set a proper shuffle partition number to fit your dataset. You just need to set a large enough number and Spark can pick the proper shuffle partition number at runtime.


shuffle partitions number -> the shuffle partition number

HyukjinKwon · 2020-03-10T08:56:30Z

docs/sql-performance-tuning.md

+## Adaptive Query Execution
+Adaptive Query Execution (AQE) is an optimization technique in Spark SQL that makes use of the runtime statistics to choose the most efficient query execution plan. AQE is disabled by default. Spark SQL can use the umbrella configuration of `spark.sql.adaptive.enabled` to control whether turn it on/off. As of Spark 3.0, there are three major features in AQE, including coalescing post-shuffle partitions, local shuffle reader optimization and skewed join optimization.
+ ### Coalescing Post Shuffle Partition Number
+ This feature coalesces the post shuffle partitions based on the map output statistics when `spark.sql.adaptive.enabled` and `spark.sql.adaptive.coalescePartitions.enabled` configuration properties are both enabled. There are four following sub-configurations in this optimization rule. This feature simplifies the tuning of shuffle partitions number when running queries. You don't need to set a proper shuffle partition number to fit your dataset. You just need to set a large enough number and Spark can pick the proper shuffle partition number at runtime.


You don't need to set a proper shuffle partition number to fit your dataset. You just need to set a large enough number and Spark can pick the proper shuffle partition number at runtime.

It sounds like a marketing phrase. I would write like Spark picks a proper shuffle number at runtime once you set the number large enough.

cloud-fan · 2020-03-12T07:34:11Z

docs/sql-performance-tuning.md

+     <td><code>spark.sql.adaptive.coalescePartitions.initialPartitionNum</code></td>
+     <td>200</td>
+     <td>
+       The advisory number of post-shuffle partitions used in adaptive execution. This is used as the initial number of pre-shuffle partitions. By default it equals to <code>spark.sql.shuffle.partitions</code>.


can we copy the doc from SQLConf?

cloud-fan · 2020-03-12T07:34:44Z

docs/sql-performance-tuning.md

+   </tr>
+   <tr>
+     <td><code>spark.sql.adaptive.advisoryPartitionSizeInBytes</code></td>
+     <td>67108864 (64 MB)</td>


We can just write 64 MB

cloud-fan · 2020-03-12T07:35:15Z

docs/sql-performance-tuning.md

+     <td><code>spark.sql.adaptive.advisoryPartitionSizeInBytes</code></td>
+     <td>67108864 (64 MB)</td>
+     <td>
+       The target post-shuffle input size in bytes of a task when <code>spark.sql.adaptive.enabled</code> and <code>spark.sql.adaptive.coalescePartitions.enabled</code> are both enabled.


ditto, let's make sure all the config doc are the same with SQLConf

cloud-fan · 2020-03-12T07:48:20Z

docs/sql-performance-tuning.md

+ ### Optimize Local Shuffle Reader
+ This feature optimize the shuffle reader to local shuffle reader when converting the sort merge join to broadcast hash join in runtime and no additional shuffle introduced. It takes effect when `spark.sql.adaptive.enabled` and `spark.sql.adaptive.localShuffleReader.enabled` configuration properties are both enabled. This feature can improve the performance by saving the network overhead of shuffle process.
+ ### Optimize Skewed Join
+ This feature choose the skewed partition and creates multi tasks to handle the skewed partition when both enable `spark.sql.adaptive.enabled` and `spark.sql.adaptive.skewJoin.enabled`. There are two following sub-configurations in this optimization rule. Data skew can severely downgrade performance of join queries. And this feature can split the skewed partition into multi parallel tasks instead of original 1 task to reduce the overhead of skewed join.


Let's make it simple:

Data skew can severely downgrade the performance of join queries. This feature dynamically handles skew in sort-merge join by splitting (and replicating if needed) skewed tasks into roughly evenly sized tasks. It takes effect when both abc and xyz configurations are enabled.

cloud-fan · 2020-03-12T07:50:44Z

A high-level comment: "Optimize Local Shuffle Reader" should be a sub-feature of converting SMJ to BHJ. We should mention that instead.

JkSelf · 2020-03-13T00:46:05Z

@HyukjinKwon @cloud-fan Thanks for your review. Updated.

SparkQA · 2020-03-13T05:13:01Z

Test build #119733 has finished for PR 27616 at commit 9edc644.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-03-13T11:13:50Z

docs/sql-performance-tuning.md

@@ -186,3 +186,63 @@ The "REPARTITION_BY_RANGE" hint must have column names and a partition number is
    SELECT /*+ REPARTITION(3, c) */ * FROM t
    SELECT /*+ REPARTITION_BY_RANGE(c) */ * FROM t
    SELECT /*+ REPARTITION_BY_RANGE(3, c) */ * FROM t
+
+## Adaptive Query Execution
+Adaptive Query Execution (AQE) is an optimization technique in Spark SQL that makes use of the runtime statistics to choose the most efficient query execution plan. AQE is disabled by default. Spark SQL can use the umbrella configuration of `spark.sql.adaptive.enabled` to control whether turn it on/off. As of Spark 3.0, there are three major features in AQE, including coalescing post-shuffle partitions, local shuffle reader optimization and skewed join optimization.


including coalescing post-shuffle partitions, converting sort-merge join to broadcast join, and skewed join optimization.

local shuffle reader optimization is just one improvement of converting sort-merge join to broadcast join

SparkQA · 2020-03-16T07:05:02Z

Test build #119832 has finished for PR 27616 at commit 13aa51b.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

JkSelf · 2020-03-16T07:08:50Z

retest this please

cloud-fan · 2020-03-16T07:42:02Z

docs/sql-performance-tuning.md

@@ -186,3 +186,63 @@ The "REPARTITION_BY_RANGE" hint must have column names and a partition number is
    SELECT /*+ REPARTITION(3, c) */ * FROM t
    SELECT /*+ REPARTITION_BY_RANGE(c) */ * FROM t
    SELECT /*+ REPARTITION_BY_RANGE(3, c) */ * FROM t
+
+## Adaptive Query Execution
+Adaptive Query Execution (AQE) is an optimization technique in Spark SQL that makes use of the runtime statistics to choose the most efficient query execution plan. AQE is disabled by default. Spark SQL can use the umbrella configuration of `spark.sql.adaptive.enabled` to control whether turn it on/off. As of Spark 3.0, there are three major features in AQE, including coalescing coalescing post-shuffle partitions, converting sort-merge join to broadcast join, and skewed join optimization.


typo: coalescing coalescing

skew join is actually more commonly used.

cloud-fan · 2020-03-16T07:43:00Z

docs/sql-performance-tuning.md

+## Adaptive Query Execution
+Adaptive Query Execution (AQE) is an optimization technique in Spark SQL that makes use of the runtime statistics to choose the most efficient query execution plan. AQE is disabled by default. Spark SQL can use the umbrella configuration of `spark.sql.adaptive.enabled` to control whether turn it on/off. As of Spark 3.0, there are three major features in AQE, including coalescing coalescing post-shuffle partitions, converting sort-merge join to broadcast join, and skewed join optimization.
+
+### Coalescing Post Shuffle Partition Number


Coalescing Post Shuffle Partitios

cloud-fan · 2020-03-16T07:44:03Z

docs/sql-performance-tuning.md

+Adaptive Query Execution (AQE) is an optimization technique in Spark SQL that makes use of the runtime statistics to choose the most efficient query execution plan. AQE is disabled by default. Spark SQL can use the umbrella configuration of `spark.sql.adaptive.enabled` to control whether turn it on/off. As of Spark 3.0, there are three major features in AQE, including coalescing coalescing post-shuffle partitions, converting sort-merge join to broadcast join, and skewed join optimization.
+
+### Coalescing Post Shuffle Partition Number
+This feature coalesces the post shuffle partitions based on the map output statistics when both `spark.sql.adaptive.enabled` and `spark.sql.adaptive.coalescePartitions.enabled` configuration properties are enabled. There are four following sub-configurations in this optimization rule. This feature simplifies the tuning of shuffle partition number when running queries. You do not need to set a proper shuffle partition number to fit your dataset. Spark can pick the proper shuffle partition number at runtime once you set a large enough initial number of shuffle partitions via `spark.sql.adaptive.coalescePartitions.initialPartitionNum` configuration.


when both xxx and yyy configuration properties are enabled reads a little weird. How about when both xxx and yyy configurations are true?

There are four following sub-configurations in this optimization rule. Can we remove this sentence? This looks not useful as users can see all the configs in the following table.

cloud-fan · 2020-03-16T07:47:49Z

docs/sql-performance-tuning.md

+   </tr>
+ </table>
+
+### Optimize Local Shuffle Reader


Converting sort-merge join to broadcast hash join

cloud-fan · 2020-03-16T07:48:05Z

docs/sql-performance-tuning.md

+ </table>
+
+### Optimize Local Shuffle Reader
+AQE converts the sort merge join to broad cast hash join when the runtime statistics of any join side is smaller than the broadcast hash join threshold. This feature can optimize the shuffle reader to local shuffle reader after converting the sort merge join to broadcast hash join at runtime and if no additional shuffle is introduced. It takes effect when both `spark.sql.adaptive.enabled` and `spark.sql.adaptive.localShuffleReader.enabled` configuration properties are enabled. This feature can improve the performance by saving the network overhead of shuffle process.


broad cast -> broadcast

AQE converts sort-merge join to broadcast hash join when the runtime statistics of any join side is smaller than the broadcast hash join threshold. This is not as efficient as planning a broadcast hash join in the first place, but it's better than keep doing the sort-merge join, as we can save the sorting of both the join sides, and read shuffle files locally to save network traffic(if `spark.sql.adaptive.localShuffleReader.enabled` is true)

cloud-fan · 2020-03-16T07:54:17Z

docs/sql-performance-tuning.md

+### Optimize Local Shuffle Reader
+AQE converts the sort merge join to broad cast hash join when the runtime statistics of any join side is smaller than the broadcast hash join threshold. This feature can optimize the shuffle reader to local shuffle reader after converting the sort merge join to broadcast hash join at runtime and if no additional shuffle is introduced. It takes effect when both `spark.sql.adaptive.enabled` and `spark.sql.adaptive.localShuffleReader.enabled` configuration properties are enabled. This feature can improve the performance by saving the network overhead of shuffle process.
+
+### Optimize Skewed Join


Optimizing Skew Join

SparkQA · 2020-03-16T12:45:36Z

Test build #119843 has finished for PR 27616 at commit 13aa51b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-03-16T15:34:21Z

thanks, merging to master/3.0!

### What changes were proposed in this pull request? This PR will add the user guide for AQE and the detailed configurations about the three mainly features in AQE. ### Why are the changes needed? Add the detailed configurations. ### Does this PR introduce any user-facing change? No ### How was this patch tested? only add doc no need ut. Closes #27616 from JkSelf/aqeuserguide. Authored-by: jiake <ke.a.jia@intel.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit 21c02ee) Signed-off-by: Wenchen Fan <wenchen@databricks.com>

SparkQA · 2020-03-16T18:08:00Z

Test build #119868 has finished for PR 27616 at commit 8289200.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

### What changes were proposed in this pull request? This PR will add the user guide for AQE and the detailed configurations about the three mainly features in AQE. ### Why are the changes needed? Add the detailed configurations. ### Does this PR introduce any user-facing change? No ### How was this patch tested? only add doc no need ut. Closes apache#27616 from JkSelf/aqeuserguide. Authored-by: jiake <ke.a.jia@intel.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>

JkSelf added 8 commits March 1, 2005 19:26

add the user guide for aqe

878e30f

address comments

5560a07

address comments and small update in the comment of OptimizeSkewedJoin

246e2a1

address comments

8cdfd55

update the user guide based on the latest configuration refine

ddda91a

address comments

9edc644

fix comment

13aa51b

fix comments

8289200

cloud-fan reviewed Feb 18, 2020

View reviewed changes

cloud-fan reviewed Feb 19, 2020

View reviewed changes

JkSelf force-pushed the aqeuserguide branch from 727f57f to 52fcdf3 Compare February 27, 2020 01:35

dongjoon-hyun added the SQL label Feb 28, 2020

JkSelf force-pushed the aqeuserguide branch from 8ff8b71 to 4fdbe10 Compare March 6, 2020 08:11

HyukjinKwon reviewed Mar 10, 2020

View reviewed changes

docs/sql-performance-tuning.md Outdated

</tr>

</table>

### Optimize Local Shuffle Reader

Copy link

Member

HyukjinKwon Mar 10, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's another leading space here.

HyukjinKwon reviewed Mar 10, 2020

View reviewed changes

cloud-fan reviewed Mar 12, 2020

View reviewed changes

JkSelf force-pushed the aqeuserguide branch from 4fdbe10 to 9edc644 Compare March 13, 2020 00:45

cloud-fan reviewed Mar 13, 2020

View reviewed changes

cloud-fan reviewed Mar 16, 2020

View reviewed changes

cloud-fan changed the title ~~[SPARK-30864] [SQL]add the user guide for Adaptive Query Execution~~ [SPARK-30864][SQL][DOC] add the user guide for Adaptive Query Execution Mar 16, 2020

cloud-fan closed this in 21c02ee Mar 16, 2020

[SPARK-30864][SQL][DOC] add the user guide for Adaptive Query Execution #27616

[SPARK-30864][SQL][DOC] add the user guide for Adaptive Query Execution #27616

Conversation

JkSelf commented Feb 18, 2020

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

JkSelf commented Feb 18, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cloud-fan Feb 18, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Feb 18, 2020

SparkQA commented Feb 18, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Feb 27, 2020

SparkQA commented Feb 27, 2020

SparkQA commented Mar 6, 2020

HyukjinKwon commented Mar 10, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cloud-fan commented Mar 12, 2020

JkSelf commented Mar 13, 2020

SparkQA commented Mar 13, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Mar 16, 2020

JkSelf commented Mar 16, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cloud-fan Mar 16, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cloud-fan Mar 16, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Mar 16, 2020

cloud-fan commented Mar 16, 2020

SparkQA commented Mar 16, 2020

cloud-fan Feb 18, 2020 •

edited

Loading

cloud-fan Mar 16, 2020 •

edited

Loading

cloud-fan Mar 16, 2020 •

edited

Loading