Skip to content

Commit

Permalink
[SPARK-31055][DOCS] Update config docs for shuffle local host reads t…
Browse files Browse the repository at this point in the history
…o have dep on external shuffle service

### What changes were proposed in this pull request?

with SPARK-27651 we now support host local reads for shuffle, but only when external shuffle service is enabled. Update the config docs to state that.

### Why are the changes needed?

clarify dependency

### Does this PR introduce any user-facing change?

no

### How was this patch tested?

n/a

Closes #27812 from tgravescs/SPARK-27651-follow.

Authored-by: Thomas Graves <tgraves@nvidia.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
  • Loading branch information
tgravescs authored and dongjoon-hyun committed Mar 9, 2020
1 parent d21aab4 commit e807118
Showing 1 changed file with 2 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -1216,7 +1216,8 @@ package object config {

private[spark] val SHUFFLE_HOST_LOCAL_DISK_READING_ENABLED =
ConfigBuilder("spark.shuffle.readHostLocalDisk")
.doc(s"If enabled (and `${SHUFFLE_USE_OLD_FETCH_PROTOCOL.key}` is disabled), shuffle " +
.doc(s"If enabled (and `${SHUFFLE_USE_OLD_FETCH_PROTOCOL.key}` is disabled and external " +
s"shuffle `${SHUFFLE_SERVICE_ENABLED.key}` is enabled), shuffle " +
"blocks requested from those block managers which are running on the same host are read " +
"from the disk directly instead of being fetched as remote blocks over the network.")
.booleanConf
Expand Down

0 comments on commit e807118

Please sign in to comment.