Skip to content

Commit

Permalink
[SPARK-10469] [DOC] Try and document the three options
Browse files Browse the repository at this point in the history
From JIRA:
Add documentation for tungsten-sort.
From the mailing list "I saw a new "spark.shuffle.manager=tungsten-sort" implemented in
https://issues.apache.org/jira/browse/SPARK-7081, but it can't be found its
corresponding description in
http://people.apache.org/~pwendell/spark-releases/spark-1.5.0-rc3-docs/configuration.html(Currenlty
there are only 'sort' and 'hash' two options)."

Author: Holden Karau <holden@pigscanfly.ca>

Closes #8638 from holdenk/SPARK-10469-document-tungsten-sort.
  • Loading branch information
holdenk authored and Andrew Or committed Sep 10, 2015
1 parent e048111 commit a76bde9
Showing 1 changed file with 6 additions and 3 deletions.
9 changes: 6 additions & 3 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -447,9 +447,12 @@ Apart from these, the following properties are also available, and may be useful
<td><code>spark.shuffle.manager</code></td>
<td>sort</td>
<td>
Implementation to use for shuffling data. There are two implementations available:
<code>sort</code> and <code>hash</code>. Sort-based shuffle is more memory-efficient and is
the default option starting in 1.2.
Implementation to use for shuffling data. There are three implementations available:
<code>sort</code>, <code>hash</code> and the new (1.5+) <code>tungsten-sort</code>.
Sort-based shuffle is more memory-efficient and is the default option starting in 1.2.
Tungsten-sort is similar to the sort based shuffle, with a direct binary cache-friendly

This comment has been minimized.

Copy link
@jaceklaskowski

jaceklaskowski Sep 14, 2015

Contributor
  • Use <code>tungsten-sort</code> for better reading experience.
  • It should be sort-based shuffle (note the hyphen) to be consistent to the other part of the sentence (also in the following line).
implementation with a fall back to regular sort based shuffle if its requirements are not
met.
</td>
</tr>
<tr>
Expand Down

0 comments on commit a76bde9

Please sign in to comment.