Skip to content
Permalink
Browse files

[SPARK-23310][CORE] Turn off read ahead input stream for unshafe shuf…

…fle reader

To fix regression for TPC-DS queries

Author: Sital Kedia <skedia@fb.com>

Closes #20492 from sitalkedia/turn_off_async_inputstream.

(cherry picked from commit 03b7e12)
Signed-off-by: Sameer Agarwal <sameerag@apache.org>
  • Loading branch information...
sitalkedia authored and sameeragarwal committed Feb 5, 2018
1 parent e688ffe commit 173449c2bd454a87487f8eebf7d74ee6ed505290
@@ -76,8 +76,10 @@ public UnsafeSorterSpillReader(
SparkEnv.get() == null ? 0.5 :
SparkEnv.get().conf().getDouble("spark.unsafe.sorter.spill.read.ahead.fraction", 0.5);

// SPARK-23310: Disable read-ahead input stream, because it is causing lock contention and perf regression for
// TPC-DS queries.
final boolean readAheadEnabled = SparkEnv.get() != null &&
SparkEnv.get().conf().getBoolean("spark.unsafe.sorter.spill.read.ahead.enabled", true);
SparkEnv.get().conf().getBoolean("spark.unsafe.sorter.spill.read.ahead.enabled", false);

final InputStream bs =
new NioBufferedFileInputStream(file, (int) bufferSizeBytes);

0 comments on commit 173449c

Please sign in to comment.
You can’t perform that action at this time.