Skip to content

Commit

Permalink
Bump up shuffle.memoryFraction to make tests pass.
Browse files Browse the repository at this point in the history
We'll want to revisit this before merging, since the large minimum memory
usage means that minimum memory requirements for shuffle may be fairly
high for local tests.
  • Loading branch information
JoshRosen committed May 11, 2015
1 parent b3b1924 commit 0d4d199
Show file tree
Hide file tree
Showing 2 changed files with 4 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -337,7 +337,7 @@ private void allocateSpaceForRecord(int requiredSpace) throws IOException {
final long memoryAcquiredAfterSpilling = shuffleMemoryManager.tryToAcquire(PAGE_SIZE);
if (memoryAcquiredAfterSpilling != PAGE_SIZE) {
shuffleMemoryManager.release(memoryAcquiredAfterSpilling);
throw new IOException("Can't allocate memory!");
throw new IOException("Unable to acquire " + PAGE_SIZE + " bytes of memory");
}
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,5 +26,8 @@ class UnsafeShuffleSuite extends ShuffleSuite with BeforeAndAfterAll {

override def beforeAll() {
conf.set("spark.shuffle.manager", "unsafe")
// UnsafeShuffleManager requires at least 128 MB of memory per task in order to be able to sort
// shuffle records.
conf.set("spark.shuffle.memoryFraction", "0.5")
}
}

0 comments on commit 0d4d199

Please sign in to comment.