TODO: write fuzz tests to uncover stuff like this.
Sorting has nice invariants; should be an easy test
to write.

…nsafeShuffle path is used.

…nsert()

I verified that the Kryo tests will fail if we remove the auto-reset
check in KryoSerializer. I also checked that this test fails if we
mistakenly enable this flag for JavaSerializer. This demonstrates that
the test case is actually capable of detecting the types of bugs that it's
trying to prevent.

Of course, it's possible that certain bugs will only surface when serializing
specific data types, so we'll still have to be cautious when overriding
`supportsRelocationOfSerializedObjects` for new serializers.

This lays some groundwork for re-using this test logic for serializers defined
in other subprojects (those projects can just declare a test-jar dependency
on Spark core).

This broke off-heap mode.

Unfortunately, this involved a TON of mocks; maybe it would be easier to split
the writer into more objects, such as a spiller and merger, as I did when the
sorting code was more generic.

Conflicts:
	core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
	core/src/main/scala/org/apache/spark/serializer/Serializer.scala
	core/src/test/scala/org/apache/spark/serializer/SerializerPropertiesSuite.scala
	sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlSerializer2.scala

This bound wasn't necessary and was causing IntelliJ to display spurious
errors when editing UnsafeShuffleWriter.java.

…er errors

…r rename.

…anager

…figurations.

It turns out that we actually rely on these flushing the underlying
stream in order to properly close streams in DiskBlockObjectWriter;
it was silly of me to not implement these methods.

This should fix a failing LZ4 test in UnsafeShuffleWriterSuite.

We'll want to revisit this before merging, since the large minimum memory
usage means that minimum memory requirements for shuffle may be fairly
high for local tests.

…ugs!)

This bumps up line coverage to 93% in UnsafeShuffleExternalSorter; now,
the only branches that are missed are exception-handling code.

There's no obvious use-case for allowing users to disable spark.shuffle.spill
and run out of memory. Because this configuration isn't deprecated as of this
patch, I've added code to log a warning to let users know if their preference
will be ignored by the new shuffle manager.

Conflicts:
	project/MimaExcludes.scala

…ream into separate file.

The problem is that TaskMemoryManager expects
offsets to include the page base address whereas
PackedRecordPointer did not.

This caused a failure in a new test; this problem
occurs when calls ShuffledRDD.setSerializer() with
a JavaSerializer.