Skip to content

Commit 25889d8

Browse files
Evan Jonessrowen
authored andcommitted
[SPARK-7490] [CORE] [Minor] MapOutputTracker.deserializeMapStatuses: close input streams
GZIPInputStream allocates native memory that is not freed until close() or when the finalizer runs. It is best to close() these streams explicitly. stephenh made the same change for serializeMapStatuses in commit b0d884f. This is the same change for deserialize. (I ran the unit test suite! it seems to have passed. I did not make a JIRA since this seems "trivial", and the guidelines suggest it is not required for trivial changes) Author: Evan Jones <ejones@twitter.com> Closes #5982 from evanj/master and squashes the following commits: 0d76e85 [Evan Jones] [CORE] MapOutputTracker.deserializeMapStatuses: close input streams
1 parent 4b3bb0e commit 25889d8

File tree

1 file changed

+5
-1
lines changed

1 file changed

+5
-1
lines changed

core/src/main/scala/org/apache/spark/MapOutputTracker.scala

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -367,7 +367,11 @@ private[spark] object MapOutputTracker extends Logging {
367367
// Opposite of serializeMapStatuses.
368368
def deserializeMapStatuses(bytes: Array[Byte]): Array[MapStatus] = {
369369
val objIn = new ObjectInputStream(new GZIPInputStream(new ByteArrayInputStream(bytes)))
370-
objIn.readObject().asInstanceOf[Array[MapStatus]]
370+
Utils.tryWithSafeFinally {
371+
objIn.readObject().asInstanceOf[Array[MapStatus]]
372+
} {
373+
objIn.close()
374+
}
371375
}
372376

373377
// Convert an array of MapStatuses to locations and sizes for a given reduce ID. If

0 commit comments

Comments
 (0)