From f70d06939bb9c164a0a6c9af42f663bc882c3211 Mon Sep 17 00:00:00 2001 From: Kyle Ellrott Date: Thu, 27 Feb 2014 13:36:06 -0800 Subject: [PATCH] Adding docs for spark.serializer.objectStreamReset configuration --- docs/configuration.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/configuration.md b/docs/configuration.md index 8e4c48c81f8be..260d38dd14da6 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -237,6 +237,17 @@ Apart from these, the following properties are also available, and may be useful exceeded" exception inside Kryo. Note that there will be one buffer per core on each worker. + + spark.serializer.objectStreamReset + 10000 + + When serializing using org.apache.spark.serializer.JavaSerializer, the serializer caches + objects to prevent writing redundant data, however that stops garbage collection of those + objects. By calling 'reset' you flush that info from the serializer, and allow old + objects to be collected. To turn off this periodic reset set it to a value of <= 0. + By default it will reset the serializer every 10,000 objects. + + spark.broadcast.factory org.apache.spark.broadcast.
HttpBroadcastFactory