Permalink
Browse files

Adding docs for spark.serializer.objectStreamReset configuration

  • Loading branch information...
kellrott committed Feb 27, 2014
1 parent 7ccc74b commit f70d06939bb9c164a0a6c9af42f663bc882c3211
Showing with 11 additions and 0 deletions.
  1. +11 −0 docs/configuration.md
View
@@ -237,6 +237,17 @@ Apart from these, the following properties are also available, and may be useful
exceeded" exception inside Kryo. Note that there will be one buffer <i>per core</i> on each worker.
</td>
</tr>
+<tr>
+ <td>spark.serializer.objectStreamReset</td>
+ <td>10000</td>
+ <td>
+ When serializing using org.apache.spark.serializer.JavaSerializer, the serializer caches
+ objects to prevent writing redundant data, however that stops garbage collection of those
+ objects. By calling 'reset' you flush that info from the serializer, and allow old
+ objects to be collected. To turn off this periodic reset set it to a value of <= 0.
+ By default it will reset the serializer every 10,000 objects.
+ </td>
+</tr>
<tr>
<td>spark.broadcast.factory</td>
<td>org.apache.spark.broadcast.<br />HttpBroadcastFactory</td>

0 comments on commit f70d069

Please sign in to comment.