Skip to content
Browse files

Adding docs for spark.serializer.objectStreamReset configuration

  • Loading branch information...
1 parent 7ccc74b commit f70d06939bb9c164a0a6c9af42f663bc882c3211 @kellrott kellrott committed
Showing with 11 additions and 0 deletions.
  1. +11 −0 docs/configuration.md
View
11 docs/configuration.md
@@ -238,6 +238,17 @@ Apart from these, the following properties are also available, and may be useful
</td>
</tr>
<tr>
+ <td>spark.serializer.objectStreamReset</td>
+ <td>10000</td>
+ <td>
+ When serializing using org.apache.spark.serializer.JavaSerializer, the serializer caches
+ objects to prevent writing redundant data, however that stops garbage collection of those
+ objects. By calling 'reset' you flush that info from the serializer, and allow old
+ objects to be collected. To turn off this periodic reset set it to a value of <= 0.
+ By default it will reset the serializer every 10,000 objects.
+ </td>
+</tr>
+<tr>
<td>spark.broadcast.factory</td>
<td>org.apache.spark.broadcast.<br />HttpBroadcastFactory</td>
<td>

0 comments on commit f70d069

Please sign in to comment.
Something went wrong with that request. Please try again.