Skip to content

Commit

Permalink
[SPARK-30908][CORE][DOC] Add version information to the configuration…
Browse files Browse the repository at this point in the history
… of Kryo

### What changes were proposed in this pull request?
1.Add version information to the configuration of `Kryo`.
2.Update the docs of `Kryo`.

I sorted out some information show below.

Item name | Since version | JIRA ID | Commit ID | Note
-- | -- | -- | -- | --
spark.kryo.registrationRequired | 1.1.0 | SPARK-2102 | efdaeb1#diff-1f81c62dad0e2dfc387a974bb08c497c |  
spark.kryo.registrator | 0.5.0 | None | 91c07a3#diff-792ed56b3398163fa14e8578549d0d98 | This is not a release version, do we need to record it?
spark.kryo.classesToRegister | 1.2.0 | SPARK-1813 | 6bb56fa#diff-529fc5c06b9731c1fbda6f3db60b16aa |  
spark.kryo.unsafe | 2.1.0 | SPARK-928 | bc167a2#diff-1f81c62dad0e2dfc387a974bb08c497c |  
spark.kryo.pool | 3.0.0 | SPARK-26466 | 38f0307#diff-a3c6b992784f9abeb9f3047d3dcf3ed9 |  
spark.kryo.referenceTracking | 0.8.0 | None | 0a8cc30#diff-1f81c62dad0e2dfc387a974bb08c497c |  
spark.kryoserializer.buffer | 1.4.0 | SPARK-5932 | 2d222fb#diff-1f81c62dad0e2dfc387a974bb08c497c |  
spark.kryoserializer.buffer.max | 1.4.0 | SPARK-5932 | 2d222fb#diff-1f81c62dad0e2dfc387a974bb08c497c |  

### Why are the changes needed?
Supplemental configuration version information.

### Does this PR introduce any user-facing change?
No

### How was this patch tested?
Exists UT

Closes #27734 from beliefer/add-version-to-kryo-config.

Authored-by: beliefer <beliefer@163.com>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
  • Loading branch information
beliefer authored and HyukjinKwon committed Mar 2, 2020
1 parent a429ac8 commit 3beb4f8
Show file tree
Hide file tree
Showing 2 changed files with 16 additions and 1 deletion.
Expand Up @@ -22,35 +22,43 @@ import org.apache.spark.network.util.ByteUnit
private[spark] object Kryo {

val KRYO_REGISTRATION_REQUIRED = ConfigBuilder("spark.kryo.registrationRequired")
.version("1.1.0")
.booleanConf
.createWithDefault(false)

val KRYO_USER_REGISTRATORS = ConfigBuilder("spark.kryo.registrator")
.version("0.5.0")
.stringConf
.createOptional

val KRYO_CLASSES_TO_REGISTER = ConfigBuilder("spark.kryo.classesToRegister")
.version("1.2.0")
.stringConf
.toSequence
.createWithDefault(Nil)

val KRYO_USE_UNSAFE = ConfigBuilder("spark.kryo.unsafe")
.version("2.1.0")
.booleanConf
.createWithDefault(false)

val KRYO_USE_POOL = ConfigBuilder("spark.kryo.pool")
.version("3.0.0")
.booleanConf
.createWithDefault(true)

val KRYO_REFERENCE_TRACKING = ConfigBuilder("spark.kryo.referenceTracking")
.version("0.8.0")
.booleanConf
.createWithDefault(true)

val KRYO_SERIALIZER_BUFFER_SIZE = ConfigBuilder("spark.kryoserializer.buffer")
.version("1.4.0")
.bytesConf(ByteUnit.KiB)
.createWithDefaultString("64k")

val KRYO_SERIALIZER_MAX_BUFFER_SIZE = ConfigBuilder("spark.kryoserializer.buffer.max")
.version("1.4.0")
.bytesConf(ByteUnit.MiB)
.createWithDefaultString("64m")

Expand Down
9 changes: 8 additions & 1 deletion docs/configuration.md
Expand Up @@ -1224,7 +1224,7 @@ Apart from these, the following properties are also available, and may be useful
### Compression and Serialization

<table class="table">
<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
<tr>
<td><code>spark.broadcast.compress</code></td>
<td>true</td>
Expand Down Expand Up @@ -1298,6 +1298,7 @@ Apart from these, the following properties are also available, and may be useful
with Kryo.
See the <a href="tuning.html#data-serialization">tuning guide</a> for more details.
</td>
<th>1.2.0</th>
</tr>
<tr>
<td><code>spark.kryo.referenceTracking</code></td>
Expand All @@ -1308,6 +1309,7 @@ Apart from these, the following properties are also available, and may be useful
copies of the same object. Can be disabled to improve performance if you know this is not the
case.
</td>
<th>0.8.0</th>
</tr>
<tr>
<td><code>spark.kryo.registrationRequired</code></td>
Expand All @@ -1319,6 +1321,7 @@ Apart from these, the following properties are also available, and may be useful
significant performance overhead, so enabling this option can enforce strictly that a
user has not omitted classes from registration.
</td>
<th>1.1.0</th>
</tr>
<tr>
<td><code>spark.kryo.registrator</code></td>
Expand All @@ -1332,6 +1335,7 @@ Apart from these, the following properties are also available, and may be useful
<code>KryoRegistrator</code></a>.
See the <a href="tuning.html#data-serialization">tuning guide</a> for more details.
</td>
<th>0.5.0</th>
</tr>
<tr>
<td><code>spark.kryo.unsafe</code></td>
Expand All @@ -1340,6 +1344,7 @@ Apart from these, the following properties are also available, and may be useful
Whether to use unsafe based Kryo serializer. Can be
substantially faster by using Unsafe Based IO.
</td>
<th>2.1.0</th>
</tr>
<tr>
<td><code>spark.kryoserializer.buffer.max</code></td>
Expand All @@ -1349,6 +1354,7 @@ Apart from these, the following properties are also available, and may be useful
This must be larger than any object you attempt to serialize and must be less than 2048m.
Increase this if you get a "buffer limit exceeded" exception inside Kryo.
</td>
<th>1.4.0</th>
</tr>
<tr>
<td><code>spark.kryoserializer.buffer</code></td>
Expand All @@ -1358,6 +1364,7 @@ Apart from these, the following properties are also available, and may be useful
Note that there will be one buffer <i>per core</i> on each worker. This buffer will grow up to
<code>spark.kryoserializer.buffer.max</code> if needed.
</td>
<th>1.4.0</th>
</tr>
<tr>
<td><code>spark.rdd.compress</code></td>
Expand Down

0 comments on commit 3beb4f8

Please sign in to comment.