Skip to content

Commit

Permalink
[SPARK-42063][CORE] Register byte[][] to KyroSerializer
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

This PR aims to register `byte[][]` to `KryoSerializer`.

### Why are the changes needed?

Spark uses `byte[][]` internally and it causes failure when we use `spark.kryo.registrationRequired=true`.
```
java.lang.IllegalArgumentException: Class is not registered: byte[][]
Note: To register this class use: kryo.register(byte[][].class);
```

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass the CIs with the newly added test case.

Closes #39569 from dongjoon-hyun/SPARK-42063.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Yuming Wang <yumwang@ebay.com>
  • Loading branch information
dongjoon-hyun authored and wangyum committed Jan 14, 2023
1 parent e424499 commit e6c01ce
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -220,6 +220,7 @@ class KryoSerializer(conf: SparkConf)
kryo.register(Utils.classForName("scala.collection.immutable.$colon$colon"))
kryo.register(Utils.classForName("scala.collection.immutable.Map$EmptyMap$"))
kryo.register(classOf[ArrayBuffer[Any]])
kryo.register(classOf[Array[Array[Byte]]])

// We can't load those class directly in order to avoid unnecessary jar dependencies.
// We load them safely, ignore it if the class not found.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,7 @@ class KryoSerializerSuite extends SparkFunSuite with SharedSparkContext {
check(java.lang.Long.MAX_VALUE)
check(java.lang.Long.MIN_VALUE)
check[String](null)
check(Array(1.toByte))
check(Array(1, 2, 3))
check(Array(1L, 2L, 3L))
check(Array(1.0, 2.0, 3.0))
Expand All @@ -114,6 +115,7 @@ class KryoSerializerSuite extends SparkFunSuite with SharedSparkContext {
check(Array('a', 'b', 'c'))
check(Array.empty[Int])
check(Array(Array("1", "2"), Array("1", "2", "3", "4")))
check(Array(Array(1.toByte)))
}

test("pairs") {
Expand Down

0 comments on commit e6c01ce

Please sign in to comment.