Skip to content

Commit

Permalink
[SPARK-42097][CORE] Register SerializedLambda and BitSet to KryoS…
Browse files Browse the repository at this point in the history
…erializer

### What changes were proposed in this pull request?

This PR aims to register `SerializedLambda` and `BitSet` to KryoSerializer.

### Why are the changes needed?

Although SPARK-42074 enabled `spark.kryo.registrationRequired` in `TPCDSQueryBenchmark`, `SerializedLambda` and `BitSet` seems to occur during code gen part.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass the CIs.

Closes #39621 from dongjoon-hyun/SPARK-42097.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
  • Loading branch information
dongjoon-hyun committed Jan 17, 2023
1 parent 489c642 commit 43d6b70
Showing 1 changed file with 5 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
package org.apache.spark.serializer

import java.io._
import java.lang.invoke.SerializedLambda
import java.nio.ByteBuffer
import java.util.Locale
import javax.annotation.Nullable
Expand Down Expand Up @@ -45,7 +46,7 @@ import org.apache.spark.network.util.ByteUnit
import org.apache.spark.scheduler.{CompressedMapStatus, HighlyCompressedMapStatus}
import org.apache.spark.storage._
import org.apache.spark.util.{BoundedPriorityQueue, ByteBufferInputStream, SerializableConfiguration, SerializableJobConf, Utils}
import org.apache.spark.util.collection.CompactBuffer
import org.apache.spark.util.collection.{BitSet, CompactBuffer}
import org.apache.spark.util.io.ChunkedByteBuffer

/**
Expand Down Expand Up @@ -487,7 +488,9 @@ private[serializer] object KryoSerializer {
classOf[Array[Array[String]]],
classOf[BoundedPriorityQueue[_]],
classOf[SparkConf],
classOf[TaskCommitMessage]
classOf[TaskCommitMessage],
classOf[SerializedLambda],
classOf[BitSet]
)

private val toRegisterSerializer = Map[Class[_], KryoClassSerializer[_]](
Expand Down

0 comments on commit 43d6b70

Please sign in to comment.