Skip to content

Commit

Permalink
[SPARK-48708][CORE] Remove three unnecessary type registrations from …
Browse files Browse the repository at this point in the history
…`KryoSerializer`

### What changes were proposed in this pull request?
This pr aims to remove three unnecessary type registrations from `KryoSerializer`, they include `None`, `Nil`, and  `scala.collection.immutable.$colon$colon` .

### Why are the changes needed?
The types that have been cleaned up are already registered in `ScalaKryoInstantiator`, so there is no need to register them again in Spark.

https://github.com/twitter/chill/blame/0a34e50f742ab74598c81f153cfe8ad90bf3a859/chill-scala/src/main/scala/com/twitter/chill/ScalaKryoInstantiator.scala#L238-L241

```scala
  None.getClass, // None
  classOf[Queue[_]],
  Nil.getClass, // Nil
  classOf[::[_]], // scala.collection.immutable.$colon$colon
```

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Exist test: "Bug: SPARK-10251" in `org.apache.spark.serializer.KryoSerializerSuite`

### Was this patch authored or co-authored using generative AI tooling?
No

Closes apache#47080 from LuciferYang/clean-kryo-register.

Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: Kent Yao <yao@apache.org>
  • Loading branch information
LuciferYang authored and yaooqinn committed Jun 27, 2024
1 parent 40ad829 commit baf461b
Showing 1 changed file with 0 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -227,10 +227,7 @@ class KryoSerializer(conf: SparkConf)

// scalastyle:on

kryo.register(None.getClass)
kryo.register(Nil.getClass)
kryo.register(Utils.classForName("scala.collection.immutable.ArraySeq$ofRef"))
kryo.register(Utils.classForName("scala.collection.immutable.$colon$colon"))
kryo.register(Utils.classForName("scala.collection.immutable.Map$EmptyMap$"))
kryo.register(Utils.classForName("scala.math.Ordering$Reverse"))
kryo.register(Utils.classForName("scala.reflect.ClassTag$GenericClassTag"))
Expand Down

0 comments on commit baf461b

Please sign in to comment.