Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-21133][CORE] Fix HighlyCompressedMapStatus#writeExternal throws NPE #18343

Closed
wants to merge 6 commits into from
Closed

[SPARK-21133][CORE] Fix HighlyCompressedMapStatus#writeExternal throws NPE #18343

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
4cf3532
Fix HighlyCompressedMapStatus#writeExternal throws NPE
wangyum Jun 18, 2017
75a9bf1
Kryo register scala.collection.immutable.Map#EmptyMap#
wangyum Jun 18, 2017
e2816ec
Remove useless register.
wangyum Jun 19, 2017
facca95
Extends KryoSerializable
wangyum Jun 19, 2017
e045bef
Add unit test.
wangyum Jun 19, 2017
7a4e6ec
Add sc.stop()
wangyum Jun 19, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala
View file
Open in desktop
Expand Up @@ -141,7 +141,7 @@ private[spark] class HighlyCompressedMapStatus private (
private[this] var numNonEmptyBlocks: Int,
private[this] var emptyBlocks: RoaringBitmap,
private[this] var avgSize: Long,
@transient private var hugeBlockSizes: Map[Int, Byte])
private[this] var hugeBlockSizes: Map[Int, Byte])
Copy link
Contributor

@cloud-fan cloud-fan Jun 19, 2017 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we do want to serialize hugeBlockSizes, but with customized logic, that why we marked it @transient.

I think the corrected fix is, make this class implements KryoSerializable, and copy the customized serialization logic of hugeBlockSizes to kryo serialization hooks.

Copy link
Member

@viirya viirya Jun 19, 2017 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good to me. However, the customized serialization logic looks similar to kyro's map serializer. So I'm not sure if it's worth duplicating the customized logic for kyro.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you can figure out a way to make it serializable with kryo and still keep the customized serialization logic for java serializer, I'm ok with it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh seems it is now, then LGTM

extends MapStatus with Externalizable {

// loc could be null when the default constructor is called during deserialization
Expand Down
1 change: 1 addition & 0 deletions core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
View file
Open in desktop
Expand Up @@ -175,6 +175,7 @@ class KryoSerializer(conf: SparkConf)
kryo.register(None.getClass)
kryo.register(Nil.getClass)
kryo.register(Utils.classForName("scala.collection.immutable.$colon$colon"))
kryo.register(Utils.classForName("scala.collection.immutable.Map$EmptyMap$"))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why Map$EmptyMap$?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because this test failed: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78224/testReport/org.apache.spark.serializer/KryoSerializerSuite/registration_of_HighlyCompressedMapStatus/

kryo.register(classOf[ArrayBuffer[Any]])

kryo.setClassLoader(classLoader)
Expand Down