[SPARK-27416][SQL]UnsafeMapData & UnsafeArrayData Kryo serialization … #24357
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…breaks when two machines have different Oops size
|
I have put Unsafe data class in KryoSerializer.newKryo. Please recheck when available. Thanks |
|
CC @cloud-fan |
kiszk
reviewed
Apr 14, 2019
| */ | ||
| final class UnsafeDataUtils { | ||
|
|
||
| private UnsafeDataUtils() { |
There was a problem hiding this comment.
Thanks for pointing it out.
kiszk
reviewed
Apr 14, 2019
| import org.apache.spark.sql.catalyst.util.MapData; | ||
| import org.apache.spark.unsafe.Platform; | ||
|
|
||
| import java.io.Externalizable; |
There was a problem hiding this comment.
It looks incorrect import order. org.apache should be last.
kiszk
reviewed
Apr 14, 2019
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/UnsafeMapSuite.scala
Show resolved
Hide resolved
kiszk
reviewed
Apr 14, 2019
| val valueArray = new UnsafeArrayData() | ||
| Platform.putLong(baseObject, offset + 8 + keyArray.getSizeInBytes, 1) | ||
| valueArray.pointTo(baseObject, offset + 8 + keyArray.getSizeInBytes, keyArraySize) | ||
| valueArray.setLong(0, 19285) |
There was a problem hiding this comment.
It may be good to use different values among key and value.
There was a problem hiding this comment.
Actually I don't think that makes any different, can you please explain a little more?
Thanks.
There was a problem hiding this comment.
We hope so. For testing typical use cases, I think that it is good to allocate a separate array for map and value.
There was a problem hiding this comment.
Maybe I am missing sth, the key/value array are currently different arrays. The value set is the same (0 -> 19285).
There was a problem hiding this comment.
How about using a different value for key and value, not just 19285 both times? that would make a slightly better test. Otherwise you'd possibly miss a weird bug where key and value arrays are swapped
|
ok to test |
kiszk
reviewed
Apr 15, 2019
| } | ||
| byte[] bytes = new byte[sizeInBytes]; | ||
| Platform.copyMemory(baseObject, baseOffset, bytes, Platform.BYTE_ARRAY_OFFSET, | ||
| sizeInBytes); |
|
Test build #104590 has finished for PR 24357 at commit
|
|
retest this please. |
|
Test build #104621 has finished for PR 24357 at commit
|
kiszk
approved these changes
Apr 16, 2019
srowen
approved these changes
Apr 16, 2019
cloud-fan
changed the title
|
thanks, merging to master! |
JoshRosen
pushed a commit
to JoshRosen/spark
that referenced
this pull request
Jul 22, 2019
Finish the rest work of apache#24317, apache#9030 a. Implement Kryo serialization for UnsafeArrayData b. fix UnsafeMapData Java/Kryo Serialization issue when two machines have different Oops size c. Move the duplicate code "getBytes()" to Utils. According Units has been added & tested Closes apache#24357 from pengbo/SPARK-27416_new. Authored-by: pengbo <bo.peng1019@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
dongjoon-hyun
pushed a commit
that referenced
this pull request
Jul 22, 2019
…erialization … This is a Spark 2.4.x backport of #24357 by pengbo. Original description follows below: --- ## What changes were proposed in this pull request? Finish the rest work of #24317, #9030 a. Implement Kryo serialization for UnsafeArrayData b. fix UnsafeMapData Java/Kryo Serialization issue when two machines have different Oops size c. Move the duplicate code "getBytes()" to Utils. ## How was this patch tested? According Units has been added & tested Closes #25223 from JoshRosen/SPARK-27416-2.4. Authored-by: pengbo <bo.peng1019@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
rluta
pushed a commit
to rluta/spark
that referenced
this pull request
Sep 17, 2019
…erialization … This is a Spark 2.4.x backport of apache#24357 by pengbo. Original description follows below: --- ## What changes were proposed in this pull request? Finish the rest work of apache#24317, apache#9030 a. Implement Kryo serialization for UnsafeArrayData b. fix UnsafeMapData Java/Kryo Serialization issue when two machines have different Oops size c. Move the duplicate code "getBytes()" to Utils. ## How was this patch tested? According Units has been added & tested Closes apache#25223 from JoshRosen/SPARK-27416-2.4. Authored-by: pengbo <bo.peng1019@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
kai-chi
pushed a commit
to kai-chi/spark
that referenced
this pull request
Sep 26, 2019
…erialization … This is a Spark 2.4.x backport of apache#24357 by pengbo. Original description follows below: --- ## What changes were proposed in this pull request? Finish the rest work of apache#24317, apache#9030 a. Implement Kryo serialization for UnsafeArrayData b. fix UnsafeMapData Java/Kryo Serialization issue when two machines have different Oops size c. Move the duplicate code "getBytes()" to Utils. ## How was this patch tested? According Units has been added & tested Closes apache#25223 from JoshRosen/SPARK-27416-2.4. Authored-by: pengbo <bo.peng1019@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
jaltekruse
added a commit
to jaltekruse/spark
that referenced
this pull request
Dec 10, 2019
…erialization This is a Spark 2.3.x backport of a 2.4.x backport of apache#24357 by pengbo. Original description follows below: --- Finish the rest work of apache#24317, apache#9030 a. Implement Kryo serialization for UnsafeArrayData b. fix UnsafeMapData Java/Kryo Serialization issue when two machines have different Oops size c. Move the duplicate code "getBytes()" to Utils. According Units has been added & tested Authored-by: pengbo <bo.peng1019@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Jason Altekruse <json@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Finish the rest work of #24317, #9030
a. Implement Kryo serialization for UnsafeArrayData
b. fix UnsafeMapData Java/Kryo Serialization issue when two machines have different Oops size
c. Move the duplicate code "getBytes()" to Utils.
How was this patch tested?
According Units has been added & tested