Spark: Remove ImmutableMap from SparkPartition for Kryo SerDe #3667
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kbendick
force-pushed
the
kryo-serialization-on-spark-partition
branch
2 times, most recently
from
7308de4
to
396cfdb
Compare
|
Hmmm it seems like when I rebased, it might have removed the co-authorship from @dubeme. Before we merge this in, can we please ensure that I can update it on my end somehow if need be. |
jackye1995
reviewed
Dec 5, 2021
spark/v3.2/spark/src/test/java/org/apache/iceberg/spark/TestSparkPartitionSerialization.java
Outdated
Show resolved
Hide resolved
Sure thing. I almost did but I wasn’t sure where exactly we’re drawing the line. But this is small and if people need it, they need it for all versions (or can edit one or more out). EDIT: I have covered and added basic tests for Spark 2.4, 3.0, and 3.1. |
kbendick
changed the title
…(spark 3.2) Co-authored-by: Dubem Enyekwe <dubem@dubemenyekwe.com>
kbendick
force-pushed
the
kryo-serialization-on-spark-partition
branch
from
5d0a14b
to
44af3f0
Compare
jackye1995
approved these changes
Dec 5, 2021
rdblue
approved these changes
Dec 5, 2021
|
Thanks for getting this finished, @kbendick! |
This was referenced Dec 5, 2021
|
Thanks @kbendick for stepping in in my absence |
ajantha-bhat
pushed a commit
to ajantha-bhat/iceberg
that referenced
this pull request
Dec 6, 2021
Co-authored-by: Dubem Enyekwe <dubem@dubemenyekwe.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR replaces #3597, as @dubeme has not responded about some minor changes and we need this patch prior to the upcoming 0.13.0 release.
I have added them as a co-author on this PR so that they still get credit for their contribution 😄
Issue #3586 shows situation where Kyro is unable to serialize the SparkPartition object (@rdblue points it out from the stack trace). This commit replaces the ImmutableMap with HashMap and adds a test for round trip Kryo serialization and Java serialization for
SparkPartitionobjects.I also tested this by running the
add_filestest suite with Kryo serialization enabled. The tests failed until this patch was applied. Unfortunately, I couldn't add a test there as the usualwithSQLConfmethod did not pick up the change inspark.serializer.Co-authored-by: Dubem Enyekwe dubem@dubemenyekwe.com
cc @rdblue @jackye1995 @dubeme