HIVE-25553: Support Map data-type natively in Arrow format #2751
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This covers the following sub-tasks as well: HIVE-25554: Upgrade arrow version to 0.15 HIVE-25555: ArrowColumnarBatchSerDe should store map natively instead of converting to list What changes were proposed in this pull request? a. Upgrading arrow version to version 0.15.0 (where map data-type is supported) b. Modifying ArrowColumnarBatchSerDe and corresponding Serializer/Deserializer to not use list as a workaround for map and use the arrow map data-type instead c. Taking care of creating non-nullable struct and non-nullable key type for the map data-type in ArrowColumnarBatchSerDe Why are the changes needed? Currently ArrowColumnarBatchSerDe converts map datatype as a list of structs data-type (where struct is containing the key-value pair of the map). This causes issues when reading Map datatype using llap-ext-client as it reads a list of structs instead. HiveWarehouseConnector which uses the llap-ext-client throws exception when the schema (containing Map data type) is different from actual data (list of structs). This change includes the fix for this issue. Does this PR introduce any user-facing change? No How was this patch tested? Enabled back the Arrow specific tests in Hive code
kgyrtkirk
added
tests pending
tests unstable
tests passed
and removed
tests pending
tests unstable
labels
sankarh
approved these changes
Oct 27, 2021
HarshitGupta11
pushed a commit
to HarshitGupta11/hive
that referenced
this pull request
Dec 12, 2021
…oriyathvariam, reviewed by Sankar Hariappan) This covers the following sub-tasks: HIVE-25554: Upgrade arrow version to 0.15 HIVE-25555: ArrowColumnarBatchSerDe should store map natively instead of converting to list a. Upgrading arrow version to version 0.15.0 (where map data-type is supported) b. Modifying ArrowColumnarBatchSerDe and corresponding Serializer/Deserializer to not use list as a workaround for map and use the arrow map data-type instead c. Taking care of creating non-nullable struct and non-nullable key type for the map data-type in ArrowColumnarBatchSerDe Signed-off-by: Sankar Hariappan <sankarh@apache.org> Closes (apache#2751)
dengzhhu653
pushed a commit
to dengzhhu653/hive
that referenced
this pull request
Dec 15, 2022
…oriyathvariam, reviewed by Sankar Hariappan) This covers the following sub-tasks: HIVE-25554: Upgrade arrow version to 0.15 HIVE-25555: ArrowColumnarBatchSerDe should store map natively instead of converting to list a. Upgrading arrow version to version 0.15.0 (where map data-type is supported) b. Modifying ArrowColumnarBatchSerDe and corresponding Serializer/Deserializer to not use list as a workaround for map and use the arrow map data-type instead c. Taking care of creating non-nullable struct and non-nullable key type for the map data-type in ArrowColumnarBatchSerDe Signed-off-by: Sankar Hariappan <sankarh@apache.org> Closes (apache#2751)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This covers the following sub-tasks as well:
HIVE-25554: Upgrade arrow version to 0.15
HIVE-25555: ArrowColumnarBatchSerDe should store map natively instead of converting to list
What changes were proposed in this pull request?
a. Upgrading arrow version to version 0.15.0 (where map data-type is supported)
b. Modifying ArrowColumnarBatchSerDe and corresponding Serializer/Deserializer to not use list as a workaround for map and use the arrow map data-type instead
c. Taking care of creating non-nullable struct and non-nullable key type for the map data-type in ArrowColumnarBatchSerDe
Why are the changes needed?
Currently, ArrowColumnarBatchSerDe converts map datatype as a list of structs data-type (where the struct is containing the key-value pair of the map).
This causes issues when reading Map datatype using llap-ext-client as it reads a list of structs instead.
HiveWarehouseConnector which uses the llap-ext-client throws exception when the schema (containing Map data type) is different from actual data (list of structs).
This change includes the fix for this issue.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Enabled back the Arrow specific tests in Hive code