Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Keep type ids in Union datatype to follow Arrow spec and integrate with other implementations #1690

Closed
viirya opened this issue May 12, 2022 · 1 comment · Fixed by #1703
Closed
Assignees
Labels
arrow Changes to the arrow crate enhancement Any new improvement worthy of a entry in the changelog

Comments

@viirya
Copy link
Member

viirya commented May 12, 2022

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Related to #1404, generate_unions_case integration test case.

We keep a vector of Field and a UnionMode in Union datatype now. According to Arrow spec, it only specifies the type ids are 8bit signed integers. So the type ids can be arbitrary integers only if they can be matched with children arrays' type ids. This is how UnionType in C++ implementation works.

Currently this crate assumes that the type ids of Union are monotonically increasing from 0 (i.e., 0, 1, 2...). When integrating with other implementations like C++, this assumption will cause error in receiving/sending UnionArray.

Describe the solution you'd like

We should keep type ids in Union datatype like C++ UnionType. The type ids are given when creating Union datatype. UnionArray related code should not assume the type ids are monotonically increasing from 0.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

@viirya viirya added the enhancement Any new improvement worthy of a entry in the changelog label May 12, 2022
@viirya viirya self-assigned this May 12, 2022
@viirya
Copy link
Member Author

viirya commented May 12, 2022

I will work on this and propose the PR later.

@alamb alamb added the arrow Changes to the arrow crate label May 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arrow Changes to the arrow crate enhancement Any new improvement worthy of a entry in the changelog
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants