[SPARK-38028][SQL] Expose Arrow Vector from ArrowColumnVector #35326
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dongjoon-hyun
approved these changes
Jan 26, 2022
|
I guess it's fine to expose .. @BryanCutler FYI. |
|
Merged to master for Apache Spark 3.3. |
|
Thank you @dongjoon-hyun @sunchao @HyukjinKwon |
|
A little late, but SGTM |
|
Thank you, @BryanCutler . |
|
Thank you @BryanCutler |
senthh
pushed a commit
to senthh/spark-1
that referenced
this pull request
Feb 3, 2022
### What changes were proposed in this pull request? This change exposes Arrow Vector from `ArrowColumnVector`. ### Why are the changes needed? In some cases we need to work with Arrow Vectors behind `ColumnVector` using Arrow APIs. For example, some Spark extension libraries need to consume Arrow Vectors. For now, it is impossible as the Arrow Vector is private member in `ArrowColumnVector`. We need to expose the Arrow Vector from `ArrowColumnVector`. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Existing tests. Closes apache#35326 from viirya/arrow_vector. Authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This change exposes Arrow Vector from
ArrowColumnVector.Why are the changes needed?
In some cases we need to work with Arrow Vectors behind
ColumnVectorusing Arrow APIs. For example, some Spark extension libraries need to consume Arrow Vectors. For now, it is impossible as the Arrow Vector is private member inArrowColumnVector. We need to expose the Arrow Vector fromArrowColumnVector.Does this PR introduce any user-facing change?
No
How was this patch tested?
Existing tests.