[java] Support Arrow batch writes across allocator roots - #73
Open
jianguotian wants to merge 2 commits into
Open
[java] Support Arrow batch writes across allocator roots#73jianguotian wants to merge 2 commits into
jianguotian wants to merge 2 commits into
Conversation
Keep same-root batches on the existing Arrow C Data export path. For independent allocator roots, export field arrays directly while charging temporary metadata to the writer allocator and rolling back partial failures. Validate nested allocator roots and cover ownership, accounting, retry, and cleanup behavior.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
MosaicWriter.write(VectorSchemaRoot)exports incoming Arrow batches through the Arrow C Data interface. Arrow 15'sData.exportVectorSchemaRootreloads the fields into a temporary root struct, which cannot reload buffers owned by an independent allocator root into the writer allocator.Support those batches without moving or reassociating their buffers: keep the existing same-root path, export cross-root fields directly, and retain temporary Arrow C Data metadata under the writer allocator. Make failed cross-root exports rollback-safe so an allocator OOM or partial child export does not retain input buffers.
Changes
Data.exportVectorSchemaRootpath.Tests
cargo fmt --all -- --checkpassed.git diff --checkpassed.