Skip to content

[java] Support Arrow batch writes across allocator roots - #73

Open
jianguotian wants to merge 2 commits into
apache:mainfrom
jianguotian:fix/cross-root-arrow-export
Open

[java] Support Arrow batch writes across allocator roots#73
jianguotian wants to merge 2 commits into
apache:mainfrom
jianguotian:fix/cross-root-arrow-export

Conversation

@jianguotian

@jianguotian jianguotian commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Purpose

MosaicWriter.write(VectorSchemaRoot) exports incoming Arrow batches through the Arrow C Data interface. Arrow 15's Data.exportVectorSchemaRoot reloads the fields into a temporary root struct, which cannot reload buffers owned by an independent allocator root into the writer allocator.

Support those batches without moving or reassociating their buffers: keep the existing same-root path, export cross-root fields directly, and retain temporary Arrow C Data metadata under the writer allocator. Make failed cross-root exports rollback-safe so an allocator OOM or partial child export does not retain input buffers.

Changes

  • Recursively validate that all top-level and nested input vectors share one input allocator root.
  • Keep same-root batches on the existing Data.exportVectorSchemaRoot path.
  • Export cross-root schemas and field arrays directly while charging temporary C Data metadata to the writer allocator.
  • Release all successfully exported children and temporary metadata when a later cross-root export step fails.
  • Document the synchronous batch lifetime, allocator-root requirement, and writer-owned metadata accounting.
  • Keep the native writer API, Rust codec, and on-disk format unchanged.

Tests

  • JDK 8: 55 tests passed, 0 failed, 0 errors, 0 skipped.
  • JDK 11: 55 tests passed, 0 failed, 0 errors, 0 skipped.
  • Covered independent roots, same-root accounting, sequential roots, nested mixed-root validation, writer allocator OOM retry, and partial child-export rollback.
  • cargo fmt --all -- --check passed.
  • git diff --check passed.

Keep same-root batches on the existing Arrow C Data export path. For independent allocator roots, export field arrays directly while charging temporary metadata to the writer allocator and rolling back partial failures. Validate nested allocator roots and cover ownership, accounting, retry, and cleanup behavior.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant