Arrow: Fix double-scaled setInitialCapacity for vectorized reads#17057
Open
thswlsqls wants to merge 1 commit into
Open
Arrow: Fix double-scaled setInitialCapacity for vectorized reads#17057thswlsqls wants to merge 1 commit into
thswlsqls wants to merge 1 commit into
Conversation
allocateVectorBasedOnTypeName() passed batchSize * typeWidth to Arrow's setInitialCapacity(int), which already multiplies by type width internally (arrow-vector 15.0.2). This double-scaled allocation for FIXED_LEN_BYTE_ARRAY, BINARY, and INT96 columns. Pass batchSize alone, matching the sibling branches and logical-type paths in the same method. Generated-by: Claude Code
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the same over-allocation described in #11672.
Summary
VectorizedArrowReader.allocateVectorBasedOnTypeName()passedbatchSize * typeWidth(orbatchSize * AVERAGE_VARIABLE_WIDTH_RECORD_SIZEforBINARY) to Arrow'ssetInitialCapacity(int).setInitialCapacity(int valueCount)already multiplies by type width internally (verified against arrow-vector 15.0.2, pinned ingradle/libs.versions.toml), so the extra multiplication double-scaled the allocation.FIXED_LEN_BYTE_ARRAY(fixed(N)/UUID),BINARY,INT96. Largefixed(N)columns can over-allocate per batch, risking OOM (defaultMAX_ALLOCATION_SIZEis unbounded, so it does not always throw).batchSizealone, matching the sibling branches in the same method and the logical-type paths (UUID, decimal) that already allocate correctly withallocateNew(batchSize).allocateVectorBasedOnTypeName(); the same pattern also exists inallocateVectorForEnumJsonBsonString()(STRING/ENUM/JSON/BSON), left for a follow-up.Testing done
TestVectorizedReaderAllocation#fixedLenByteArrayVectorCapacityMatchesBatchSizeand#binaryVectorCapacityMatchesBatchSize, writing a single row through a real Parquet file and asserting the vector'sgetValueCapacity()stays within a small multiple of the batch size.fixed(1000)column withbatchSize=16measured capacity 16775 (bound 64), and aBINARYcolumn measured 247 (bound 64)../gradlew :iceberg-arrow:check— 31 tests passed (0 failures), including spotlessCheck and checkstyle.