Arrow: Improve initial capacity estimation for variable-width vectors

[huan233usc]

Apache Iceberg version

main (development)

Query engine

Spark

Please describe the improvement

VectorizedArrowReader initializes variable-width vectors with:

vector.setInitialCapacity(batchSize * AVERAGE_VARIABLE_WIDTH_RECORD_SIZE);

The single-argument Arrow API accepts a value count, not a byte count. For a batch size of 5,000 and an average-width constant of 10, this reserves offsets for 50,000 values and uses Arrow's default density of 8 bytes/value, rather than reserving 5,000 values at an estimated 10 bytes/value.

The reader already receives ColumnChunkMetaData in setRowGroupInfo, including total compressed/uncompressed sizes and value count. We should investigate using that metadata to estimate a bounded average width and initialize BaseVariableWidthVector with the density-aware API:

setInitialCapacity(batchSize, estimatedBytesPerValue)

The estimate should:

• use batchSize as the value capacity;
• fall back to a conservative default when metadata is missing or unusable;
• account for page/dictionary overhead and avoid unbounded over-allocation;
• cover physical BINARY and logical variable-width types such as string, JSON, BSON, geometry, and geography;
• preserve safe reallocation for unusually large values.

Related: #11672 identified the value-count/byte-count mismatch but was closed automatically as stale without a fix.

Willingness to contribute

• I can contribute a fix for this improvement independently
• I would be willing to contribute a fix with guidance from the Iceberg community
• I cannot contribute a fix at this time