arrow 60.1.0

Changelog

60.1.0 - (2026-10-23)

Full Changelog

Security fixes

• Redact Parquet encryption keys from Debug output by @autoCAT6 in #11614

Enhancements

• Implement BatchCoalescer::push_batch_interleaved by @Rich-T-kid in #11519
• feat(parquet): convert INT96 timestamp statistics by @Seanium in #11348
• feat: Introduce ArrayReader and ArrayIterator concepts by @kylebarron in #11161
• feat: Add separate Parquet PageIndex decoder to ParquetMetaDataPushDecoder and ParquetMetaDataReader by @etseidl in #11438
• feat: add GapInfo to timezone by @mhilton in #11535
• Make ByteStreamSplit and Alp Encoder/Decoder and their constructors experimentally public by @jhorstmann in #11481
• Sync Arrow format documentation and Go metadata by @Phoenix500526 in #11499
• Add ArrayData::sliced to avoid allocations when slicing owned ArrayData by @memosmorgan in #11433
• feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning by @adriangb in #10555
• feat(arrow-buffer): add i256::wrapping_div_rem and i256::checked_div_rem by @mbutrovich in #11425
• feat(parquet): Support per-column data page row count limit (#11339) by @DKER2 in #11349
• Allow configuring DELTA_BINARY_PACKED block layout by @emecii in #11051
• feature: add safe version of ArrayData::from_pyarrow by @Rich-T-kid in #10894
• [Arrow] Move parquet-variant-compute StructArrayAssembler to arrow-array by @cakeni in #10636
• feat(parquet): expose RowSelection::force_mask by @haohuaijin in #11337
• fix(parquet): support non-primitive map keys in record reader by @hsdfat in #11229
• feat(arrow-schema): expose FixedShapeTensor logical shape by @yongster in #11249
• feat(arrow-schema): expose FixedShapeTensor shape by @yongster in #11247
• perf: Add APIs for filter with known selected-row count by @neilconway in #11206
• perf(parquet): fuse consecutive filters on the same projection by @haohuaijin in #10859
• feat(variant): allow creating VariantArray from Option items convertible to Variant by @jabrailkhalil in #11053
• feat(parquet): provide direct access to dictionary pages by @DarkWanderer in #10420
• Allow for serialization of Parquet Page Index via custom PageIndexProvider by @etseidl in #11031
• chore(arrow-flight-sql): deprecate register_sql_info from FlightSqlService by @mnpw in #10904
• Infer CSV headers from sampled column types by @Jo2234 in #10361
• Feat: support ndv for View types by @Rich-T-kid in #11163
• feat(ffi): add unsafe constructors and accessors for FFI_ArrowArrayStream by @kylebarron in #11125
• feat(fixed-sized-binary builder): introduce capacity by @Rich-T-kid in #11127
• Add Sbbf::estimated_fpp to expose the bloom filter FPP estimate by @adriangb in #11088
• [Variant] Support list paths in ShreddedSchemaBuilder by @cakeni in #10635
• deps: removed unneeded once_cell dependency if favor of std::sync::LazyLock by @aectaan in #11066

Bug fixes

• fix(arrow-select): reject nulls in non-nullable columns in take_record_batch by @mbutrovich in #11831
• [Variant] Restore object field offsets after failed append by @osipovartem in #11403
• fix(arrow-select): bounds-check take_record_batch once, with a faster check_bounds for null indices by @mbutrovich in #11829
• fix(parquet): align encoded Arrow schemas and nested nullability by @HippoBaro in #11278
• Fix incorrect Decimal statistics for BYTE_ARRAY columns written via ArrowColumnWriter by @DeviousCardi in #11092
• fix(parquet): return an error for RLE run lengths above the spec limit by @hhhizzz in #11517
• fix: use correct length in runarray validation by @Jefffrey in #11668
• fix(parquet): return an error when the footer metadata length exceeds the file in ParquetMetaDataPushDecoder by @ump45nose in #11697
• fix: account for offset when building RunArray from ArrayData by @Jefffrey in #11660
• fix(arrow-data): account for sparse union parent offset in MutableArrayData by @ump45nose in #11693
• Write empty validity buffers for arrays without nulls in IPC writer by @azecevic000 in #11694
• Parquet: list the dictionary page first in page_encoding_stats by @adriangb in #11682
• Fix IPC serialization of unions in sliced lists by @skyon0522-ai in #11651
• fix: assemble split IPC messages in StreamDecoder at their declared size by @ranflarion in #11648
• fix(arrow-cast): truncate Timestamp to Date64 casts to whole days by @ump45nose in #11632
• fix: Fix zip with a short falsy scalar that still has buffers by @mkleen in #11625
• Respect dictionary value nulls in aggregates by @skyon0522-ai in #11650
• Reject mismatched input schemas in FlightDataEncoder by @RanaPriyansh in #11293
• Fix sliced sparse union ArrayData conversion by @skyon0522-ai in #11649
• fix: handle zero-length null extension for empty unions by @efegokdemir in #11209
• fix(arrow-data): check view nulls at the right index in ArrayData equality by @ump45nose in #11642
• fix(arrow-array): use the field's type id in dense union logical_nulls by @ahlag in #11643
• fix(arrow-schema): validate UnionFields when deserializing by @ump45nose in #11638
• fix(arrow-row): map type ids to field indexes when decoding dense union offsets by @ahlag in #11644
• fix(arrow-schema): keep map keys-sorted flag in FFI field export by @spicysharkco in #11630
• Fix Fields::filter_leaves for ListView and LargeListView by @patrickswedish in #11657
• Return errors for undersized struct validity buffers in IPC reader by @azecevic000 in #11658
• fix(parquet-variant-json): preserve whole float formatting by @cetra3 in #11513
• fix(arrow-data): account for FixedSizeList parent offset in MutableArrayData by @ahlag in #11636
• fix(csv): return an error instead of panicking on dictionary key overflow by @spicysharkco in #11628
• fix(arrow-cast): bump atoi to 3.1.1 to fix Int16 parse overflow by @mbutrovich in #11626
• fix(arrow-select): keep row count in take for zero-width FixedSizeBinary by @ahlag in #11637
• fix(variant): reject duplicate object field names in shred_variant by @Abhisheklearn12 in #11351
• fix: move float-to-decimal precision validation inside the cast kernel by @sdf-jkl in #11302
• fix(parquet): Fix physical type for precision-one decimals by @CTTY in #11544
• fix: UnionFields::from_iter should validate input by @Jefffrey in #11533
• fix: account for offsets in runarrays when using extend for MutableArrayData by @Jefffrey in #11546
• fix: implement ArrayData::validate_values for union types by @Jefffrey in #11534
• fix: account for arraydata offset when comparing runarrays by @Jefffrey in #11536
• fix(parquet): encode FixedSizeBinary dictionaries as FLBA by @HippoBaro in #11262
• fix(parquet): return an error, not a panic, when an offset index page goes past its column chunk by @adriangb in #11484
• fix(parquet): release the bytes of a row group when the push decoder is done with it by @adriangb in #11235
• fix(ffi): report missing C stream callbacks by @dhruv-dagar in #11422
• fix(parquet): bound row-group count by remaining input before reserving by @pmoust in #11419
• fix(arrow-arith): keep decimal division precision valid by @snoopuppy582 in #11420
• fix(arrow-cast): round decimal to float conversions once by @davidlghellin in #10509
• fix(arrow-buffer): round i256 to f64 correctly at f64 midpoints by @miguelcorderopamphile in #11315
• fix(variant): avoid panic on variant_get for non-nullable field by @neilconway in #11356
• Fix filter_nulls for predicates selecting all or no rows by @viirya in #11378
• fix(arrow-row): return an error instead of wrapping view offsets past 4 GiB when decoding BinaryView/Utf8View by @kita-renji in #11224
• fix(arrow-data): account for parent offset in FixedSizeList null validation by @viirya in #11362
• Fix IPC roundtrip for empty RunArray slices by @viirya in #11361
• fix: Only require valid UTF-8 within the visible span of string arrays by @neilconway in #11323
• fix(parquet): honor explicit V1 FLBA dictionary opt-in by @subotac in #10569
• fix(arrow-cast): only cast visible elements for lists and maps by @neilconway in #11359
• fix(arrow-select): fix take output capacity for sliced lists by @neilconway in #11360
• fix(arrow-cast): avoid panicking during failed List -> ListView cast by @neilconway in #11355
• fix(arrow-string): sum view element lengths in usize when concatenating by @Cintu07 in #11318
• fix(arrow-data): fix gaps in ListView equality by @neilconway in #11345
• fix(variant): support closing brackets inside quoted path fields by @DKER2 in #11231
• fix(arrow-cast): allow whitespace around temporal values (#11225) by @wilhelmagren in #11299
• fix(arrow-array): avoid returning corrupt array in GenericByteArray::into_builder by @neilconway in #11311
• fix(arrow-array): PrimitiveRunBuilder was buggy after finish by @neilconway in #11326
• fix(arrow-array): preserve type params in PrimitiveArray::into_builder by @neilconway in #11321
• fix: finish methods in PrimitiveDictionaryBuilder discard type params by @neilconway in #11327
• fix: Avoid dropping type parameters in PrimitiveArray::into_builder by @neilconway in #11310
• fix(parquet): don't preallocate 1MiB in DeltaBitPackEncoder by @cetra3 in #11149
• Avoid overflow in decimal precision and scale arithmetic by @snoopuppy582 in #10996
• Resolve DST-ambiguous and nonexistent local times when casting timestamps to a named timezone by @adriangb in #11038
• Fix sums of sliced run-end encoded arrays by @Boulea7 in #11283
• fix(arrow-data): new_null on an empty union by @Cintu07 in #11258
• fix(parquet): add early rejection of unsupported encodings by @HippoBaro in #11266
• fix(parquet): preserve distinct-count invalidation across page flushes by @HippoBaro in #11264
• Fix unshredded Variant field extraction benchmark by @sdf-jkl in #11253
• fix: Parquet ByteArrayDecoderPlain::read can silently fail to read the correct number of values by @etseidl in #11222
• fix: error on invalid type representation when parsing REE/Map types by @Rich-T-kid in #11160
• parquet: Fix page index loading with suffix prefetch by @etseidl in #11208
• fix(arrow-array): check logical nulls, not is_nullable, for a non-nullable list field by @amitvijapur in #11178
• fix(arrow-select): preserve nullability for REE and Union take by @yongster in #10994
• fix: ndv calculation for dictionary arrays by @Rich-T-kid in #11173
• Fix async/push decoder read when only some columns have an offset index by @adriangb in #11182
• Limit LZ4 buffer to declared decompressed size by @azecevic000 in #10959
• fix(arrow-array): deprecate FixedSizeListArray::value_offset, which wraps past i32::MAX by @Cintu07 in #11070
• fix(variant): reject FixedSizeList shredding by @cakeni in #10639
• fix(arrow-cast): check remaining temporal cast overflows by @yinli-systems in #10907
• fix(parquet): keep virtual columns in the schema reported with a schema hint by @bharadwaj-pendyala in #11047
• fix(arrow-string): saturate the offset casts in substring by @Cintu07 in #10995
• fix(arrow-cast): respect sliced list offsets in fixed-size list casts by @CodingCossack in #11065
• Return errors for undersized ArrayData validity buffers by @emecii in #11044
• fix(csv): Infer leading-plus integers and decimals by @Jo2234 in #11105
• fix(arrow-ord): return an error for nested dictionary comparisons by @jaideeppyne in #11104
• fix(arrow-array): reset NullBuilder length in finish by @ranflarion in #11110
• fix: preserve zero-width FixedSizeBinary length when filtering by @Rich-T-kid in #11042
• fix(arrow-cast): preserve length when casting zero-width FixedSizeListArray by @Rich-T-kid in #11043
• fix: remove stale Interval -> Int64 arm from can_cast_types by @dylanpulver in #10939
• fix(parquet): do not cache an ALP preset built from an empty first page by @Abdallah-Afifi in #11004

Performance improvements

• perf(arrow-buffer): divide i256 by a single word with a reciprocal on 64-bit targets other than x86_64 by @mbutrovich in #11580
• perf(parquet): skip per-batch key-bounds scan when RLE bit width prov… by @Rich-T-kid in #11462
• perf(arrow-cmp): add fast paths to dictionary_cmp by @Rich-T-kid in #11407
• speed up dict cast key width by @Rich-T-kid in #11502
• perf: Inline SlicesIterator::next by @haohuaijin in #11646
• perf(parquet): trim bitmap-backed row selections using 64-bit words by @haohuaijin in #11639
• Perf(parquet): RLE_dictionary reads; avoid round trip allocation by @Rich-T-kid in #11474
• perf(parquet): optimize i32, u32, i64, u64, f32, f64 min_max calculation by @rluvaton in #11523
• refactor(parquet): do not store empty pushed ranges in PushBuffers by @adriangb in #11569
• Defer dictionary page decode on the skip path by @zhuqi-lucas in #11168
• perf(parquet): make PushBuffers release and clear cost independent of the buffer count by @adriangb in #11459
• perf(parquet): scan bitmap selections directly for page ranges by @haohuaijin in #11553
• perf(arrow-buffer): divide i256 magnitudes below 2^128 with one u128 division by @mbutrovich in #11530
• Speed up DELTA_BINARY_PACKED skip by @flashmouse in #11143
• perf(arrow-array): move the closure into the iterator in PrimitiveArray::unary by @mbutrovich in #11525
• feat(arrow-array): add div_rem_wrapping and div_rem_checked to ArrowNativeTypeOp by @mbutrovich in #11528
• perf(arrow-arith): divide once per row in multiply_fixed_point by @mbutrovich in #11526
• perf(arrow-buffer): faster portable fallback for bit_util::compress by @mightsleep in #11322
• perf(arrow-ipc): Avoid copies and write dictionary batches directly to writers when possible by @JakeDern in #10128
• perf(parquet): scan MaskCursor selections in 64-bit words by @haohuaijin in #11510
• perf(arrow-arith): don't build an error for each decimal div value that overflows when scaled by @mbutrovich in #11479
• perf(parquet): optimize level counting by @neilconway in #11408
• feat: specialize boolean interleave by @dhruvtnwr28 in #11501
• perf(parquet): store a ScanPlan cursor as one heap entry by @adriangb in #11334
• perf: Avoid expanding masks to selectors in mixed RowSelection intersection and union by @haohuaijin in #11390
• perf(parquet): keep PushBuffers sorted by offset by @adriangb in #11237
• perf(arrow-select): concat FixedSizeList without per-array ArrayData by @mightsleep in #11435
• perf(parquet): Skip per-level validity pass for list children with no nulls by @neilconway in #11376
• feat: Allow for partial population of the Parquet Page Index by @etseidl in #11157
• chore: use extend_from_slice rather than extend by @rluvaton in #11410
• chore: reuse allocations in partition by @rluvaton in #11324
• perf: Optimize int -> decimal cast by @neilconway in #11226
• perf: Optimize date_part for timestamp values without timezones by @neilconway in #11187
• perf: Optimize casting timestamp -> Date32, Time32, Time64 by @neilconway in #11199
• perf: Optimize boolean -> string casts by @neilconway in #11251
• perf: override count, nth, nth_back and last for RowsIter by @rluvaton in #11230
• Reuse buffers when interleaving byte views by @cetra3 in #11281
• perf: Reuse capture locations in regexp_match by @neilconway in #11274
• perf: Optimize infallible numeric casts by @neilconway in #11219
• perf: Reserve concat output space using visible offset span by @neilconway in #11270
• perf: Optimize bool <-> numeric casts by @neilconway in #11242
• perf: Optimize display of Boolean values by @neilconway in #11254
• perf(arrow-select): add take_record_batch_unchecked to skip redundant bounds checks by @Rich-T-kid in #10945
• [arrow-select] perf: Replace ArrayData with direct Array construction in concat by @liamzwbao in #10377
• Make CachedArrayReader cache cleanup incremental instead of quadratic by @zhuqi-lucas in #10901
• perf: Optimize RowSelection::and_then for dense masks by @haohuaijin in #10968
• feat(perf): Improve filter performance with per word bit filtering (and BMI when supported) by @devanbenz in #10136
• feat: coalesce fixed size binary by @Rich-T-kid in #11122
• [Parquet] Improve Auto RowSelection construction for scattered predicate results by @hhhizzz in #10852
• [Parquet] Vectorise Pack Bits within BitWriter by @sdf-jkl in #10432
• perf(arrow-data): Format decimal values without intermediate allocs by @neilconway in #11002
• Replace OffsetBufferBuilder with Vec in scalar byte-array zip by @yashkuceriya in #10913

Documentation updates

• Fix BatchCoalescer::is_empty documentation by @Rich-T-kid in #11790
• docs: Add example to BitChunks::chunk by @alamb in #11214
• Document how to customize JSON decoding via DecoderFactory by @alamb in #10897
• doc: Describe potential footgun in MapArray::keys and values by @neilconway in #11145
• MINOR: add complete lexsort_to_indices usage example by @wilhelmagren in #11133
• docs(avro): add OpenDAL read and write example by @Xuanwo in #11080
• Document platform endianness support by @sdf-jkl in #11056
• chore: Fix duplicated word typos in comments and docs by @geographybuff in #11034
• docs(parquet): add OpenDAL read and write example by @Xuanwo in #11014