Repository navigation
Changelog
60.2.0 - (2026-11-18)
Breaking changes
Enhancements
- fix(parquet): accept empty Thrift lists regardless of declared element type by @bharadwaj-pendyala in #12266
- feat(arrow-array): add GenericByteArray::try_new_repeated (#12143) by @Ayush-CS-89112521 in #12149
- [Variant] Remove
TopLevelRowSinkby @sdf-jkl in #12092 - feat(parquet): allow updating cached file key-value metadata by @manuzhang in #12117
- Add Sbbf::check_hashes and Sbbf::union, make Sbbf::insert_hash public by @adriangb in #11641
- feat(arrow-string): support run-end encoded operands in like kernels by @asubiotto in #12154
- feat(parquet): Improved back-end storage for
PageIndexby @etseidl in #11475 - feat: support casting strings to durations by @miguelcorderopamphile in #11572
- Add
ArrowSbbfto check Arrow arrays against Parquet bloom filters by @bhaskargurram-ai in #11549 - feat(parquet): Add Bloom filter access to
ParquetMetaDataReaderby @etseidl in #12090 - Optimize byte-array dictionary gathering in arrow-row by @JonSnow1807 in #12113
- feat(arrow-cast): Distinguish unsupported casts from value conversion failures by @ZENOTME in #11507
- Perf: Improve dictionary slice concatenation by @Rich-T-kid in #12008
- Use fallible MutableBuffer methods in fallible APIs by @Rich-T-kid in #11912
- feat(json): add opt-in non-finite float parsing to JSON reader by @dentiny in #11725
- parquet: Enforce
PackedDecoderinvariant upon construction by @etseidl in #11981 - Parquet: map the FILE logical type to the arrow.parquet.file extension type by @dejankrak-db in #11877
- feat(parquet): Consolidate bloom filter reads with a push decoder by @etseidl in #11975
- [Variant] Move VariantType into arrow-schema canonical extensions by @sdf-jkl in #12033
- docs: add README to published crates missing one by @EaswarKilari in #12022
- feat(parquet): Add async
Sbbf::read_from_column_chunk_asyncBloom filter API by @etseidl in #11953 - feat(parquet): Make ParquetMetaData more easily clonable by @etseidl in #11297
- [Variant] Move
VariantTypefromparquet-variant-computetoparquet-variantby @sdf-jkl in #12005 - feat(arrow-array): support Date32/Date64 in record_batch!/create_array! macros by @gabotechs in #11962
- fix(arrow-array): make
record_batch!usable without a directarrow_schemadependency by @tz70s in #11734 - fix(arrow-cast): parse ISO 8601 timestamps without seconds by @amitvijapur in #11179
- docs(flight): document Flight SQL schema listing by @Rich-T-kid in #11897
- feat(parquet): expose encryption key metadata by column name by @autoCAT6 in #11733
- Re-export
GetTableTypesBuilder,GetSqlInfoBuilderandGetXdbcTypeInfoBuilderfromarrow_flight::sql::metadataby @morristai in #11846 - fix(arrow-select): include column name in BatchCoalescer errors by @Rich-T-kid in #11870
Bug fixes
- [Variant] Validate storage fields in
VariantTypeby @sdf-jkl in #12245 - fix(parquet-variant-compute): reject unsupported types in empty and all-null cast_to_variant input by @dayan02dev in #12246
- fix(arrow-string): handle dictionaries without values by @Rich-T-kid in #12253
- fix(arrow-ipc): do not write a validity bitmap for RunEndEncoded in V4 by @Mola-maker in #12267
- Fix CSV schema inference for Unicode literals by @peterolkhov in #12144
- fix(parquet): count only shared predicate cache hits in cache-read metrics by @dragonknightk in #12180
- fix(arrow-buffer): return None from NullBufferBuilder when it has no nulls by @Mola-maker in #12251
- fix(arrow-array): keep precision and scale in null_if_overflow_precision by @dayan02dev in #12242
- fix(parquet): Harden Thrift decoding of binary lengths and collection counts by @etseidl in #12142
- fix(variant): preserve nulls for missing shredded list indexes by @sdf-jkl in #12224
- fix(parquet): check PLAIN byte array end offsets for overflow in read by @hsdfat in #12215
- Fix escaping in DataType parsing and list field names by @timmyb32r in #11931
- fix(parquet): honor requested dictionary value type by @Rich-T-kid in #12161
- fix(parquet): return error instead of panic on corrupt PLAIN BYTE_ARRAY pages by @viirya in #12179
- fix(arrow-array): avoid null bitmap allocation in non-nullable StructArray field validation by @llxgdtop in #12124
- fix(arrow-cast): write timestamps whose zone offset has seconds in UTC by @hsdfat in #12102
- fix(parquet): return error on negative DELTA_BYTE_ARRAY lengths in arrow decoder by @hsdfat in #12108
- fix(arrow-arith): match DuckDB's rounding when multiplying an interval by a float by @EaswarKilari in #12029
- fix(ffi): read byte array offsets unaligned in buffer_len by @AndreaBozzo in #12010
- fix(variant): avoid overflow for minimum scale by @Phoenix500526 in #12104
- fix(array): validate decimals with negative scale by @Phoenix500526 in #12101
- fix(arrow-ipc): cap the decompression buffer reserved from the declared length by @geoHeil in #12077
- fix(arrow-ipc): honor skip_validation for record batches in StreamDecoder by @comphead in #12094
- fix(parquet): return error for out-of-range FLBA dictionary index by @linzch3 in #12065
- fix(arrow-json): reject out-of-range float timestamps instead of saturating by @hsdfat in #12007
- fix(arrow-string): saturate the length in Utf8View/BinaryView substring by @hsdfat in #12006
- fix(arrow-array): return an error instead of wrapping on MapBuilder offset overflow by @llxgdtop in #12052
- fix(parquet): cap byte-budget level windows by value count by @adriangb in #11996
- fix(parquet): copy Decimal/Float16 column chunk statistics instead of aliasing the written values by @xudong963 in #12028
- fix(parquet): do not write a dictionary page in empty row group by @jhorstmann in #12003
- Resolve DST-ambiguous and nonexistent local times in
string_to_datetimeby @adriangb in #11054 - Accept buffers with trailing bytes in
ArrayDatavalidation instead of panicking by @vntoinekaio in #11944 - fix(parquet): return byte array overflow and map decoding errors by @hhhizzz in #11791
- fix(parquet): write
specification_version = 1on the VARIANT logical type by @raghuvanshraj in #11961 - fix(parquet): reject truncated definition level runs by @efegokdemir in #11700
- Fix memory reservation leak in Buffer::into_vec by @timmyb32r in #11878
- fix(parquet): skip empty CDC page flushes in add_data_page by @M-Tesla in #11003
- fix(parquet): count levels, not rows, when marking null pages in the column index by @mightsleep in #11828
- fix(arrow-select): merge dictionaries in zip instead of panicking on key overflow by @ump45nose in #11663
- fix(arrow-select): merge dictionaries in merge and merge_n instead of panicking on key overflow by @ninglinLiu in #11925
- fix(ipc): decode zero-length bodies without EOS by @Phoenix500526 in #11967
- fix(parquet): validate Arrow writer shapes and map-key schemas by @HippoBaro in #11838
- fix(arrow-ipc): return an error instead of panicking on out-of-bounds IPC buffers by @wleeaf in #11881
- fix(arrow-avro): resolve named types declared with a full name by @sebxt in #11730
- fix(parquet): Properly account for boolean lists in Parquet Thrift skipping code by @etseidl in #11923
- Fix incorrect reversed
DataType::containscheck for union type by @timmyb32r in #11826 - Reject merging
Fields of unions with different modes by @timmyb32r in #11827 - fix(arrow-array): return overflow error instead of panicking when negating
i32::MINmonths by @choksi2212 in #11611 - fix(arrow-select): preserve nullif nulls for REE and Union by @yongster in #10991
- fix(arrow-avro): keep nulls when reading strings as Utf8View by @ninglinLiu in #11915
- fix(parquet): apply max_row_group_bytes to the first batch of a row group by @Tpt in #11798
- fix(arrow-data): extend Interval(DayTime) as IntervalDayTime in MutableArrayData by @ninglinLiu in #11906
- arrow-select: error on empty input and keep row count for column-less batches in interleave_record_batch by @princejain756 in #11845
- fix(array-data): return an error on FixedSizeList validation overflow by @mvanhorn in #11859
- fix(arrow-ipc): return an error instead of panicking on unions with more than 128 fields by @wleeaf in #11882
- fix(arrow-ipc): reject negative FixedSizeBinary byte width in IPC schema by @wleeaf in #11883
- fix(arrow-array): reset the lone offset of an empty FFI byte array by @ahlag in #11661
Performance improvements
- perf(coalesce): hoist indexed ByteView buffer checks by @Rich-T-kid in #12212
- perf(arrow-arith): inline decimal
divandremby a scalar intotry_unaryby @mbutrovich in #12080 - perf(arrow-arith): divide integers by a scalar with a precomputed divisor by @mbutrovich in #12076
- Use non-null value counts for Parquet dictionary caches by @JonSnow1807 in #12122
- perf(coalesce): directly copy sparse UInt32 selections by @Rich-T-kid in #11174
- perf(parquet): always inline
OffsetBuffer::try_pushby @viirya in #12177 - perf(arrow): fuse Boolean filter copying by @Rich-T-kid in #11918
- perf: Add a dictionary cache for parquet dict types by @EmilyMatt in #11982
- perf(arrow-ipc): reuse lz4 frame encoder buffers across IPC writes by @gabotechs in #12084
- perf(arrow-buffer): pack bools branchlessly in
BooleanBufferBuilder::append_sliceby @viirya in #12024 - perf(arrow): coalesce dictionary keys without concat by @Rich-T-kid in #11743
- perf(arrow-ipc): find dictionary fields without collecting the schema by @asubiotto in #12071
- perf(arrow-select): zip a primitive array with a scalar without MutableArrayData by @mightsleep in #11559
- perf(arrow-row): avoid encoding oversized dictionary values by @Rich-T-kid in #12043
- perf(arrow-buffer): add precomputed divisors for 32-bit and 64-bit integers by @mbutrovich in #11467
- Use
ahashand a single lookup inPrimitiveDictionaryBuilderby @mhk197 in #12057 - perf(parquet): avoid repeated page-index range discovery by @sunchao in #11276
- perf(arrow-buffer): faster count_set_bits on aarch64 by summing byte popcounts before widening by @mbutrovich in #11818
- perf(parquet): share Arrow write-schema plans across row groups by @HippoBaro in #11971
- perf: optimize building out-of-line views in
MutableArrayDataby @neilconway in #11992 - perf(parquet): optimize dictionary computation for byte arrays by @rluvaton in #11524
- perf(arrow): specialize sparse list filtering by @Rich-T-kid in #11744
- perf(parquet): convert FIXED_LEN_BYTE_ARRAY decimals with a width-specialized loop by @yihua in #11910
- perf(coalesce): specialize sparse byte array filters by @Rich-T-kid in #11696
- perf(variant): avoid bracket path string allocations by @DKER2 in #11232
- perf: Optimize appending out-of-line views in
concat,BatchCoalescerby @neilconway in #11770 - perf(parquet-variant): Avoid intermediate string allocations in JSON serialization by @rizzabhdwaj in #11402
- fix(parquet): release bloom filter capacity after folding by @starpact in #11701
- perf(arrow): specialize Boolean filtering by @Rich-T-kid in #11732
- perf(arrow-array): skip
DataTypedrop glue in generic temporal converters by @mbutrovich in #11896 - [Parquet] Create zstd contexts on first use by @azecevic000 in #11858
- feat(buffer): add
NullBufferBuilder::append_wordby @Rich-T-kid in #11867 - perf: Do not compute the entire byte view buffer size in
set_sourceby @nuno-faria in #11830
Documentation updates
- docs(parquet): add SAFETY comments for build_unchecked in array readers by @EaswarKilari in #12119
- docs(arrow-select): document shorter filter predicates by @WaterWhisperer in #12116
- docs: fix NaN ordering in
min/maxrustdoc by @EaswarKilari in #12021 - docs: document RunArray logical offset/len and child array slicing by @alamb in #11710
- docs: make fork CI setup harder to miss by @Rich-T-kid in #11914
Miscellaneous
- Fix: prevent time64us and time64ns from wrapping out-of-bounds seconds to valid times (#11899) by @CodeMasterXZT