v12.0.0

What's Changed

Breaking Changes 🛠

• feat(io)!: paginated directory listing on ObjectStore by @wjones127 in #8606
• perf(mem_wal)!: serve the shard manifest from the store that wrote it by @hamersaw in #8640
• feat(table): predecessor-conditioned publication for external manifest stores by @wkalt in #8800
• feat(namespace)!: accept multiple columns for the merge insert on key by @wjones127 in #8915
• feat: make data file parts serializable by @Xuanwo in #9072
• fix(index)!: do not route json_extract predicates to JSON indices by @westonpace in #9101

New Features 🎉

• feat(java): support segment-based distributed vector search by @summaryzb in #7169
• feat(index): share IVF partition scans across batch vector queries by @sezruby in #7640
• feat(mem_wal): make write backpressure injectable and account index memory by @hamersaw in #7831
• feat(python): add Bitmap binding for RoaringBitmap by @wjones127 in #7837
• feat(python): add LanceDataset.slice() method by @Sapnilb15 in #8059
• fix(mem-wal): resolve writer shard spec identity by @u70b3 in #8112
• feat(python): expose more scanner options on LanceFragment by @keunhong in #8429
• perf(java): reduce fragment statistics allocations by @everySympathy in #8452
• feat(python): expose ObjectStoreProvider registration in pylance by @farazshaikh in #8522
• feat: add mixed data file version capability by @Xuanwo in #8580
• feat: validate mixed V2 data file versions by @Xuanwo in #8581
• feat: support per-operation V2 write targets by @Xuanwo in #8582
• feat: propagate exact V2 versions across dataset operations by @Xuanwo in #8583
• feat: target exact V2 versions in compaction by @Xuanwo in #8584
• feat(cleanup): support cleaning specific dataset versions by @hfutatzhanghb in #8617
• feat: resolve stable file format to 2.2 by @Xuanwo in #8657
• feat: add column slice stitching by @Xuanwo in #8660
• feat: support BinaryView in packed blob writer via iterator-based extraction by @lichuang in #8700
• feat(python): restore index retraining by @ddupg in #8786
• feat(encoding): add CANDIDATE_BATCH_SIZES and plan_decoded_bytes trait defaults by @westonpace in #8791
• feat(java): expose createIndex progress callbacks by @hfutatzhanghb in #8823
• feat(java): align inverted index options by @ddupg in #8829
• feat(mem_wal): let a writer read its own indexed prefix by @hamersaw in #8835
• fix(mem_wal): honor distance_range in the LSM vector planner by @hamersaw in #8845
• feat(python): return matched row offsets from fragment update_columns by @zhangyue19921010 in #8891
• feat(java): expose merge insert write mode by @zhangyue19921010 in #8918
• feat: budget blob materialization readahead by @jackye1995 in #8919
• feat: add caller-managed data file parts by @Xuanwo in #8923
• feat(java): complete transaction operation mappings by @everySympathy in #8925
• feat(index): default ivf_rq to 5-bit quantization by @BubbleCal in #8936
• feat(java): expose referenced file metadata by @everySympathy in #8962
• feat(mem_wal): record what a flushed SSTable holds by @xuanyu-z in #8981
• feat(index): proportional hierarchical k-means for IVF training by @BubbleCal in #9050
• feat(index): split IVF partitions to target size and join undersized ones in one optimize pass by @BubbleCal in #9051
• feat: expose public accessor for the fragment reuse index by @brendanclement in #9112
• feat(dataset): expose lance dataset deep clone to python by @dentiny in #9181
• feat(search): select metric-aware initial IVF probe budgets by @BubbleCal in #9195

Bug Fixes 🐛

• fix: make float filters treat -0.0 and 0.0 as the same value by @LuciferYang in #6236
• fix(arrow): avoid duplicate field when merging identical List columns by @LuciferYang in #7494
• fix: don't fail dataset open on an undecodable inline transaction by @jackye1995 in #7740
• fix(core): reject non-finite values in AimdConfig::validate by @LuciferYang in #7942
• fix(index): replace scalar cache entries on store rotation by @u70b3 in #7962
• fix: validate primary key nullability on every path that can change it by @xuanyu-z in #8101
• fix(index): filter stale vector segment rows before top-k by @app/lance-gatefixer in #8351
• fix: cast nested columns during schema evolution by @app/lance-gatefixer in #8379
• fix(encoding): read wrapped miniblock level counts by @app/lance-gatefixer in #8410
• fix(index): keep an index this build cannot read out of the erase path by @wombatu-kun in #8427
• fix(index): keep a commit alive when an index cannot be opened by @wombatu-kun in #8441
• fix(python): declare LanceSchema.field and field_case_insensitive in the type stub by @jonasdedden in #8486
• fix(python): correct the misspelled materialize_deletions_threshold option by @jonasdedden in #8487
• fix(python): declare CompactionOptions with total=False by @jonasdedden in #8488
• fix(python): accept None for max_rows_per_group by @jonasdedden in #8489
• fix(python): let create_index_uncommitted accept an IndexConfig by @jonasdedden in #8490
• fix(python): declare the index training helpers as returning FixedSizeListArray by @jonasdedden in #8491
• fix(python): annotate LanceFragment.data_files and deletion_file by @jonasdedden in #8492
• fix(python): annotate count_rows, versions and optimize_indices by @jonasdedden in #8493
• fix(python): annotate BatchUDF and the batch_udf decorator by @jonasdedden in #8494
• fix: reject lossy null structs in v2.0 by @app/lance-gatefixer in #8507
• fix: support Time32 integer casts in filters by @app/lance-gatefixer in #8508
• fix(compaction): reject unsafe binary column layouts by @app/lance-gatefixer in #8510
• fix: keep LabelList index current after reordered merge insert by @app/lance-gatefixer in #8511
• fix(index): remap ZoneMap search results by @app/lance-gatefixer in #8512
• fix(compaction): avoid stranded remainder fragments by @app/lance-gatefixer in #8513
• fix(index): skip unrecognized index types by @app/lance-gatefixer in #8529
• fix(linalg): pick the sum-of-squares accumulator per element type by @LuciferYang in #8533
• fix: defer blank rows when a fully deleted batch has no row to copy by @LuciferYang in #8612
• fix(sql): include system columns when requested for SQL-based query by @dentiny in #8626
• fix(linalg): enforce the hamming_batch_u64 length contract in release by @LuciferYang in #8639
• fix(linalg): report AVX-512 test coverage by @app/lance-gatefixer in #8664
• fix: refuse to drop a path that is not a Lance dataset by @xiaguanglei in #8665
• fix: preserve typed DataFusion errors through Shared and ReplayExec by @app/copilot-swe-agent in #8677
• fix(linalg): correct multivector distance aggregation by @ddupg in #8710
• fix(io): prevent io_uring requests from hanging by @Xuanwo in #8725
• fix(io): reject unordered scheduler ranges by @app/lance-gatefixer in #8728
• fix(linalg): enforce cosine u8 length contract by @app/lance-gatefixer in #8737
• fix(fts): preserve global scoring for no-impact indexes by @BubbleCal in #8748
• fix(fts): canonicalize fuzzy expansion across segments by @BubbleCal in #8758
• fix(python): spell misnamed without a hyphen so the typos gate passes by @jackylee-ch in #8766
• fix(index): stream bulk index file copies by @jackye1995 in #8770
• fix(merge_insert): make analyze_plan follow execute's source routing by @LuciferYang in #8772
• fix(datagen): align random primitive buffers by @app/lance-gatefixer in #8776
• fix(fts): use substring-safe ngram defaults by @app/lance-gatefixer in #8779
• fix(python): don't seed IVF centroids with NaN vectors by @jonasdedden in #8782
• fix(ci): stabilize test_scan_slice against page-cache state by @RupertMaiti2005 in #8815
• fix(encoding): normalize out-of-range null dictionary keys by @app/lance-gatefixer in #8827
• fix(substrait): decode deprecated timestamp literals as microseconds by @jonasdedden in #8832
• fix(hnsw): make a persisted HNSW cover its rows and reject invalid ones by @xuanyu-z in #8834
• fix(mem_wal): seal the memtable on max_memtable_rows by @hamersaw in #8837
• fix(fs): fix dataset files permission on local filesystem by @dentiny in #8839
• fix(encoding): handle empty all-null dictionaries by @app/lance-gatefixer in #8841
• fix(index): widen the RaBitQ FastScan accumulator above 1024 dims by @wombatu-kun in #8842
• fix(index): declare FM index row address results by @app/lance-gatefixer in #8855
• fix(linalg): widen u8 distance accumulators by @app/lance-gatefixer in #8888
• fix: resolve clone sources with source commit handler by @Xuanwo in #8902
• fix(index): key metadata cache on manifest etag by @ChilePiquin in #8904
• fix(linalg): distinguish arrow batch type errors by @app/lance-gatefixer in #8911
• fix(fts): align v1 merge reads with writer batches by @BubbleCal in #8922
• fix: preserve complete blob v2 logical schemas by @Xuanwo in #8929
• fix(io): carry batch_size_bytes through distributed FilteredReadOptions by @lichuang in #8933
• fix(encoding): accept 1-bit booleans in the single-row full-zip fallback by @zhangyue19921010 in #8934
• fix: avoid panic when rebasing a commit whose read version was cleaned up by @qiuyuhang in #8935
• fix(io): reject non-lowercase GooseFS storage_options keys by @XuQianJin-Stars in #8940
• fix(io): parse GooseFS chunk/block size suffixes like 4MB by @XuQianJin-Stars in #8943
• fix: remove namespace-managed clone handler plumbing by @Xuanwo in #8964
• fix(sql): classify malformed SQL queries as invalid input by @wjones127 in #8974
• fix: let analyze_plan run under a caller-provided TaskContext by @LuQQiu in #8980
• fix(index): prevent maintenance from undoing index drops by @app/lance-gatefixer in #8984
• fix(java): remove duplicate IntoJava impl for &BasePath by @LuciferYang in #8990
• fix: count only visible nulls in the 2.1+ writer by @jackye1995 in #9020
• fix(mem_wal): run grace eviction on its own task, not the flush handler's ticker by @xuanyu-z in #9022
• fix(cache): bypass Moka single-flight for zero-capacity caches by @jiaqizho in #9038
• fix: copy sliced arrays from logical offsets by @jackye1995 in #9048
• fix(index): keep every HNSW node reachable after a parallel build by @BubbleCal in #9053
• fix(fts): preserve prewarmed state across request store wrappers by @BubbleCal in #9075
• fix: stamp overlay row lineage at commit by @app/lance-gatefixer in #9084
• fix: honor zero scanner limits by @app/lance-gatefixer in #9111
• fix(encoding): prevent bits_per_value=0 in FullZipLayout for FSL with AllNull child by @westonpace in #9130
• fix: load external fragment reuse index details from the correct base path by @LuQQiu in #9141
• fix: skip size probes for bounded blob reads by @Xuanwo in #9146
• fix(python): preserve filter pushdown with row IDs by @app/lance-gatefixer in #9154
• fix(dataset): handle lance dataset blobs on deep clone by @dentiny in #9185
• fix(scanner): respect negated flag when converting InList filters to take operations by @dentiny in #9188

Documentation 📚

• docs: classify the fragment's non-Lance referenced files by format by @LuciferYang in #8562
• docs: correct stable row ID migration and storage by @app/lance-gatefixer in #8852
• docs(linalg): document SIMD conversion panics by @app/lance-gatefixer in #8862
• docs(core): correct AVX tier feature semantics by @app/lance-gatefixer in #8910
• docs: add Apache Doris integration by @Gabriel39 in #8983

Performance Improvements 🚀

• perf(rowids): count bitmap bits a word at a time by @pengw0048 in #8642
• perf: stop listing the entire _versions directory for limited version queries by @LuQQiu in #8679
• perf: run-optimize roaring bitmaps before persisting by @zhangyue19921010 in #8688
• perf(rowids): reuse cursors across sequential batches by @jiaoew1991 in #8713
• perf(python): stream reader batches from one producer by @jiaoew1991 in #8714
• perf(rowids): accelerate dense bitmap decoding by @jiaoew1991 in #8715
• perf(rowids): stream dataset version runs with a cursor by @jiaoew1991 in #8716
• perf(merge_insert): probe the most selective indexed key first and stop early by @LuciferYang in #8719
• perf(commit): drop O(C^2) field lookups from column index validation by @zhangyue19921010 in #8720
• perf(scan): reduce system column pipeline overhead by @jiaoew1991 in #8747
• perf(fts): defer phrase confirmation with score bounds by @BubbleCal in #8749
• perf(fts): share MultiMatch prefilter materialization by @BubbleCal in #8750
• perf(fts): complete ambiguous WAND ties without full replay by @BubbleCal in #8751
• perf(build): revert default target back to haswell (avx2) by @westonpace in #8754
• perf(commit): skip overlap detection for single-segment indices by @zhangyue19921010 in #8778
• perf(fts): keep compound scorer in fast search by @BubbleCal in #8817
• perf(fts): index residual compound rows once by @BubbleCal in #8818
• perf(fts): remove benchmark metrics from query hot paths by @BubbleCal in #8825
• perf(rowids): probe the fragments instead of merging every row id by @dshepelev15 in #8883
• perf(index): open fragment reuse indexes 10-2200x faster by @Xuanwo in #8887
• perf(index): prevent two-file shuffle starvation by @BubbleCal in #8894
• perf(io): read large metadata msgs as concurrent chunked range requests by @dshepelev15 in #8914
• perf(index): accelerate RQ scale search by @BubbleCal in #8924
• perf(rowids): keep the row id index cached and rank bitmaps for lookups by @dshepelev15 in #8930
• perf(cleanup): avoid reading manifests for version retention by @ziwenzhang in #9019
• perf(fts): accelerate wide and queries with bulk intersection by @BubbleCal in #9030
• perf(fts): defer decoding for underfilled conjunctions by @BubbleCal in #9031
• perf(fts): reuse resident statistics and prepared scorers by @BubbleCal in #9032
• perf(fts): bound classic conjunction scoring in dense windows by @BubbleCal in #9033
• perf(index): read IVF prewarm in parallel byte windows by @BubbleCal in #9049
• perf(fts): skip phrase position decode when the BM25 score cannot ent… by @mikewhb in #9153
• perf: check table deregistration status concurrently in list_directory_tables by @ChilePiquin in #9164
• perf: push table listing pagination down to the object store by @ChilePiquin in #9165
• perf: disable manifest indices by default by @jackye1995 in #9180

Other Changes

• refactor: merge JsonConvertingReader into SchemaAdapter by @xloya in #7678
• refactor(encoding): align block codecs with miniblock by @Xuanwo in #8324
• refactor: replace hand-rolled string→bool parsing with str_is_truthy/str_to_bool by @app/copilot-swe-agent in #8675
• refactor: express feature flags as bit shifts by @Xuanwo in #8893
• revert: add column slice stitching by @Xuanwo in #8926

Full Changelog: release-root/12.0.0-beta.N...v12.0.0