v14.0.0-rc.2

What's Changed

Breaking Changes 🛠

• feat(encoding): compress v2.3 mini-block offsets by @Xuanwo in #8331
• feat(index): change btree scalar index to row-address domain by @westonpace in #9481
• refactor(index): pass index_version into ScalarIndexPlugin::load_index by @westonpace in #9518
• feat(mem_wal): evaluate fuzzy conjunctions on the active memtable by @hamersaw in #9526
• refactor!: vendor jsonb into the lance-jsonb crate by @Xuanwo in #9534
• perf(mem_wal): optimize Blob v2 writes and promotion by @jackye1995 in #9717
• feat(cleanup): add configurable delete concurrency by @majin1102 in #9774
• perf(linalg)!: select f32 l2/dot kernels once and fill batches via _into APIs by @wjones127 in #10086

Critical Fixes ‼️

• fix(encoding): use each layer's dimension when flattening nested FSL by @LuciferYang in #9760
• fix(scan): preserve candidates from inexact scalar index results by @app/lance-gatefixer in #9789
• fix: include overlay-stale rows in indexed merge insert by @app/lance-gatefixer in #9865
• fix(encoding): read RLE-compressed mini-block dictionary values by @lichuang in #10145

New Features 🎉

• feat(encoding): add range and delta block codecs by @Xuanwo in #8325
• feat(encoding): add bounded block dictionary by @Xuanwo in #8326
• feat(encoding): compose generic block codecs by @Xuanwo in #8327
• feat(encoding): select compressed offset blocks by @Xuanwo in #8330
• feat(index): support Hamming distance in streaming coreset k-means by @a-erofeev in #8610
• feat: add non-reusable field IDs by @Xuanwo in #8658
• feat(index): f64 centroids support in streaming coreset k-means by @a-erofeev in #8662
• feat(java): scan fragment physical row slices by @majin1102 in #8951
• feat(namespace): return inserted row count and version from insert_into_table by @XuQianJin-Stars in #9229
• feat(blob): reuse independently addressed managed objects by @Xuanwo in #9293
• feat(format): add target_data_type to JsonIndexDetails by @wjones127 in #9303
• feat: support SQL IS [NOT] DISTINCT FROM as null-safe comparison by @jonasdedden in #9331
• feat(mem_wal): maintain every index by intent, replaceable on a live writer by @xuanyu-z in #9335
• feat(datafusion): estimate total_byte_size for Lance scans by @vivek-bharathan in #9430
• feat: allow FRI to be used with stable row IDs by @westonpace in #9514
• feat(format): define Arrow-ordered zone write seeds by @LuQQiu in #9564
• feat(python): expose the fragment ids of a dataset by @zhangyue19921010 in #9592
• feat(python): expose the checked out version by @zhangyue19921010 in #9594
• feat: record row-address moves for deferred compactions on stable-row-ID tables by @brendanclement in #9611
• feat(java): count indexed rows on explicit scalar segments by @hfutatzhanghb in #9644
• feat(java): open a dataset at a branch or tag by @zy-kkk in #9703
• feat(mem_wal): move a live writer onto a schema change instead of reopening it by @xuanyu-z in #9712
• feat(count): look through row-preserving custom nodes in CountPushdown by @wkalt in #9716
• feat(scanner): make Scanner::project_with_schema public by @amunra in #9735
• feat(exec): late-materialize non-sort columns under a top-k by @wkalt in #9750
• feat(search): add search effort and auto probing through k=100000 by @BubbleCal in #9798
• feat(dataset): support stable spilled row lineage by @BubbleCal in #9802
• feat(mem_wal): make memtable index kinds pluggable by @xuanyu-z in #9803
• feat(mem_wal): answer memtable filters with Lance's scalar expression pass by @xuanyu-z in #9804
• feat(mem_wal): pass writer settings and flush build parameters to memtable index plugins by @xuanyu-z in #9805
• feat(cache): add tier probes and selective entry reads by @BubbleCal in #9829
• feat(io): make file-read coalescing configurable by @BubbleCal in #9833
• feat(fts): allow external indexed input for hybrid queries by @BubbleCal in #9847
• feat(index): distributed execution of optimize indices by @zhangyue19921010 in #9871
• feat(format): define provider-managed clustering metadata by @LuQQiu in #9878
• feat(blob): expose managed sharing in Python preparation API by @Xuanwo in #9880
• feat: synchronous FilteredReadExec encoding given a resolved table identifier by @LuQQiu in #9919
• feat(format): define independent planar metadata for v2.3 by @Xuanwo in #9920
• feat(encoding): implement planar page execution for Lance 2.3 by @Xuanwo in #9921
• feat(encoding): add indexed values to sparse planar pages by @Xuanwo in #9925
• feat(encoding): cap the dictionary size a page may be encoded with by @dshepelev15 in #9939
• feat(java): expose MinHash LSH indices to Java by @zhangyue19921010 in #9965
• feat(python): expose the MinHash LSH index to python by @zhangyue19921010 in #9966
• feat(index): add a PCA model and vector pre-transform by @beautygua in #9967
• feat(dataset): spill carried lineage in merge insert by @BubbleCal in #10062
• feat(blob): support managed data file parts and descriptor reuse by @Xuanwo in #10176
• feat(index): allow training IVF from supplied samples by @BubbleCal in #10188
• feat(namespace): add the spec operations LanceNamespace lacked by @twmb in #10192

Bug Fixes 🐛

• fix: assign fragment and row ids to new fragments in Merge commits by @wkalt in #7706
• fix(encoding): validate block bitpacking payloads by @Xuanwo in #8328
• fix(encoding): bound general block decompression by @Xuanwo in #8329
• fix: choose cheapest blanks when restoring deleted rows by @chenyu-x in #8946
• fix(file): verify 2.1+ nullability against name-selected columns by @jackylee-ch in #9247
• fix: prune overlays for dropped columns by @app/lance-gatefixer in #9314
• fix(datafusion): preserve boolean expression literal types by @app/lance-gatefixer in #9322
• fix(python): say that create_index ignores index_cache_size by @LuciferYang in #9384
• fix(index): store the converted metric in IvfTransformer::new_flat by @LuciferYang in #9399
• fix(index): report shuffle-stage failures instead of panicking by @LuciferYang in #9405
• fix(index): reject the hamming metric for IVF_RQ before training by @LuciferYang in #9423
• fix(blob): deduplicate managed reference scans during cleanup by @Xuanwo in #9435
• fix: avoid panic for missing duplicate field id by @zhangyue19921010 in #9448
• fix: move inline transaction to a new manifest field by @wjones127 in #9482
• fix(index): carry merged segment coverage through compaction only while indexed data matches by @xuanyu-z in #9524
• fix: write nullable structs that contain a Null-typed field by @breken-ai in #9579
• fix: read back FixedSizeList columns of the Null type by @breken-ai in #9580
• fix(namespace): handle legacy auto-cleanup policy on directory manifest by @Xuanwo in #9600
• fix: preserve fragment reuse mappings across concurrent maintenance by @brendanclement in #9620
• fix(index): bound bitmap cache state lookup batches by @Xuanwo in #9622
• fix: start the commit retry budget after catching up by @Xuanwo in #9631
• fix(io): count HEAD requests without object body bytes by @geruh in #9648
• fix(scan): avoid exact counts when retaining deleted rows by @geruh in #9652
• fix(java): include baseId in DataFile/DeletionFile equals and add hashCode by @Zhuoxi2000 in #9686
• fix(mem_wal): rank a dot-product memtable index by distance, not by product by @xuanyu-z in #9689
• fix: prevent incomplete unordered listings after retry errors by @kamronis in #9691
• fix(datafusion): preserve regexp_match null checks by @app/lance-gatefixer in #9698
• fix(scanner): preserve requested projection in fast search by @app/lance-gatefixer in #9699
• fix: enforce AIMD delete request budget by @kamronis in #9707
• fix(cleanup): tolerate missing objects without error logs by @app/lance-gatefixer in #9709
• fix(io): resolve cloud option aliases deterministically by @app/lance-gatefixer in #9719
• fix(index): reject JSON array and object index paths by @app/lance-gatefixer in #9727
• fix(index): remap every segment of a tagged index instead of only the first by @LuQQiu in #9728
• fix(index): admit MemWAL state updates on tables with a tagged fragment reuse history by @LuQQiu in #9729
• fix(index): check rewrite groups one by one and decode the FRI ledger only when needed by @LuQQiu in #9731
• fix(mem_wal): full-text search across columns added or renamed since a source was written by @xuanyu-z in #9740
• fix(merge_insert): accept legacy blob input for Blob v2 columns by @wkalt in #9741
• fix(merge_insert): rewrite rows for partial blob sources on indexed joins by @wkalt in #9742
• fix(cleanup): reclaim abandoned temporary data files by @app/lance-gatefixer in #9747
• fix: make MultiMatchQuery serde lossless by @hamersaw in #9753
• fix(io): reject zero multipart upload concurrency by @kamronis in #9756
• fix(merge-insert): preserve join columns in horizontal updates by @app/lance-gatefixer in #9765
• fix: keep stable row IDs when cloning or branching an emptied table by @brendanclement in #9768
• fix(index): bound preflight scans and batch vector sampling by @BubbleCal in #9776
• fix(cleanup): share delete rate limit across cascaded branches by @majin1102 in #9782
• fix(mem_wal): cast a point lookup's base struct to the selected child order by @hamersaw in #9788
• fix(python): accept version 0 as the begin of an explicit delta range by @LuciferYang in #9794
• fix(index): refuse preprocessed training data in a shape the index does not train from by @xuanyu-z in #9795
• fix(mem_wal): keep null rows out of memtable B-tree range filters by @xuanyu-z in #9796
• fix(commit): let a deferred rewrite exempt a reindex across processes on untagged FRI tables by @xuanyu-z in #9808
• fix(mem_wal): report one column statistic per output column from memtable scans by @hamersaw in #9809
• fix: keep scan order when the optimizer adds parallelism by @vivek-bharathan in #9819
• fix(namespace): preserve unbounded IVF probe maximum by @jackye1995 in #9822
• fix(cache): count IndexMetadata::index_details in deep_size_of by @zehiko in #9828
• fix(mem_wal): skip already-flushed generations when replaying the WAL by @xuanyu-z in #9840
• fix(mem_wal): read each memtable's visible batches once per LSM query by @xuanyu-z in #9841
• fix(mem_wal): rank every vector search arm on one distance scale by @hamersaw in #9849
• fix: allow renames alongside column casts by @app/lance-gatefixer in #9854
• fix(mem_wal): keep the table's field ids in flushed generations by @xuanyu-z in #9857
• fix(index): write zone map seeds when a seeded column gets a new data file by @LuQQiu in #9868
• fix(blob): require explicit opt-in for managed storage by @jackye1995 in #9891
• fix(mem_wal): compare maintained indexes by resolved settings, not base details by @xuanyu-z in #9899
• fix(index): keep a vector index's details when remapping it by @xuanyu-z in #9900
• fix(encoding): size full-zip validity by decoded items, not rows by @dshepelev15 in #9906
• fix: count rows selected by row id or row address by @vivek-bharathan in #9907
• fix(encoding): do not materialize the constant scalar into null slots by @dsingh-exa in #9909
• fix(encoding): bounds-check the item payload in VariableFullZipDecoder::unzip by @dsingh-exa in #9910
• fix(io): call multipart complete once and retry it at the HTTP layer by @dsingh-exa in #9913
• fix(index): trace segment coverage across recorded compactions by @brendanclement in #9946
• fix(merge_insert): build update_if structs by name on indexed joins by @wkalt in #9950
• fix(lance-jsonb): escape object keys when displaying Value by @westonpace in #9951
• fix(mem_wal): make shard manifest versions safe to prune by @xuanyu-z in #9953
• fix: read struct children added below a list item at any depth by @wkalt in #9954
• fix(mem_wal): keep field ids on prepared blob v2 columns by @xuanyu-z in #9957
• fix(mem_wal): emit an LSM scan's sources in a fixed order by @hamersaw in #9983
• fix(namespace): use POST for REST count_rows, tags/list and stats by @twmb in #9993
• fix(namespace): tell tables and namespaces apart in the dir manifest by @twmb in #9998
• fix(index): harvest zone map seeds from the serving data file by @BubbleCal in #10003
• fix(index): reject mixed row domains in scalar index segments by @BubbleCal in #10004
• fix(io): stop ignoring storage configuration that lance accepts by @LuciferYang in #10008
• fix(io): resolve locations that the rest of the stack accepts by @LuciferYang in #10009
• fix(io): record the IO that currently goes uncounted by @LuciferYang in #10010
• fix(scanner): align row offset statistics with output schema by @BubbleCal in #10030
• fix(mem_wal): retry a failed memtable flush before any later generation by @hamersaw in #10031
• fix(mem_wal): reject unhashable primary key types at initialization by @hamersaw in #10040
• fix(compaction): remap address-domain indices eagerly on stable-row-id tables by @westonpace in #10041
• fix(fts): refine stop words across latin-script languages by @BubbleCal in #10045
• fix(fts): preserve Chinese and Japanese search terms by @BubbleCal in #10046
• fix(fts): preserve Korean content and numeric tokens by @BubbleCal in #10048
• fix(fts): preserve Arabic content and polarity by @BubbleCal in #10049
• fix(fts): preserve Greek content and polarity by @BubbleCal in #10050
• fix(fts): preserve Russian content and numeric terms by @BubbleCal in #10051
• fix(fts): preserve Tamil content and numeric terms by @BubbleCal in #10052
• fix(index): allow partition rebalancing within explicit merge windows by @BubbleCal in #10076
• fix(index): keep the earliest-created segment when every index segment is empty by @ump45nose in #10107
• fix(index): drop old fragment ids from stale indices on deferred overlay compaction by @ump45nose in #10109
• fix(fsst): bound sampling for sparse inputs by @LuQQiu in #10120
• fix(fts): size the staged finalize from the build's memory config and report one progress stage by @LuQQiu in #10141
• fix(encoding): admit repeated Planar dictionary candidates by @Xuanwo in #10184
• fix(namespace): conform the REST client and adapter to the spec by @twmb in #10190
• fix(io): poll every sibling file reader in MergeStream to avoid deadlock by @westonpace in #10191
• fix(blob): backport descriptor subfield queries to v14 by @lance-community in #10247

Documentation 📚

• docs(format): define generic mini-block offsets by @Xuanwo in #8332
• docs: specify JSONB logical and physical storage by @Xuanwo in #9194
• docs: apache ozone as object store configuration guide by @ChenSammi in #9601
• docs: document the managed blob table capability by @Xuanwo in #9778
• docs(format): stabilize spilled row lineage by @BubbleCal in #9800
• docs(format): stabilize spilled row lineage by @BubbleCal in #9886
• docs(format): allow variable-length zones in version 1 write seeds by @LuQQiu in #10155

Performance Improvements 🚀

• perf(scanner): renmove backtics from column names by @yuvalif in #7782
• perf(encoding): measure compressed mini-block offsets by @Xuanwo in #8333
• perf(index): merge Bitmap index segments instead of rescanning the dataset by @amunra in #9290
• perf(index): fetch the IVF and quantizer global buffers together on index open by @LuciferYang in #9378
• perf(index): bucket shuffle rows by partition id instead of sorting keys by @LuciferYang in #9382
• perf: prune CDF fragments using row-version metadata by @Abhisheklearn12 in #9409
• perf(index): skip empty ranges when planning a partition read by @LuciferYang in #9413
• perf(index): reduce RQ threshold search overhead by @BubbleCal in #9464
• perf(index): reuse bounded scratch without serializing RQ batches by @BubbleCal in #9465
• perf(index): fuse RQ rotation quantization and factor computation by @BubbleCal in #9466
• perf(table): keep clustered row ID ranges compact in index by @Abhisheklearn12 in #9501
• perf(filter): narrow exact double literals against Float32 columns by @jonasdedden in #9512
• perf(python): read the version number without building the summary by @zhangyue19921010 in #9593
• perf(mem_wal): build LSM source scans with bounded concurrency by @beinan in #9642
• perf(scan): reuse fragment lookup when decoding plans by @geruh in #9653
• perf(core): index column names in Schema::project by @LuciferYang in #9711
• perf(index): bound merge_segments prefetch to one scan's byte budget by @amunra in #9761
• perf(mem_wal): answer a filtered LSM scan of memtables from their filter indexes by @xuanyu-z in #9806
• perf(mem_wal): rank only prefiltered, newest memtable rows in a filtered vector search by @hamersaw in #9812
• perf(index): add four-chain native u2 and u6 ex-dot kernels by @BubbleCal in #9839
• perf(mem_wal): grow memtable B-tree arena chunks from 4 KiB by @xuanyu-z in #9848
• perf(index): record zone map seed null positions as runs by @LuQQiu in #9860
• perf(index): skip nested value traversal when collecting zone map statistics by @LuQQiu in #9874
• perf(transaction): handle Rewrite groups in O(n) and bound-check group ranges by @dsingh-exa in #9908
• perf(dataset): serve read_version_transaction from the held or cached manifest by @dsingh-exa in #9915
• perf(python): read the version id directly when committing a transaction by @dsingh-exa in #9916
• perf(fragment): read the version id directly when writing deletion files by @dsingh-exa in #9917
• perf(encoding): write zstd output into reserved capacity instead of zero-filling by @dsingh-exa in #9918
• perf(encoding): bound planar sequence and offset codec selection by @Xuanwo in #9923
• perf(encoding): reduce composite RLE selection and materialization costs by @Xuanwo in #9924
• perf(file): decode lazily loaded column metadata from the tail read by @dshepelev15 in #9938
• perf(index): read the manifest version without summarizing every fragment by @dshepelev15 in #9942
• perf(index): cache the ownership mask of a vector index segment by @dshepelev15 in #9943
• perf(index): train and encode PQ codebooks through an f32 SoA centroid table by @dshepelev15 in #9945
• perf(scanner): serve null-check filters from a narrow leaf by @wkalt in #9948
• perf(merge_insert): evaluate target-column conditions on the rows they gate by @wkalt in #9949
• perf(mem_wal): allocate HNSW neighbor lists per node and charge missed index memory by @xuanyu-z in #9990
• perf(dataset): sum manifest row counts in Dataset::count_rows(None) by @zhangyue19921010 in #10016
• perf(scan): share one planned filter across all fragments of a scan by @zhangyue19921010 in #10042
• perf(merge_insert): prune target columns from the join scan unless matched rows are updated by @zhangyue19921010 in #10043
• perf(rowids): load fragment sequences on demand by @BubbleCal in #10070
• perf(fts): finalize staged partitions in place with budgeted parallel copies and consolidation by @LuQQiu in #10138
• perf(io): raise the default AIMD ceiling to 10k by @BubbleCal in #10153
• perf(dataset): take rows by id without reading deletion files by @dshepelev15 in #10163
• perf(dataset): open only the data files that hold a projected field by @dshepelev15 in #10178
• perf(encoding): dictionary-encode repeated fixed-size vectors by @Xuanwo in #10187
• perf(fts): load requested postings on cold searches by @BubbleCal in #10198
• perf(fts): pack postings during position prewarm by @BubbleCal in #10201

Other Changes

• refactor(mem-wal): finish the SSTable/frozen naming pass by @hamersaw in #8836
• refactor(core): share CloneableError via Arc instead of rebuilding on clone by @LuciferYang in #9626
• refactor: stop using DataFusion APIs deprecated or changed in DataFusion 55 by @slachiewicz in #9640
• refactor(index): expose RaBitQ encoding and factor primitives by @BubbleCal in #9832
• refactor(mem_wal): drop a dead branch and repeated tests, correct stale docs by @xuanyu-z in #9838
• refactor(encoding): reuse generic blocks for sparse structural arrays by @Xuanwo in #9843
• revert: undo premature spilled row lineage stabilization by @BubbleCal in #9885
• Revert "docs: specify JSONB logical and physical storage" by @Xuanwo in #9889
• refactor(format): remove unstable declarations from stable encoding schema by @Xuanwo in #9922
• refactor: split dataset/blob.rs into submodules by stage by @wjones127 in #9989
• refactor(namespace): split lance-namespace-impls dir.rs into submodules by @zhangyue19921010 in #10073
• refactor(encoding): split encodings/logical/primitive.rs into submodules by @zhangyue19921010 in #10075

Full Changelog: release-root/14.0.0-beta.N...v14.0.0-rc.2