v15.0.0-beta.2

What's Changed

Breaking Changes 🛠

• feat(format)!: add file metadata size hint by @Ali2Arslan in #9168
• fix(mem_wal): stop publication after terminal L0 flush failures by @app/lance-gatefixer in #9959
• refactor!: add lance-datafusion-exec so lance-index skips datafusion by @wjones127 in #10251
• feat(encoding): add modern ALP miniblock compression by @Xuanwo in #10267

Critical Fixes ‼️

• fix: support file-less fragments by @skyshineb in #7238
• fix: resolve struct validity from data overlays by @app/lance-gatefixer in #9092
• fix: scope row address mask cache by manifest generation by @app/lance-gatefixer in #9094
• fix(core): narrow LargeList children in schema intersection by @LuciferYang in #9503
• fix(io): prevent scan scheduler byte-budget deadlock by @app/lance-gatefixer in #9576
• fix(stable_row_id): segmented scalar index returns stale rows under stable row ids by @zhangyue19921010 in #9785
• fix(fts): segmented full-text index returns stale rows under stable row ids by @zhangyue19921010 in #9872
• fix(encoding): preserve the declared dictionary key type when merging dictionary arrays by @lichuang in #9958
• fix(encoding): reject packed-struct children the encoder cannot pack by @LuciferYang in #9963
• fix(table): validate DynamoDB key types and page the manifest delete by @LuciferYang in #9977
• fix(compaction): keep rewritten files when fragment-id reservation fails by @dsingh-exa in #9995
• fix(io): avoid hanging and spurious Pending in ObjectWriter::poll_write by @dsingh-exa in #9996
• fix: evaluate constant-first spatial predicates correctly by @sarwarbhuiyan in #10080
• fix(mem_wal): search an SSTable generation without a vector index flat by @hamersaw in #10139
• fix(index): read LIKE escapes like the executor when extracting a prefix by @LuciferYang in #10228

New Features 🎉

• feat: add metrics for allow list / block list size from query search by @westonpace in #9255
• feat(filter): compare integer columns with float literals exactly and keep the scalar index by @jonasdedden in #9333
• feat(java): expose vector distance bounds by @nyl3532016 in #9374
• feat(python): implement LanceScanner.take() matching pyarrow semantics by @lichuang in #9468
• feat(mem_wal): add a streaming to_reader to the MemWAL LsmScanner by @zhangyue19921010 in #9694
• feat(format): add data_change to DataReplacement by @LuciferYang in #9705
• feat(index): change bitmap and label_list scalar indexes to row-address domain by @westonpace in #9814
• feat: add ScalarIndex::lookup for batched exact-match key lookups by @wjones127 in #9875
• feat: pass WriteParams to merge insert by @ragnorc in #9888
• feat(format): let a new table keep the field IDs its input supplies by @xuanyu-z in #9898
• feat(index): recover scalar segment build parameters by @Seanium in #9994
• feat(fragment): expose deletion markers through FragReadConfig by @dsingh-exa in #9997
• feat(python): export memwal and audit transaction fields by @XuQianJin-Stars in #10064
• feat(python): expose the FM index to python by @zhangyue19921010 in #10196
• feat(java): expose the FM index to java by @zhangyue19921010 in #10199
• feat(blob): add managed blob writer for explicit sharing by @Xuanwo in #10233
• feat: expose data_cache_bytes in CompactionOptions by @ilya-zlobintsev in #10238
• feat(format): define per-tensor axis delta encoding for Lance 2.3 by @Xuanwo in #10240
• feat(encoding): automatically select axis delta for compressed integer tensors by @Xuanwo in #10242
• feat(index): consolidate bounded vector segments without retraining by @Xuanwo in #10335

Bug Fixes 🐛

• fix: improve retry handling for rename_if_not_exists by @xuzifu666 in #6544
• fix(index): defer scalar index parser selection until operator context is known by @ztorchan in #7509
• fix(index): report distance comparisons from HNSW search by @wombatu-kun in #8142
• fix(dataset): track fragment add-columns cleanup by @app/lance-gatefixer in #8402
• fix: request vended credentials in from_namespace's initial describe by @rpgreen in #8518
• fix(encoding): handle empty miniblock ranges by @app/lance-gatefixer in #8559
• fix(linalg): make u8x16 multiplication wrap on x86 by @app/lance-gatefixer in #8712
• fix(namespace): forward opaque backfill/refresh options through RestNamespace by @sarwarbhuiyan in #8762
• fix(linalg): make SIMD slice conversions baseline-safe by @app/lance-gatefixer in #8874
• fix: list unreferenced data files newer than the retained floor by @jennifermell in #8945
• fix(python): correct the asof and reader input annotations by @jonasdedden in #8955
• fix(index): write a single-page rtree index as one page by @jackylee-ch in #8988
• fix(index): reject out-of-range partition ids when computing residuals by @LuciferYang in #9004
• fix(python): reject a num_partitions that disagrees with ivf_centroids_file by @LuciferYang in #9009
• fix(index): reject cosine and hamming in float kmeans instead of panicking by @LuciferYang in #9027
• fix(mem-wal): wait for primary key index before flush by @app/lance-gatefixer in #9036
• fix(index): keep scalar indexes on a LIKE with a mixed string literal by @wombatu-kun in #9054
• fix(substrait): detect unsupported fixed-size-list filters by @app/lance-gatefixer in #9057
• fix(cleanup): collect orphaned staging manifests by @app/lance-gatefixer in #9062
• fix(lance-io): reject malformed message length prefix in read_message_from_buf by @foobar in #9070
• fix: preserve nested field path when optimizing scalar index by @app/lance-gatefixer in #9147
• fix: support unordered Fragment.take offsets by @app/lance-gatefixer in #9158
• fix: list hierarchical branches on object stores by @TheR1sing3un in #9198
• fix(encoding): reject mismatched miniblock buffer sizes by @app/lance-gatefixer in #9237
• fix(fts): preserve corpus statistics for restricted matches by @ruslan-shaydullin in #9243
• fix(index): keep the trailing rows when shuffling a v2 buffer by @jackylee-ch in #9246
• fix(index): decline incompatible JSON accessor paths by @app/lance-gatefixer in #9260
• fix(file): report the bytes actually read on invalid magic by @crossthebridgetpa in #9300
• fix(index): error on a malformed precomputed partitions file by @LuciferYang in #9342
• fix(index): reject unattainable partition counts in weighted k-means by @LuciferYang in #9346
• fix(index): enforce the precomputed-input contracts in the current IVF build path by @LuciferYang in #9376
• fix(dataset): validate nested schema names and stored system columns by @app/lance-gatefixer in #9499
• fix(index): don't recompute fragment coverage for bitmap/label_list/inverted by @jackylee-ch in #9544
• fix: reject compared_against combined with a version range in delta by @jackylee-ch in #9552
• fix(compaction): reject zero num_threads instead of hanging by @jackylee-ch in #9556
• fix(scan): include sort columns in indexed read projections by @app/lance-gatefixer in #9625
• fix(index): keep HNSW partition metadata parseable after empty remap by @kamronis in #9627
• fix(python): honor set_epoch in sharded samplers by @MohammadHijjawi97 in #9636
• fix(python): make maybe_sample return the requested number of rows by @MohammadHijjawi97 in #9637
• fix(scanner): respect sliced list query offsets in vector search by @kamronis in #9639
• fix(datafusion): reject a zero batch_size in LanceExecutionOptions by @slachiewicz in #9641
• fix(fts): honor same-position alternatives in flat phrases by @app/lance-gatefixer in #9683
• fix(fts): reject incompatible segments before commit by @app/lance-gatefixer in #9684
• fix(fts): preserve query position gaps in MemWAL phrase search by @app/lance-gatefixer in #9685
• fix(mem_wal): read frozen memtables in ShardWriter.lsm_scanner by @zhangyue19921010 in #9687
• fix(mem_wal): return JSON columns as Arrow JSON from LsmScanner by @zhangyue19921010 in #9693
• fix(mem_wal): sync MemWAL type stubs with the binding signatures by @zhangyue19921010 in #9701
• fix(cleanup): remove empty blob sidecar directories by @app/lance-gatefixer in #9748
• fix(python): train cosine PQ codebook with L2 on normalized residuals in accelerated IVF_PQ by @Chastrlove in #9755
• docs(guide): document S3 conditional writes and external locks by @app/lance-gatefixer in #9763
• fix(index): filter obsolete RTree rows during updates by @kamronis in #9779
• fix(io): delete hierarchical storage directories after their contents by @everySympathy in #9827
• fix: apply a refined vector search's distance range on the index's scale by @hamersaw in #9851
• fix(arrow): preserve rechunked batch tails by @app/lance-gatefixer in #9852
• fix(python): declare compaction JSON helpers in type stubs by @hfutatzhanghb in #9893
• fix(scanner): select a vector index by the query's metric, not just veto the first by @hello-peter-tang in #9895
• fix(encoding): restore 2.0 scan speed by sending indirect reads early by @keunhong in #9903
• fix: refuse to rebind a base ID that is already registered by @brendanclement in #9904
• fix(encoding): recover wrapped mini-block level counts for bitpacked levels by @dsingh-exa in #9911
• fix(encoding): don't panic reading a fixed-size-list over a list by @LuciferYang in #9930
• fix(index): release cached FTS readers when requests end by @app/lance-gatefixer in #9937
• fix(index): fold data overlays into a remapping scalar segment merge by @amunra in #9941
• fix(compression): honor configured LZ4 compression by @dentiny in #9956
• fix(index): avoid order-biased hierarchical k-means sampling by @ddupg in #9964
• fix(table): detect schema-metadata conflicts per key by @LuciferYang in #9975
• fix(table): recognize leaf-only struct files in merge rewrite validation by @LuciferYang in #9976
• fix(scan): trim filter-only children from nested projections by @app/lance-gatefixer in #10029
• fix(encoding): sort a lone unsorted batched read by @dsingh-exa in #10036
• fix(mem_wal): keep prepared blob v2 columns prepared when reconciling batches by @xuanyu-z in #10081
• fix(linalg): unify Arrow distance dispatch by @app/lance-gatefixer in #10085
• fix(index): match nested fields by subtree in index maintenance by @ump45nose in #10108
• fix(commit): retry a data overlay rebased over a projection that drops its field by @ump45nose in #10123
• fix(datafusion): reject unplannable filter literals and expression lists, narrow under NOT by @LuciferYang in #10131
• fix(datafusion): make the JSON UDFs match the JSON guide by @LuciferYang in #10132
• fix(datafusion): read every row of the contains_tokens query by @LuciferYang in #10133
• fix(substrait): prune fixed-size lists nested in large lists and maps by @LuciferYang in #10134
• fix(datafusion): load system columns used inside projection expressions by @LuciferYang in #10135
• fix(write): reject zero row limits instead of panicking or writing nothing by @LuciferYang in #10136
• fix: report bloom filters as intersecting only when a block could hold a shared key by @xuanyu-z in #10143
• perf(mem_wal): publish HNSW level 0 only in the packed snapshot by @xuanyu-z in #10161
• fix(merge_insert): prune source columns by what the write exec reads by @zhangyue19921010 in #10175
• fix(mem_wal): raise a valid ef to each indexed arm's candidate count by @xuanyu-z in #10181
• fix(mem_wal): give the memtable graph the beam a refined search gives a flushed generation by @xuanyu-z in #10185
• fix(mem_wal): keep repeated vectors reachable in the memtable HNSW graph by @xuanyu-z in #10186
• fix: prune stale R-tree coverage after column updates by @app/lance-gatefixer in #10209
• fix: enforce non-nullable field coverage when committing v2 Append by @Ppei-Wang in #10211
• fix(encoding): stop emitting empty chunk instructions when a row ends by @lichuang in #10219
• fix(io): release spill quota after aborted writes by @kamronis in #10231
• fix(java): add explicit physical-address blob selection by @a-erofeev in #10237
• fix(java): preserve shared sessions when opening namespace datasets by @jischeng in #10239
• fix(index): use tagged FRI coverage for overlay stale detection by @app/lance-gatefixer in #10249
• fix: reject unsafe legacy index coverage by @app/lance-gatefixer in #10256
• fix(python): isolate TorchInductor caches per pytest worker by @app/lance-gatefixer in #10261
• fix: unblock main CI after jsonb vendor rename and pb23 metadata switch by @Xuanwo in #10262
• fix(index): continue coreset splits and stabilize PQ smoke test by @BubbleCal in #10270
• fix(index): translate staged row ids in pairwise staging on tagged FRI histories by @LuciferYang in #10272
• fix(scanner): use the index metric in the flat fallback by @wombatu-kun in #10273
• fix: return every non-null row for contains(col, '') with an FM index by @zhangyue19921010 in #10275
• fix: honor FM-Index num_segments for IndexType::Scalar builds by @zhangyue19921010 in #10276
• fix(encoding): merge page dictionaries exactly when a batch spans pages by @LuciferYang in #10286
• fix(index): update HNSW tests and bench for new search return type by @Xuanwo in #10292
• fix(python): pass commit failures to lease context managers by @easyrev in #10293
• fix(index): repair HNSW prefetch checks by @BubbleCal in #10294
• fix(namespace): address nested/non-ASCII locations when probing manifests and dropping tables by @linhongyu510 in #10307
• fix: keep transaction reads progressing with one I/O slot by @beinan in #10308
• fix(index): preserve HNSW build params in empty partition metadata by @Xuanwo in #10316
• fix(fts): share the lindera dictionary across tokenizer clones by @Shell-Wataru in #10319
• fix(index): reject a zero target partition size and out-of-range precomputed partition ids by @LuciferYang in #10322
• fix(index): filter retired fragments after coverage pruning by @Xuanwo in #10325
• fix(deps): update zstd for new RustSec advisories by @Xuanwo in #10348

Documentation 📚

• docs: describe Lance file encodings as container extensions by @niraj-mx07 in #8124
• docs: add specification page for standard table config keys by @app/copilot-swe-agent in #8673
• docs: mount lance-flink connector docs into the integrations site by @fightBoxing in #8987
• docs: re-sync the Rust HNSW example page with its compiled sources by @wombatu-kun in #9074
• docs(linalg): document the argmin/argmax optional-value kernels by @fightBoxing in #9124
• docs: specify Blob v2 input, storage, and lifecycle contracts by @Xuanwo in #9193
• docs: list supported cloud storage in README by @minhh-nguyen in #9749
• docs: document the FTS code analyzer by @hello-peter-tang in #9780
• docs: fix broken quickstart and branch guide examples by @wjones127 in #10032
• docs(format): specify btree indices on stable row ids by @BubbleCal in #10150
• docs: introduce Blob v2 input values before schemas by @Xuanwo in #10278
• docs: document ALP encoding and configuration by @Xuanwo in #10287

Performance Improvements 🚀

• perf: avoid repeated concatenation in strict batch sizing by @sliortega295-ops in #8881
• perf(index): cap HNSW prefetch bytes and honor prefetch_distance by @RupertMaiti2005 in #8882
• perf(index): floor the default HNSW search ef at 256 by @wombatu-kun in #9073
• perf(lance-table): slice a Range segment in O(1) by @LuciferYang in #9309
• perf(scan): resolve deletions from row offsets instead of row addresses by @taozhiyi13 in #9373
• perf(index): use peek_mut for in-place top-k heap updates in RQ storage by @kamronis in #9484
• perf(core): count Set deletion-vector ranges by iterating members by @LuciferYang in #9500
• perf(encoding): decode inline bitpacking directly into output by @kamronis in #9603
• perf(index): stream bitmap segments sequentially during merge by @Xuanwo in #9633
• perf(knn): bound batched ANN candidate merge across segments by @kamronis in #9688
• perf(cleanup): stop reading expired manifests when delete_unverified is set by @jackye1995 in #9764
• perf(dataset): load deletion vectors only for selected offsets by @kamronis in #9824
• perf(index): add fused layered ex-dot kernels by @BubbleCal in #9831
• perf(select): move owned row-address masks during union by @WanSuiMm in #9884
• perf(index): batch ngram posting-list reads instead of one row per request by @hello-peter-tang in #9887
• perf(commit): dedupe fix_schema field-list scans and encode the manifest once by @dsingh-exa in #9914
• perf(encoding): stop recomputing fixed-size-list child statistics by @LuciferYang in #9931
• perf(encoding): slice miniblock binary values zero-copy by @LuciferYang in #9933
• perf(linalg): select u8 l2/dot/cosine kernels once via Dispatch by @wjones127 in #10103
• perf(mem_wal): scan each on-disk source once per batch of keys, not once per key by @xuanyu-z in #10142
• perf(dataset): select sample offsets without traversing every row by @kamronis in #10172
• perf(encoding): skip the RLE size scan when RunCount already rules RLE out by @zhangyue19921010 in #10182
• perf(encoding): build validity bitmaps in bulk when decoding nullable columns by @zhangyue19921010 in #10200
• perf(fts): cache compact postings from persistent groups by @BubbleCal in #10203
• perf(fts): avoid read ahead for sparse posting groups by @BubbleCal in #10204
• perf(fts): adapt search batches to query concurrency by @BubbleCal in #10205
• perf: avoid deep-copying read-version fragments on each CreateIndex commit attempt by @zhangyue19921010 in #10206
• perf(file): join the short tail read with memcpy instead of a byte loop by @dshepelev15 in #10282
• perf(table): probe the row id index by segment pieces instead of fragment spans by @dshepelev15 in #10284
• perf(encoding): speed up ALP parameter selection by @Xuanwo in #10289
• perf(fts): stream ICU tokens during text analysis by @BubbleCal in #10290
• perf(fts): yield between indexing input batches by @BubbleCal in #10296
• perf(file): overlap and coalesce reads during concatenation by @BubbleCal in #10301
• perf(encoding): vectorize ALP decode restore with SIMD delta scan by @Xuanwo in #10309
• perf(io): validate checkpoint-sized small parts in one GET by @Xuanwo in #10315
• perf(cleanup): bound concurrent managed blob reference scans by @Xuanwo in #10333

Other Changes

• refactor(lance-index): cache tokenize_and_count's constant schema fie… by @jagrutipatilp in #8684
• refactor(index): let the v3 shufflers own their scratch directory by @LuciferYang in #9265
• refactor(core): generate Kernel::call impls with a macro by @wjones127 in #10214

Full Changelog: v15.0.0-beta.1...v15.0.0-beta.2