v15.0.0-beta.8

What's Changed

Breaking Changes 🛠

• feat(tools): add table manifest command by @bugwz in #7970
• perf(index)!: parallelize zone statistics computation in scalar index training by @zhangyue19921010 in #8469
• feat(dataset)!: expose manifest_size on Version by @hiltonhe in #9102
• perf(index): reuse RaBitQ LUTs and share IVF query state by @xiaguanglei in #9190
• perf(linalg)!: select f32/f64 cosine kernels once from a host feature set by @wjones127 in #10088
• feat(table): store small deletion vectors in the manifest by @dshepelev15 in #10164

Critical Fixes ‼️

• fix(file): normalize fixed-size struct list row counts by @vera-rubin in #9999
• fix: protect pre-3.0 branches from cleanup by @LuciferYang in #10327
• fix(cleanup): reject cleanup from a detached checkout by @LuciferYang in #10331
• fix(index): conflict CreateIndex with a DataReplacement of a packed parent by @easyrev in #10341
• fix(encoding): size dictionary decode output exactly for large values by @wombatu-kun in #10376
• fix(scanner): inherit index metrics in vector search filters by @app/lance-gatefixer in #10377
• fix(index): reject a PQ codebook of another value type and quantize cosine transforms with L2 by @LuciferYang in #10380
• fix(fts): refresh WAND windows before advancing parked clauses by @app/lance-gatefixer in #10408

New Features 🎉

• feat(namespace-dir): support manifest-backed table renames by @Ar-maan05 in #6975
• feat(java): expose SourceDedupeBehavior in MergeInsertParams by @sezruby in #7296
• feat(index): tokenize JSON arrays into sub-documents by @wirybeaver in #7377
• feat(sql): register additional tables for Dataset.sql queries by @sbrunk in #7839
• feat(io): add retry support for OpenDAL stores by @ddupg in #8363
• feat(index): add opt-in quantized vector refinement by @kamronis in #8741
• feat(commit): default unrecognized schemes to put-if-not-exists commit handler by @farazshaikh in #8773
• feat: added ability to configure scheduler error handling mode by @a-erofeev in #8780
• feat(encoding): implement plan_decoded_bytes for variable-width structural decoder by @westonpace in #8793
• feat(fts): expose global BM25 statistics for distributed queries by @zhangstar333 in #8938
• feat(java): expose aggregate index fragment coverage by @everySympathy in #8961
• feat(scanner): ignore selected scalar indices by @majin1102 in #9034
• feat(python): add scan_stats_callback to LanceFragment.scanner by @manucorporat in #9149
• feat: warn if using legacy storage by @westonpace in #9476
• feat(cleanup): add configurable version retention policies by @jackye1995 in #10306
• feat: skip index segments a prefilter proves empty in ANN and FTS searches by @LuQQiu in #10485
• feat: add predecessor-fenced dataset restore by @Xuanwo in #10556
• feat(fts): expose flat match state for plan reconstruction by @BubbleCal in #10561

Bug Fixes 🐛

• fix(file): return errors instead of panicking on malformed files by @lichuang in #10334
• fix(encoding): count validity per list, not per item, when a batch spans pages by @zhangyue19921010 in #10338
• fix(encoding): account for compacted slices in page buffers by @qidian99 in #10344
• fix(index): include covering fields in index dependency checks by @Ali2Arslan in #10345
• fix(encoding): validate zstd and LZ4 length prefixes before allocating by @zhangyue19921010 in #10360
• fix(fts): preserve corpus across match and combined-field scans by @app/lance-gatefixer in #10362
• fix(python): leave an unset VectorSearchQuery filter metric to the index by @LuciferYang in #10375
• fix(io): prune dead entries from ObjectStoreRegistry cache, with a configurable threshold by @ryc111 in #10385
• fix(dataset): preserve branch references during cleanup by @geruh in #10392
• fix(index): draw disjoint rows for each streaming coreset IVF step by @wombatu-kun in #10410
• fix(table): quiet Windows version hint write failures by @app/lance-gatefixer in #10438
• fix: protect fragment reuse history on stable-row-ID tables (re-land) by @wjones127 in #10441
• fix(dataset): protect parent files during nested branch cleanup by @geruh in #10447
• fix(blob): decode legacy blob pages in 2.2 files after promotion by @jackylee-ch in #10454
• fix(python): expose direct tag lookup by @app/lance-gatefixer in #10457
• fix(encoding): reject fixed-size lists of strings and binaries instead of panicking by @jackylee-ch in #10460
• fix(fts): require all ngrams in AND queries by @app/lance-gatefixer in #10476
• fix(namespace): protect shared locations when dropping tables by @app/lance-gatefixer in #10483
• fix(dataset): preserve complete branch source snapshots during cleanup by @app/lance-gatefixer in #10488
• fix(dataset): clean deleted branches from shared directories by @app/lance-gatefixer in #10489
• fix(merge-insert): preserve null-key rows in indexed merges by @app/lance-gatefixer in #10494
• fix(fts): preserve column-ordered pagination by @app/lance-gatefixer in #10501
• fix: reject appends after concurrent column replacement by @app/lance-gatefixer in #10502
• fix(compaction): reject eager remaps over pending fragment reuse moves by @app/lance-gatefixer in #10507
• fix(java): release Dataset lock during compaction and segment index builds by @zhangyue19921010 in #10508
• fix: prevent metadata loss from concurrent column projections by @app/lance-gatefixer in #10512
• fix(index): reject builds over dropped or cast columns by @app/lance-gatefixer in #10519
• fix(dataset): read delta rows from the end version by @app/lance-gatefixer in #10520
• fix(dataset): require stable row ids for row change tracking by @app/lance-gatefixer in #10521
• fix(encoding): warn when a requested per-value codec is ignored by @dshepelev15 in #10525
• fix(index): stop cached legacy indexes from pinning file descriptors by @hello-peter-tang in #10534
• fix(java): decode strings outside the BMP passed from Java by @jackylee-ch in #10540
• fix(cleanup): keep tagged versions in automatic cleanup instead of failing by @jackylee-ch in #10541
• fix: call WriteFragmentProgress::complete for the last file of a write by @wombatu-kun in #10545
• fix(substrait): decode filters on fields inside struct columns by @jackylee-ch in #10546

Documentation 📚

• docs(linalg): document the length contract on the public distance entry points by @LuciferYang in #8879
• docs(linalg): fill in the empty Safety section on load_unaligned by @LuciferYang in #8885

Performance Improvements 🚀

• perf(encoding): shrink flat mini-block chunk index metadata by @Ali2Arslan in #7947
• perf: added external sort-merge join algorithm for large updates by @a-erofeev in #9129
• perf(linalg): select dist_table kernels once from a host feature set by @wjones127 in #10113
• perf(compaction): skip unused address maps using plugin capabilities by @majin1102 in #10148
• perf(encoding): skip the unused size pass for non-final RLE mini-blocks by @zhangyue19921010 in #10448
• perf(exec): report decode time in LanceRead metrics by @dshepelev15 in #10532
• perf(fts): bound prewarmed posting groups by bytes by @BubbleCal in #10555

Other Changes

• refactor(arrow): decode IPC streams in one pass by @Xuanwo in #10462

Full Changelog: v15.0.0-beta.7...v15.0.0-beta.8