v15.0.0-beta.9

What's Changed

Breaking Changes 🛠

• feat(dataset): add expire_versions to remove history without reading manifests by @jackye1995 in #9438
• refactor(index): split Dataset index lifecycle by @Xuanwo in #10600
• refactor(python): organize dataset bindings by capability by @Xuanwo in #10602
• refactor(mem_wal): split write path responsibilities by @Xuanwo in #10605

Critical Fixes ‼️

• fix(io): prevent scan scheduler byte-budget deadlock by @app/lance-gatefixer in #9576
• fix(index): remap FM-index rows after deferred compaction by @app/lance-gatefixer in #10559

New Features 🎉

• feat(datafusion): report measured widths for variable-width scan columns by @vivek-bharathan in #9477
• feat(java): support reusable IVF_RQ rotation models by @ddupg in #9489
• feat(python): expose the data overlay writer in the Python bindings by @jan-exa in #9536
• feat(compaction): add Dataset::column_layout_stats by @LuciferYang in #9605
• feat(skills): add a table maintenance skill by @zhangyue19921010 in #9610
• feat(python): preserve re-scannable merge sources by @easyrev in #9617
• feat(index): expose json query state for distributed planners by @BubbleCal in #10582
• feat(index): support native lazy RaBitQ through fragment reuse by @BubbleCal in #10609
• feat(format): define pcodec mini-block compression for Lance 2.3 by @dakenf in #10617
• feat(fts): expose document expansion plan reconstruction by @BubbleCal in #10622
• feat(fts): expose shared corpus owner reconstruction by @BubbleCal in #10632
• feat(fts): expose bm25f leaf reconstruction state by @BubbleCal in #10638
• feat(fts): expose indexed leaf reconstruction state by @BubbleCal in #10643
• feat(fts): expose compound composition state by @BubbleCal in #10645
• feat(fts): expose compound leaf reconstruction state by @BubbleCal in #10647
• feat(fts): expose flat filter document binding by @BubbleCal in #10649
• feat: add expected-version field metadata commits by @Xuanwo in #10655

Bug Fixes 🐛

• fix: start a new data file before a dictionary outgrows its key type by @LuciferYang in #10368
• fix(index): recheck contains_tokens rows from an FTS index and skip queries it cannot narrow by @LuciferYang in #10571
• fix(index): bound the WAND window max score across all blocks for lists without impacts by @LuciferYang in #10572
• fix(index): reject invalid values in FTS JSON queries by @LuciferYang in #10573
• fix(test): respect row-address writer fence release boundary by @Xuanwo in #10577
• fix(blob): promote legacy blob input to v2 on fragment and update writes by @zhangyue19921010 in #10586
• fix(io): initialize OpenDAL without process constructors by @Xuanwo in #10592
• fix(io): request unencoded bodies from OpenDAL object stores by @plusplusjiajia in #10593
• fix(merge-insert): track indexed primary key inserts by @app/lance-gatefixer in #10606
• fix: reject fragment mutations with missing transaction history by @app/lance-gatefixer in #10608
• fix(filter): compare decimal columns with numeric literals exactly by @jackylee-ch in #10614
• fix(index): keep zone map zones without extrema for LIKE prefix queries by @jackylee-ch in #10615
• fix(compaction): add shared byte-aware consolidation policy by @ChilePiquin in #10635
• fix(dataset): recover V2.0 struct headers after nested casts by @app/lance-gatefixer in #10637
• fix(index): withdraw stale coverage after transaction cleanup by @app/lance-gatefixer in #10663
• fix: reject primary-key inserts with missing transaction history by @app/lance-gatefixer in #10665
• fix(io): keep io_uring buffers and files alive while reads are in flight by @wombatu-kun in #10676

Documentation 📚

• docs: state the stable row id guarantee the same way everywhere by @LuciferYang in #9329
• docs: explain huge pages for decode buffers and the per-value compression rules by @dshepelev15 in #10524
• docs: add repository architecture navigation by @Xuanwo in #10607
• docs: organize user guide navigation and clarify versioning by @Xuanwo in #10621
• docs(examples): replace single-row I/O and internal HNSW calls with batched dataset APIs by @Xuanwo in #10626
• docs: add LanceDB and Apache Spark options to user guide by @Xuanwo in #10628
• docs(quickstart): move PyLance quickstarts to CI-tested snippets by @Xuanwo in #10664
• docs(theme): add track switcher and collapsible sidenav hierarchy by @Xuanwo in #10671
• docs(quickstart): organize get-started by stack layer and add Rust/Java quickstarts by @Xuanwo in #10672
• docs(io): add SAFETY comments to lance-io unsafe blocks by @wombatu-kun in #10673
• docs: separate format user guide from layered ecosystem integrations by @Xuanwo in #10674
• docs: add Python, Rust, and Java tabs across core user guides by @Xuanwo in #10675
• docs: add Python, Rust, and Java tabs to remaining core user guides by @Xuanwo in #10679
• docs: replace homepage static ecosystem graphic with clickable three-layer stack by @Xuanwo in #10680

Performance Improvements 🚀

• perf(vector): reuse prepared f32 PQ8 codebooks during partition search by @xiaguanglei in #9230
• perf(index): admit 16-centroid blocks into the AMX-FP16 GEMM by @xtangxtang in #9234
• perf(index): skip the per-row walk in KeepFiniteVectors when the batch is finite by @LuciferYang in #9371
• perf(core): skip the entry deep-size traversal for size-blind backends by @LuciferYang in #9522
• perf(index): fetch per-segment BM25 statistics concurrently by @LuciferYang in #10574
• perf(rowids): interleave batched position lookups by @BubbleCal in #10613
• perf(index): preserve raw RaBitQ caches across FRI changes by @BubbleCal in #10642
• perf(index): scope stable row id conversion to scanned fragments by @BubbleCal in #10646

Other Changes

• refactor(scanner): split dataset/scanner.rs into submodules by @zhangyue19921010 in #10072
• refactor(merge_insert): split dataset/write/merge_insert.rs into submodules by @zhangyue19921010 in #10074
• refactor(dataset): separate blob rewrite and compaction responsibilities by @Xuanwo in #10597
• refactor(fts): split exec responsibilities into submodules by @Xuanwo in #10598
• refactor(java): organize blocking dataset JNI responsibilities by @Xuanwo in #10601
• refactor(cleanup): move branch lineage tests into their own module by @LuciferYang in #10620

Full Changelog: v15.0.0-beta.8...v15.0.0-beta.9