v10.0.0-rc.1

What's Changed

Breaking Changes 🛠

• perf(compaction)!: skip building row-address maps when index remapping is not needed by @everySympathy in #7778
• perf(cache)!: use fixed-size cache keys by @Ali2Arslan in #7878
• fix(blob)!: preserve null selections across blob APIs by @Xuanwo in #7903

Critical Fixes ‼️

• fix(exec): preserve order when coalescing a sorted multi-partition plan by @wjones127 in #7868

New Features 🎉

• feat(index): support segmented ngram index by @zhangyue19921010 in #7244
• feat: accept TableProvider write inputs for merge_insert and insert by @wjones127 in #7368
• feat(python): add Pydantic model instance auto-conversion and from_pydantic_model() classmethod by @vinaysurtani in #7383
• feat(index): write zone map seeds into data file footers during append by @westonpace in #7427
• feat(python): expose DataOverlay commit operation by @wjones127 in #7540
• feat: support cross-store (cross-account) deep_clone by @sezruby in #7545
• feat(index): mask data overlay files in scalar and vector index queries by @wjones127 in #7549
• feat: support structured_query in the directory namespace query_table by @hamersaw in #7592
• feat: support merge_insert when source omits blob columns by @nyl3532016 in #7615
• feat: add code analyzer for FTS by @Xuanwo in #7681
• feat(compaction): compact fragments over an overlay-count limit by @wjones127 in #7772
• feat(fts): record scorer_build_ms for the exec-local BM25 scorer fallback by @LuQQiu in #7807
• feat(fts): read inverted index params without opening the segment by @LuQQiu in #7816
• feat: expose cached file metadata APIs on FileFragment by @jiaqizho in #7820
• feat(index): per-query index cache hit/miss stats by @yanghua in #7862
• feat: add batch blob range reads by @Xuanwo in #7864
• feat: introduce exact file format identity and persisted codecs by @Xuanwo in #7879
• feat(python): expose LABEL_LIST segment index build by @ddupg in #7884
• feat(java): expose session metadata cache stats to java api by @zhangyue19921010 in #7885
• feat(format): read and write sparse structural pages by @Xuanwo in #7889
• feat(mem_wal): drive the WAL append on a flush-interval ticker by @hamersaw in #7894
• feat(index): support segmented bloom filters by @jackye1995 in #7925
• feat(index): use ACORN-1 traversal for prefiltered HNSW search by @leohoare in #7927
• feat: support segmented RTree indexes by @jackye1995 in #7932
• feat(namespace): idempotent create_table_version with strict CAS and dotfile-safe reserved marker by @XuQianJin-Stars in #7940
• feat(fts): add native exact segment selection by @Xuanwo in #7976

Bug Fixes 🐛

• fix(index): normalize same-column scalar index predicates by @zouhuajian in #6782
• fix: avoid panic due to empty IO buffers on tokio runtime shutdown by @ilya-zlobintsev in #7478
• fix: don't quote column names in list_indices by @dantasse in #7503
• fix(commit): verify finalized external manifests by @BubbleCal in #7578
• fix: skip empty string docs in FTS stats by @Ecthlion in #7699
• fix: keep ids and addresses aligned in filter_deleted_ids under stabl… by @wkalt in #7704
• fix(mem-wal): propagate final flush failures from ShardWriter::close by @u70b3 in #7769
• fix: bracket match query display by @jackye1995 in #7796
• fix(sql): reject invalid LIKE/ILIKE escape strings by @wjones127 in #7810
• fix(table): route tos:// scheme to ConditionalPutCommitHandler by @zhangyue19921010 in #7824
• fix(python): allow blob writer cleanup across threads by @justinrmiller in #7827
• fix(index): reuse cached FTS document lengths by @Xuanwo in #7830
• fix(index): sort JSON-path values once after extraction, not the raw column by @wjones127 in #7835
• fix: make FTS metadata loading retry-safe by @Xuanwo in #7838
• fix: reenable flaky tests on test_torch_kmeans.py by @cswpy in #7849
• fix(fts): restore format version environment fallback by @Xuanwo in #7855
• fix(core): validate CPU/IO env vars with a diagnosable error by @LuciferYang in #7856
• fix: dispatch concurrent blob range reads by @Xuanwo in #7861
• fix(fts): use v3 as the capability gate by @Xuanwo in #7866
• fix(index): return error instead of panicking when hierarchical k-means can't reach target_k by @wjones127 in #7869
• fix: trim fragment-reuse index for non-covering indexes by @xuanyu-z in #7870
• fix(mem_wal): harden shard open and fail reads on a poisoned writer by @hamersaw in #7872
• fix(fts): preserve parameter compatibility by @Xuanwo in #7874
• fix(core): bound SlotBackoff retry sleeps and fix slot overflow by @LuciferYang in #7883
• fix(mem_wal): read-your-writes via split index-apply and dual visibility cursors by @hamersaw in #7888
• fix(mem-wal): propagate task shutdown failures by @u70b3 in #7915
• fix(mem-wal): refill scans after shadow filtering by @u70b3 in #7917
• fix(index): correct offset→row-id translation for overlay masking under deletions by @wjones127 in #7918
• fix(index): keep overlays masked when updating a non-indexed column by @wjones127 in #7926
• fix(namespace): propagate dir-namespace storage errors instead of mapping to TableNotFound by @brendanclement in #7931
• fix(index): rebind vector readers to current object store by @u70b3 in #7944
• fix(blob): unwrap Option in merge_insert blob tests after take_blobs null API by @XuQianJin-Stars in #7963
• fix: persist finalized external manifest metadata by @Xuanwo in #7964
• fix(blob): preserve null and empty values during compaction by @Xuanwo in #7965
• fix(ci): unblock cargo publish and dry-run packaging in CI by @wjones127 in #7989
• fix: preserve blob selection cardinality by @Xuanwo in #8003
• fix(index): prune stale index coverage during rebase by @ddupg in #8011
• fix(index): make vector append segment-set-native by @Xuanwo in #8047
• fix(arrow): preserve all-null struct validity during merge by @mmatczuk in #8049
• fix(io): stabilize wrapper identity in object store cache by @Xuanwo in #8068
• fix(blob): preserve null and empty values across reads and compaction by @Xuanwo in #8070
• fix(fts): propagate nested multimatch limits by @BubbleCal in #8073

Documentation 📚

• docs: implement the Lance Docs design as the mkdocs site theme by @Xuanwo in #7821
• docs: make the docs site theme usable on mobile and narrow screens by @Xuanwo in #7853
• docs: fix nested agent guide loading and trim root AGENTS.md by @Xuanwo in #7992
• docs: include lance-context in the full website build by @beinan in #8014
• docs: define legacy compatibility boundaries by @Xuanwo in #8039

Performance Improvements 🚀

• perf(encoding): cache complex-all-null levels in RLE run form and drain run-by-run by @Ali2Arslan in #7564
• perf(index): norm-addend cache and slim top-k heap for FTS scoring by @BubbleCal in #7629
• perf(encoding): compact mini-block chunk index for cached page state by @Ali2Arslan in #7651
• perf(index): list index segment files concurrently when committing segments by @xloya in #7657
• perf: plan take operations through FilteredReadExec's range-read path by @LuQQiu in #7672
• perf(encoding): avoid inline bitpacking input copy by @u70b3 in #7696
• perf(encoding): auto-select sparse structural pages by @Xuanwo in #7756
• perf(filtered-read): reuse plan-time fragment metadata for the read stream by @LuQQiu in #7792
• perf(dataset): read transactions by version without populating session caches by @xuanyu-z in #7817
• perf(fts): resolve deferred row_ids via the cached whole ROW_ID column by @LuQQiu in #7897
• perf(fts): search partitions in pipelined chunks by @LuQQiu in #7950
• perf(cache): opt-in quick_cache backend for contention-free reads by @LuQQiu in #7953
• perf: improve performance of CandidateBin.split_for_size by @fangbo in #8000
• perf(cache): use the quick_cache backend for the session metadata cache by @LuQQiu in #8013
• perf: avoid cloning every fragment descriptor in take and FTS planning by @LuQQiu in #8102

Other Changes

• refactor(fts): separate document identity from row addresses by @Xuanwo in #7863
• refactor(mem-wal): rename flushed MemTable/generation to SSTable by @jackye1995 in #7943
• refactor: rename MemWAL compaction progress by @jackye1995 in #7957
• refactor(index): remove dead segment commit branch by @ddupg in #7971
• refactor: make v1 the canonical legacy file owner by @Xuanwo in #8020
• refactor: make encoding mechanisms version-free by @Xuanwo in #8021
• refactor: add exact current-format writers by @Xuanwo in #8022
• refactor(encoding): pass mini-block compression context by @Xuanwo in #8038

Full Changelog: release-root/10.0.0-beta.N...v10.0.0-rc.1