v8.0.0-rc.1

What's Changed

Breaking Changes 🛠

• feat!: migrate bitmap to index segment based by @Xuanwo in #6869
• fix(python)!: derive index type from details instead of opening the index by @wjones127 in #6903
• refactor!: remove index segment builder by @Xuanwo in #6997
• refactor(index)!: move distributed BTree build to segmented index framework by @zhangyue19921010 in #7013
• feat!: return write summaries from file writers by @Xuanwo in #7096
• perf!: avoid listing index files after writes by @Xuanwo in #7129
• fix(dataset)!: fail-fast casting for columns with attached indices by @WenDing-Y in #7158
• feat(vector)!: add approx mode for RaBitQ search by @BubbleCal in #7179
• perf(vector)!: add dedicated SIMD kernels for RaBitQ ex-code reranking by @BubbleCal in #7205

Critical Fixes ‼️

• fix: merge_insert silently drops matches when a leading payload column is all-null by @Ar-maan05 in #7251

New Features 🎉

• feat: expose tracked_files and all_files on LanceDataset by @wjones127 in #6011
• feat: add commit timeout to CommitBuilder by @wjones127 in #6773
• feat: add segmented BTree index merge_segments support by @zhangyue19921010 in #6889
• feat(index): add streaming ivf kmeans training by @BubbleCal in #6913
• feat: use indexes to accelerate filtered count_rows by @westonpace in #6916
• feat(java): allow schema override for fragment writes by @beinan in #6919
• feat: make ICU the default FTS tokenizer by @Xuanwo in #6968
• feat: support Utf8View and BinaryView in encoding and filter coercion by @xuanyu-z in #6985
• feat(python): add shared RaBitQ rotation for distributed IVF_RQ builds by @gstamatakis95 in #7014
• feat(blobv2): support all BlobKind types in blob v2 compact_files by @yyzhao2025 in #7017
• feat: add TOS object store support via OpenDAL by @ddupg in #7019
• feat(index): implement FM-Index scalar index for exact substring search by @beinan in #7026
• feat(python): expose external blob mode and outside-base option for fragments by @plotor in #7028
• feat(lance-io): add GooseFS object store provider by @XuQianJin-Stars in #7034
• feat(index): support multi-bit IVF_RQ storage by @BubbleCal in #7038
• feat: expose getters for ScalarIndexExec by @LuQQiu in #7039
• feat: expose methods and getters of scalar index for possible distributed execution by @LuQQiu in #7045
• feat(mem_wal): configurable HNSW build params for MemTable writers by @touch-of-grey in #7054
• feat: dedup FTS results across LSM tiers in LsmFtsSearchPlanner by @hamersaw in #7066
• feat(index): support raw-query ivf rq search by @BubbleCal in #7078
• feat: add EnforceDistribution to physical optimizer by @wjones127 in #7086
• feat(java): add missing scanner and merge insert params to align with Python/Rust by @WenDing-Y in #7100
• feat: populate enriched IndexContent fields in dir namespace ListTableIndices by @wjones127 in #7109
• feat(index): support configurable multi-segment FM-Index builds by @beinan in #7123
• feat: support merging zonemap index segments by @Xuanwo in #7128
• feat(index): accelerate regex and infix LIKE with the ngram index by @wombatu-kun in #7139
• feat(rust): add cleanup explain API by @yanghua in #7147
• feat: stabilize cache codec with a versioned envelope by @wjones127 in #7163
• feat(lance-select): expose selected rows accessor on NullableRowAddrSet by @LuQQiu in #7164
• feat: branch-aware table version ops in directory and rest namespaces by @brendanclement in #7166
• feat(python): expose segment FTS build through create_index_uncommitted by @ddupg in #7170
• feat(python): expose zonemap segment builds by @everySympathy in #7177
• feat(dir-catalog): add reader/writer feature flags to __manifest by @jackye1995 in #7191
• feat(index): expose per-query I/O metrics on ANN operators by @wombatu-kun in #7204
• feat(mem-wal): snapshot-consistent as-of cut for fresh-tier membership by @hamersaw in #7215
• feat: expose io_buffer_size in CompactionOptions by @aimanmalib in #7226
• fix(python): expose stable row id property in stub by @BubbleCal in #7249
• feat: bump lance-namespace-reqwest-client to 0.8.6 (source_task_size) by @justinrmiller in #7254
• feat: configure blob inline threshold per column by @Xuanwo in #7269
• feat(mem_wal): warm flushed generations into shared caches before query by @hamersaw in #7284
• feat(java): expose RTree scalar index type to Java by @zhangyue19921010 in #7291
• feat: expose session cache key inventory by @jackye1995 in #7298
• feat: support mixed-language FTS stop words by @Xuanwo in #7324
• feat(scalar): expose LogicalScalarIndex::try_new and load_named_scalar_segments by @LuQQiu in #7339
• feat: allow tuning miniblock value chunks to 32k by @Xuanwo in #7356

Bug Fixes 🐛

• fix(encoding): plan sparse structural miniblock pages by @Xuanwo in #6787
• fix(java): resolve JNI classloader bug on dispatcher thread in Spark by @sezruby in #6946
• fix: preserve zero-length buffers in binary copy compaction by @zhangyue19921010 in #6992
• fix(storage): retry throttled fts metadata listing by @BubbleCal in #6994
• fix: support multivector IVF centroids in segment builds by @ddupg in #6995
• fix(python): validate BFloat16.from_bytes length by @ddupg in #6998
• fix(test): tolerate boundary-tie in IVF distance_range assertions by @xuanyu-z in #6999
• fix(datafusion): coerce filter literals for dictionary-encoded columns by @valkum in #7003
• fix: stream object copies larger than cloud's CopyObject limit by @vivek-bharathan in #7004
• fix: restore simple FTS tokenizer default by @Xuanwo in #7006
• fix: expose Hugging Face download mode by @Xuanwo in #7022
• fix(python): clamp target partition sizing by @ddupg in #7036
• fix(fts): handle empty query tokens in flat full-text search by @vivek-bharathan in #7046
• fix(lance-index): fix some flaky tests by @XuQianJin-Stars in #7052
• fix: specify roaring's patch version by @HuaHuaY in #7056
• fix(filtered-read): record IO metrics even when filter matches no rows by @westonpace in #7057
• fix: handle nested JSON conversion recursively by @Xuanwo in #7060
• fix: retry S3 multipart request timeouts by @Xuanwo in #7061
• fix(fts): size cached posting lists by referenced slice by @vivek-bharathan in #7068
• fix: cap exec-node parallelism to DataFusion target_partitions by @wjones127 in #7087
• fix(mem-wal): fence predecessor with a WAL sentinel on claim by @hamersaw in #7110
• fix(fts): reset TokenSet next_id and total_length after remap by @vivek-bharathan in #7115
• fix(linalg): reduce cosine bench TOTAL to avoid FixedSizeBinaryArray i32 overflow by @westonpace in #7116
• fix: compile AVX-512 dist table for target CPU by @Xuanwo in #7121
• fix(namespace): tolerate reqwest client model additions by @Xuanwo in #7124
• fix(python): avoid pandas requirement for add_columns UDFs by @prrao87 in #7131
• fix(mem-wal): surface fenced flush to durability waiters instead of hanging by @hamersaw in #7132
• fix: remove empty index segment when committing real segments by @wkalt in #7141
• fix(dataset): resolve Blob v2 external URIs and robustly clean failed writes in add_columns by @yyzhao2025 in #7152
• fix(encoding): improve dict sampling for sorted strings by @Xuanwo in #7159
• fix(index): support HNSW segment merge by @Xuanwo in #7178
• fix: search all unindexed fragments in FTS flat match by @wkalt in #7188
• fix(fts): recompute next_id when loading FST token set by @wkalt in #7200
• fix(python): python binding of Compaction.commit by @gstamatakis95 in #7210
• fix(namespace): tolerate new IndexContent fields from reqwest client by @wkalt in #7212
• fix(rust): fixed IVF_RQ indexes ignoring the frag-reuse index by @gstamatakis95 in #7217
• fix: branch support on object stores, python branch ops, s3/ddb integration tests by @brendanclement in #7219
• fix: record TakeExec output and I/O metrics by @skyshineb in #7228
• fix: handle empty batch from deletion in add_columns_from_stream by @zhangyang0418 in #7233
• perf(dir-catalog): avoid unchanged manifest reloads by @jackye1995 in #7234
• fix: return error instead of panicking on zero-dimension fixed-size-list columns by @DanielMao1 in #7247
• fix: avoid versioned describe table for namespace opens by @FANNG1 in #7250
• fix(mem_wal): push LIMIT/OFFSET down to fresh-tier scan sources by @hamersaw in #7256
• fix: make FTS finalization idempotent by @jackye1995 in #7272
• fix(index): share FTS position stream offsets by @BubbleCal in #7275
• fix: reject nested legacy blobs in v2.2 by @Xuanwo in #7278
• fix: support blob v2 in nested structs by @Xuanwo in #7281
• fix: remap index data when its fragment bitmap was already coverage-remapped by @xuanyu-z in #7286
• fix(index): refresh PQ storage row ids after fragment-reuse remap by @xuanyu-z in #7315
• fix(fts): keep inverted index correct under deletions in the fragment-reuse window by @xuanyu-z in #7325
• fix: stream FM index partition builds by @jackye1995 in #7354

Documentation 📚

• docs: fix incorrect method call in create_scalar_index example by @wayneadams in #5399
• docs: add new maintainers and update affiliations by @prrao87 in #6990
• docs: update Yang Jie ecosystem roles by @LuciferYang in #7055
• docs: fix broken pre-release install commands by @prrao87 in #7088
• docs: correct alter_columns nullable docstring by @Alowator in #7095
• docs: update ecosystem role for maintainer by @prrao87 in #7102
• docs: clarify distributed vector model scopes by @Xuanwo in #7148
• docs(object_store): add Tencent Cloud COS and GooseFS configuration by @XuQianJin-Stars in #7151

Performance Improvements 🚀

• perf: make HNSW cheaper to load by @wombatu-kun in #6798
• perf(merge_insert): probe every indexed key column for composite joins by @ytyky in #6878
• perf(index): reduce describe indices fragment scans by @hfutatzhanghb in #6958
• perf(index): reduce HNSW build distance overhead by @BubbleCal in #6978
• perf(search): reuse IVF_RQ query scratch by @BubbleCal in #6982
• perf(fts): defer DocSet load until wand actually needs it by @wkalt in #6983
• perf(mem_wal): match hnswlib throughput via runtime AVX-512 f32 distance by @touch-of-grey in #7009
• perf(mem_wal): faster MemTable point lookups by @touch-of-grey in #7011
• perf(index): cache wand head posting doc id by @BubbleCal in #7023
• perf(mem_wal): close the FTS MemTable performance gap with Lucene by @touch-of-grey in #7029
• perf(file): serve global-buffer reads from the captured tail bytes by @wjones127 in #7042
• perf(fts): group small posting lists into shared cache entries by @wjones127 in #7043
• perf(fts): share a top-k threshold across partitions during WAND by @wkalt in #7062
• perf(index): avoid HEAD call when opening vector indexes by @gstamatakis95 in #7064
• perf(index): fix O(N log N) warm-cache regression in BitmapIndex by @westonpace in #7079
• perf: block-max pruning on the disjunctive WAND path by @wkalt in #7089
• perf(pq): wrap l2_targets in Arc to eliminate per-partition clones by @yuju-huang in #7093
• perf: reuse WAND lead buffer capacity in push_back_leads by @wkalt in #7153
• perf(index): cache parsed btree lookup state by @LuQQiu in #7161
• perf(dir-catalog): rewrite manifest mutations with copy-on-write by @jackye1995 in #7176
• perf(index): query BTree lookup batch directly by @wjones127 in #7186
• perf(fts): stream prewarm posting lists in chunks by @wkalt in #7208
• perf(vector): vectorize RaBitQ dist table quantization by @BubbleCal in #7241
• perf(vector): vectorize RaBitQ top-k lower-bound pruning scan by @BubbleCal in #7243
• perf(mem_wal): parallelize fresh-tier source planning and execution by @hamersaw in #7257
• perf(index): set ivf_rq target partition size to 4096 by @BubbleCal in #7273
• perf(fts): push prefilter through scalar index on flat FTS by @LuQQiu in #7283
• perf(scalar): compile IsIn predicate once across BTree pages by @JulianYG in #7287
• perf: batch fragment-reuse index remap into a single rebuild + commit by @xuanyu-z in #7317
• perf(fts): defer term frequency collection by @BubbleCal in #7357
• perf(index): reuse WAND doc info by @BubbleCal in #7358

Other Changes

• refactor: replace deepsize crate with custom DeepSizeOf trait for Arrow-aware memory accounting by @wjones127 in #6229
• refactor: move from 3-variant index search result to single generic variant by @westonpace in #6902
• refactor(io/exec): address #6799 review (drop helper, inline RAII timer) by @brendanclement in #6907
• refactor: clean up lance-index/lance-table boundary by @wjones127 in #6988
• refactor(index): scaffold ivfrq split code layout by @BubbleCal in #7021
• fix: advance kmeans redo random init rng by @ddupg in #7074
• refactor!(dir-catalog): remove table_version_storage_enabled and __manifest version storage by @jackye1995 in #7222

Full Changelog: release-root/8.0.0-beta.N...v8.0.0-rc.1