v11.0.0-rc.1

What's Changed

Breaking Changes 🛠

• refactor!: compose exact current-format readers by @Xuanwo in #8024
• refactor: centralize exact dataset write policy by @Xuanwo in #8025
• refactor: move file-version selectors to lance-file by @Xuanwo in #8026
• refactor: centralize dataset version policies by @Xuanwo in #8027
• refactor: make index file versions exact by @Xuanwo in #8028
• feat(mem_wal): return the sealed generation from force_seal_active by @hamersaw in #8051
• feat(mem_wal): expose whether an index type can be maintained by @hamersaw in #8095
• fix(cache): deduplicate deep size accounting by @app/lance-gatekeeper in #8159
• fix(encoding): return errors for malformed variable-width offsets by @app/lance-gatekeeper in #8172
• fix(index): align HNSW construction with reference by @app/lance-gatekeeper in #8188
• fix(core)!: stop reusing fragment ids across an overwrite by @wjones127 in #8206
• feat(compaction)!: support max_source_rows and max_source_bytes limits by @zhangyue19921010 in #8235
• fix(dataset)!: conflict a NOT NULL tightening with concurrent writes by @wkalt in #8347
• feat(mem_wal): validate maintained indexes against the writer's rules by @hamersaw in #8360
• fix(datafusion): use caller session for filter planning by @app/lance-gatefixer in #8407
• feat(format): add IndexMetadata.covering_fields for covering indexes by @vivek-bharathan in #8535

Critical Fixes ‼️

• fix(encoding): bounds-check the length prefix in VariableFullZipDecoder::unzip by @professor-moody in #8138
• fix(encoding): reject out-of-bounds variable-width offsets on decode by @Xuanwo in #8144

New Features 🎉

• feat(java): expose ArrowArrayStream export on LanceScanner by @sezruby in #7259
• feat(scanner): add external row-address mask prefilter by @JulianYG in #7288
• feat(cache): support register and switch pluggable cache backend by @yanghua in #7683
• feat: per value support for fixed-length packed structs by @morales-t-netflix in #7714
• feat(index): add FTS document granularity by @Xuanwo in #7788
• feat(python): expose file size from LanceFileWriter by @FANNG1 in #7876
• perf(dataset): spill large inline transaction out of manifest by @zhangyue19921010 in #7881
• feat(index): add zone map support for all data types by @westonpace in #8017
• feat: support shared session for fragment API by @zhangyue19921010 in #8034
• feat(index): stop HNSW greedy descent before level 0 by @u70b3 in #8035
• feat(java): add OpenTelemetry metrics bridge by @yanghua in #8064
• feat(java): expose fragment statics through java blocking dataset by @zhangyue19921010 in #8072
• feat(sql): add blob handling to query builder by @dentiny in #8087
• feat(java): expose index build progress callbacks by @hfutatzhanghb in #8090
• feat(fts): add composable scorer core for multimatch by @BubbleCal in #8092
• feat(fts): compose boolean and phrase scorers by @BubbleCal in #8093
• feat(fts): compose boosted compound scorers by @BubbleCal in #8094
• feat(object-store): add explicit IRSA and ECS credential storage options by @westonpace in #8103
• feat(python): expose max_source_fragments in compact_files by @zhangyue19921010 in #8116
• feat(fts): export CompoundQueryExec and support an injected base scorer by @LuQQiu in #8131
• feat(goosefs): implement safe commit with ConditionalPutCommitHandler by @XuQianJin-Stars in #8134
• feat(mem_wal): expose frozen memtable and backpressure stats by @hamersaw in #8241
• feat(rowids): validate stable row ids in Dataset::validate by @wjones127 in #8258
• feat(mem-wal): record index catch-up positions and withdraw them on index change by @xuanyu-z in #8263
• feat(java): add scanner performance and memory options by @everySympathy in #8288
• feat(dataset): per-fragment column writes that survive compaction by @wkalt in #8313
• feat(python): expose BlobFile.read_ranges by @keunhong in #8319
• feat(mem_wal): support delete against non-nullable base columns by @hamersaw in #8352
• feat(java): expose index segment sizes by @majin1102 in #8355
• feat(python): add RowIdSequence for constructing row id metadata by @wjones127 in #8356
• feat(fts): show tokenized queries in analyze plan by @BubbleCal in #8414
• feat(python): expose FTS query tokenization by @BubbleCal in #8415
• feat(merge_insert): add write_mode to select how merged rows are written by @LuciferYang in #8423
• feat(python): expose LanceFragment.validate by @keunhong in #8428
• feat(java): support cache backend register and switch by @yanghua in #8446
• feat(python): expose updated_fragment_offsets in the Update operation binding by @pengw0048 in #8447
• feat(java): expose manifest location metadata by @everySympathy in #8450
• feat(java): expose manifest writer version by @everySympathy in #8451
• feat(java): add efficient dataset version count by @everySympathy in #8453
• feat(fts): expose pure SHOULD MAXSCORE metrics by @BubbleCal in #8475
• feat(mem-wal): derive index catch-up from the version a commit read by @xuanyu-z in #8481
• feat: add Dataset::migrate_to_stable_row_ids migration method by @westonpace in #8521
• feat(dataset): add lightweight dataset version references by @dentiny in #8523
• feat(deps): prune unused dependencies by @dentiny in #8524
• feat(python): expose frozen memtable metrics in memtable_stats by @hamersaw in #8526
• feat(compaction): support excluding fragments from planning by @everySympathy in #8532
• feat: report the row ids deleted between two versions by @wkalt in #8589
• feat(mem_wal): observe flush latency through the metrics facade by @hamersaw in #8625
• feat(sql): expose batch size to sql builder by @dentiny in #8648
• feat(fts): add cross-column scorer composition by @BubbleCal in #8685

Bug Fixes 🐛

• fix(java): expose updatedFragmentOffsets on Update operation for RewriteColumns by @jerryjch in #6748
• fix(index): drop stale per-segment rows in vector search after in-place column update by @wombatu-kun in #7371
• fix: fall back to AVX2 f16 kernels on AVX-512 CPUs by @paramt in #7428
• fix: validate merge schema field bindings by @wkalt in #7703
• fix: recognize own commit on conflict and avoid deleting committed artifacts by @jackye1995 in #7722
• fix(bloomfilter): bound read/write to prevent BinaryArray i32 offset overflow by @zhangyue19921010 in #7753
• fix(ngram): recheck contains() with sub-trigram needle instead of dropping rows by @Ar-maan05 in #7845
• fix(index): don't report a ZoneMap value range that an overlay may have escaped by @wjones127 in #7924
• fix(datagen): preserve timestamp time zones by @xixiGoodLuck in #7946
• fix(index): align scalar index field type support by @ddupg in #7966
• fix(fts): re-evaluate overlay-stale rows without segment scans by @Xuanwo in #7975
• fix(lance-io): request vended credentials explicitly in namespace provider by @rpgreen in #7978
• fix(io): publish object store metrics for local reads and writes by @wjones127 in #7994
• fix(jni): align CommitBuilder.storageFormat() parsing with LanceFileVersion::from_str by @ivscheianu in #8063
• fix(core): key row id sequence cache on fragment contents by @wjones127 in #8078
• fix(index): keep address-domain indices off replaced fragments by @wjones127 in #8084
• fix(index): use zone map index for IS NOT NULL predicates by @westonpace in #8088
• fix(datagen): make default temporal ranges deterministic by @westonpace in #8121
• fix(index): compare Hamming IVF centroids during merge by @jackye1995 in #8132
• fix: preserve row address filters with stable row ids by @app/lance-gatekeeper in #8145
• fix(index): resolve nested vector index field paths by @justinrmiller in #8148
• fix: preserve schema metadata precedence on merge by @app/lance-gatekeeper in #8160
• fix(python): preserve commit handler exceptions by @app/lance-gatekeeper in #8162
• fix: type scoring columns in projection expressions by @app/lance-gatekeeper in #8163
• fix(io): return an error for short reads by @app/lance-gatekeeper in #8164
• fix(index): preserve row addresses in JSON index training by @app/lance-gatekeeper in #8165
• fix(index): replace same-name index segments across types by @app/lance-gatekeeper in #8166
• fix(index): count HNSW entry point at every level by @BubbleCal in #8167
• fix: reject missing merge insert update addresses by @app/lance-gatekeeper in #8168
• fix: bound commit conflict retry backoff by @app/lance-gatekeeper in #8169
• fix: dedupe unmatched merge insert source rows by @app/lance-gatekeeper in #8171
• fix(encoding): reject invalid miniblock row counts by @app/lance-gatekeeper in #8173
• fix(storage): preserve multipart part identity across retries by @app/lance-gatekeeper in #8174
• fix(merge_insert): deduplicate indexed candidates by @app/lance-gatekeeper in #8176
• fix(ci): treat missing merge_insert base tag as not generated by @Xuanwo in #8177
• fix: preserve large unsigned SQL filter literals by @app/lance-gatekeeper in #8178
• fix: avoid Greek stemmer UTF-8 panic by @app/lance-gatekeeper in #8183
• fix(java): type scanner operation failures by @app/lance-gatekeeper in #8184
• fix(fts): align list post-filter with indexed semantics by @Ecthlion in #8185
• fix(index): push down deeply nested scalar filters by @app/lance-gatekeeper in #8186
• fix: respect external base layout in tracked files by @app/lance-gatekeeper in #8187
• fix: align byte-sized batches across data files by @app/lance-gatekeeper in #8189
• fix(ci): enlarge mem pool only around merge_insert index build by @Xuanwo in #8191
• fix(index): initialize indices on nested fields by @app/lance-gatekeeper in #8192
• fix: deduplicate MemWAL IN point keys by @app/lance-gatekeeper in #8197
• fix: return error for divergent MemWAL schemas by @app/lance-gatekeeper in #8198
• fix: prevent update false negatives for address-domain indices by @app/lance-gatekeeper in #8203
• fix: retain empty vector index definitions in retain_relevant_indices by @wkalt in #8208
• fix(fts): cover all segments for fragment scans by @ddupg in #8211
• fix: tolerate concurrent manifest cleanup by @app/lance-gatefixer in #8213
• fix(index): accept "RQ" when parsing QuantizationType by @jackylee-ch in #8214
• fix: avoid v2.2 panic for full dictionaries by @app/lance-gatefixer in #8219
• fix(encoding): reject mismatched dictionary index widths by @app/lance-gatefixer in #8220
• fix(index): remap bloom filter search results by @app/lance-gatefixer in #8223
• fix: validate GBO descriptors before metadata I/O by @app/lance-gatefixer in #8227
• fix(docs): stop make help from running the website build by @jackylee-ch in #8230
• fix: restore main CI after exact-format reader refactor by @Xuanwo in #8231
• fix(namespace): self-heal DirectoryNamespace read manifest cache by @rpgreen in #8233
• fix(table): reject corrupt row ID metadata by @app/lance-gatefixer in #8254
• fix: compute row offsets for unsorted addresses by @app/lance-gatefixer in #8255
• fix(lance): propagate deletion vector read errors by @app/lance-gatefixer in #8256
• fix: include highest fragment in legacy prefilter masks by @app/lance-gatefixer in #8257
• fix(rowids): validate serialized row ID metadata by @app/lance-gatefixer in #8259
• fix: remove duplicate LanceFileVersion import in lance-file tests by @wjones127 in #8261
• fix(mem-wal): reject stale SSTable compaction progress by @xuanyu-z in #8262
• fix: manifest file walkers miss data-overlay files by @LuciferYang in #8267
• fix(ci): budget merge insert benchmark memory by @app/lance-gatefixer in #8271
• fix: validate top-level schema names in linear time by @app/lance-gatefixer in #8274
• fix(geo): preserve RTree bounds for empty batches by @app/lance-gatefixer in #8287
• fix(fts): sum all boolean must scores by @BubbleCal in #8292
• fix(index): handle partial FTS prewarm under cache pressure by @ChilePiquin in #8298
• fix: add fallback case for SIMD_SUPPORT by @dtolnay in #8303
• fix: clear stale manifest index section by @app/lance-gatefixer in #8308
• fix(test): pass stable_row_ids to create_text_dataset call site by @wjones127 in #8311
• fix(fts): normalize token cursor before incremental merge by @app/lance-gatefixer in #8312
• fix: remove empty index directories during cleanup by @app/lance-gatefixer in #8322
• fix(cleanup): reclaim superseded index files promptly by @app/lance-gatefixer in #8323
• fix(dataset): advance row lineage on DataReplacement by @wkalt in #8337
• fix(index): drop stale rows when merging vector index segments by @wombatu-kun in #8342
• fix: support blob v2 in fragment update_columns by @app/lance-gatefixer in #8344
• fix(index): release IVF late-search barrier for skipped deltas by @app/lance-gatefixer in #8350
• fix(dataset): prevent deleting branches referenced by tags by @dentiny in #8365
• fix(java): correct CompactionMode type across the JNI boundary by @zhangyue19921010 in #8367
• fix(rowids): wrap Vec as InlineRowIds in validate tests by @Xuanwo in #8368
• fix: prevent unsafe COS manifest commits by @app/lance-gatefixer in #8369
• fix(encoding): return errors for unsupported v2.0 field types by @app/lance-gatefixer in #8370
• fix(python): avoid reopening dataset when pickling fragments by @app/lance-gatefixer in #8373
• fix(build): support pre-Haswell x86_64 hosts by @app/lance-gatefixer in #8377
• fix(io): preserve Windows UNC share roots by @app/lance-gatefixer in #8378
• fix: normalize PyArrow Substrait list names by @app/lance-gatefixer in #8381
• fix(encoding): validate variable-width Arrow offsets by @app/lance-gatefixer in #8382
• fix: return error for invalid list page encoding by @app/lance-gatefixer in #8384
• fix: support bitwise shift SQL expressions by @app/lance-gatefixer in #8388
• fix(io): support GCS workload identity credentials by @app/lance-gatefixer in #8389
• fix(io): respect custom S3 endpoints from AWS profiles by @app/lance-gatefixer in #8390
• fix(index): correct cosine RabitQ query factor by @app/lance-gatefixer in #8393
• fix: allow empty V1 writes with non-nullable columns by @app/lance-gatefixer in #8394
• fix(index): search legacy truncated PQ indexes by @app/lance-gatefixer in #8396
• fix: avoid deadlock when opening scan fragments by @app/lance-gatefixer in #8397
• fix: filter stale physical row ids in take by @app/lance-gatefixer in #8401
• fix(index): convert JSON values during index updates by @app/lance-gatefixer in #8404
• fix(storage): report AWS credential failures as I/O by @app/lance-gatefixer in #8406
• fix(java): carry alterColumns cast type across FFI via C Data Interface by @puchengy in #8417
• fix(docs): align integrations nav with website assembly script by @jackylee-ch in #8419
• fix(linalg): drop casts that make clippy::use_self fail off the CI baseline by @LuciferYang in #8424
• fix: report JSON index statistics by @Xuanwo in #8437
• fix(mem-wal): carry fragments through UpdateMemWalState by @xuanyu-z in #8438
• fix: compile the lance tests after the compacted_sstables rename by @wombatu-kun in #8440
• fix(fts): include must-not clauses in query planning by @ddupg in #8443
• fix(tag): make tag creation atomic by @dentiny in #8459
• fix(doc): fix documentation on create_branch behavior by @dentiny in #8462
• fix: reject zero cleanup retention by @app/lance-gatefixer in #8467
• fix(fts): make score sum upper bounds conservative by @BubbleCal in #8473
• fix(indexing): write FTS metadata even without partitions by @cmccabe in #8482
• fix: validate substrait filters that need no field pruning by @1fanwang in #8484
• fix(ci): resolve stable file versions in slow tests by @Xuanwo in #8485
• fix(index): align HNSW flat search distance bounds by @app/lance-gatefixer in #8496
• fix(commit): make object storage authoritative for external manifests by @Xuanwo in #8499
• fix: handle reordered sources in indexed merge insert by @app/lance-gatefixer in #8509
• fix(fts): ignore posting format version across mixed FTS segments by @Xuanwo in #8525
• fix(scan): preserve filtered read schema metadata by @BubbleCal in #8536
• fix(mem-wal): install conflicts with concurrent schema-changing merge by @wkalt in #8539
• fix(linalg): validate multivec_distance input instead of panicking by @LuciferYang in #8542
• fix(build): fix build in main by @dentiny in #8543
• fix(geo): skip empty geometries in RTree indexes by @app/lance-gatefixer in #8545
• fix: raise typed CommitConflictError instead of bare OSError by @lichuang in #8563
• fix(java): hold Dataset read lock when passing Dataset into native calls by @zhangyue19921010 in #8575
• fix(linalg): use a lane-wise intrinsic for i32x8 multiplication on x86 by @LuciferYang in #8579
• fix(fsst): reject corrupt symbol tables and offsets on decode by @Xuanwo in #8588
• fix: make MemWAL HNSW snapshots publication-consistent by @Xuanwo in #8591
• fix(linalg): validate distance input lengths by @Xuanwo in #8594
• fix(python): return a shared background executor by @Xuanwo in #8596
• fix(encoding): always validate Arrow conversions by @Xuanwo in #8597
• fix(index): initialize BQ distance buffers by @Xuanwo in #8598
• fix(encoding): initialize bitpacking output buffers by @Xuanwo in #8599
• fix(select): make row address iterators safe by @Xuanwo in #8600
• fix(encoding): reject bytepacking overflow by @Xuanwo in #8601
• fix(index): make IVF buffer setter safe by @Xuanwo in #8602
• fix(java): narrow JNI byte slice casts by @Xuanwo in #8603
• fix(dataset): isolate wrapped additional base stores by @BubbleCal in #8609
• fix(encoding): reject zero fixed-size-list dimension in repdef decimation by @westonpace in #8618
• fix(index): avoid fragment reuse cache deadlock by @app/lance-gatefixer in #8620
• fix(table): restore migration_next_row_id on ManifestBuildConfig by @wjones127 in #8629
• fix(java): preserve sessions on dataset checkout by @app/lance-gatefixer in #8650
• fix(io): normalize OpenDAL listed paths by @app/lance-gatefixer in #8654
• fix(fts): preserve exact wand score bounds by @BubbleCal in #8666
• fix(fts): chunk posting reads during segment merge by @BubbleCal in #8668
• fix: encode JSON string expressions during updates by @app/lance-gatefixer in #8669
• fix: preserve the row id high-water mark across restore by @wkalt in #8671
• fix(io): add upload context to object store write failures by @jackye1995 in #8682
• fix(fts): derive empty segment codec from format by @BubbleCal in #8687
• fix(linalg): gate AMX FP16 import by target by @BubbleCal in #8690
• fix(fts): preserve exact top-level multimatch ties by @BubbleCal in #8702
• fix(io): stop serializing multipart uploads behind batch encoding by @jackye1995 in #8706
• fix: conflict schema metadata updates with merges by @Xuanwo in #8707
• fix: delete data files when cleanup retains an older tag by @XuQianJin-Stars in #8708

Documentation 📚

• docs(format): correct vector index schemas and fix broken examples by @jackylee-ch in #8215
• docs: convert ignore doctests to compiling examples by @jackylee-ch in #8216
• docs: require duplicate PR checks by @Xuanwo in #8265
• docs(python): show how to specify row ids in update transactions by @wjones127 in #8357
• docs: explain remote scan concurrency controls by @app/lance-gatefixer in #8387
• docs(format): add FM-Index to scalar index navigation by @jackylee-ch in #8395
• docs: update crate READMEs to lance-format org by @jackylee-ch in #8398
• docs: use indices instead of indexes in guide and quickstart by @jackylee-ch in #8399
• docs: fix broken observability and tokenizer links by @Xuanwo in #8413
• docs(python): fix docstring parameter names that do not exist by @jackylee-ch in #8420
• docs: avoid mint wording in stable row id docs by @Xuanwo in #8425
• docs: fix stale links, a broken path, and typos in contributor and quickstart docs by @jackylee-ch in #8432
• docs: trim redundant agent guidance by @Xuanwo in #8439
• docs: update Jeremy Leibs' affiliation by @Xuanwo in #8683

Performance Improvements 🚀

• perf: serialize updated_fragment_offsets as RoaringBitmap bytes in proto by @jerryjch in #7432
• perf(index): use dot distance for cosine in FlatFloatStorage by @clearlove10-c in #7777
• perf(index): sample PQ training data before sub-vector copies by @u70b3 in #7930
• perf(encoding): preallocate fixed-width decode buffers exactly by @everySympathy in #8091
• perf: skip index metadata deep clone and compare when no vector inference is needed by @LuQQiu in #8107
• perf(fts): pre-load doc lengths in parallel on cold deferred search path by @westonpace in #8119
• perf(index): skip the write-only distance column when loading HNSW partitions by @wombatu-kun in #8136
• perf(fts): bound compound score-floor tie buffering by @BubbleCal in #8161
• perf(transaction): avoid O(n*m) fragment compares in build_manifest for Update/Delete by @zhangyue19921010 in #8210
• perf(fts): order conjunction approximations by cost by @BubbleCal in #8299
• perf(fts): short-circuit conjunction confirmations by @BubbleCal in #8354
• perf(fts): lazily score optional boolean clauses by @BubbleCal in #8448
• perf(fts): add MAXSCORE for pure SHOULD queries by @BubbleCal in #8474
• perf: push down fragment-scoped filtered counts by @BubbleCal in #8515
• perf(deps): gate lance-datafusion datagen tooling by @dentiny in #8527
• perf(dataset): cache object stores for additional base paths by @LuQQiu in #8530
• perf(rowids): keep the scan position when reading a segment in order by @pengw0048 in #8534
• perf(index): route fp16 IVF partition assignment through an AMX-FP16 GEMM by @xtangxtang in #8540
• perf(index): skip unnecessary null tracking for BTree and Bitmap indexes by @everySympathy in #8577
• perf(filtered-read): prune non-candidate fragment metadata by @BubbleCal in #8587
• perf(ci): parallelize Python tests and shorten the macOS Rust job by @wjones127 in #8628
• perf(dataset): binary search the manifest in get_fragment by @zhangyue19921010 in #8636
• perf(fts): enable bounded cross-column search by @BubbleCal in #8689
• perf(fts): add WAND exactness certificate by @BubbleCal in #8703

Other Changes

• refactor: activate exact current-format writers by @Xuanwo in #8023
• refactor(table): move transaction helpers down into lance-table by @wjones127 in #8053
• refactor(table): move transaction.rs into lance-table by @wjones127 in #8054
• refactor(table): split transaction.rs into a module tree by @wjones127 in #8056
• refactor: name blob v2 layouts by @Xuanwo in #8266
• refactor: initialize constant Arrow schemas once via LazyLock by @MannXo in #8284
• refactor(fts): split inverted index module by @app/lance-gatefixer in #8309
• refactor: move row id metadata into submodule by @cswpy in #8433
• refactor: extract the shared manifest walk behind tracked_files by @LuciferYang in #8449
• fix(core): keep assumptions checked in release builds by @Xuanwo in #8592
• fix(linalg): validate slice lengths before SIMD loads by @Xuanwo in #8593
• fix: serialize BatchStore writers by @Xuanwo in #8595
• fix(deps): bump h2 to 0.4.16 for RUSTSEC-2026-0258 by @Xuanwo in #8613
• refactor(dataset): rename FileFragment::write_column by @Xuanwo in #8622
• refactor(fts): add exact posting load policies by @BubbleCal in #8667
• refactor(mem-wal): remove the index-catchup feature bit by @xuanyu-z in #8680

Full Changelog: release-root/11.0.0-beta.N...v11.0.0-rc.1