v11.0.0-rc.1

What's Changed

Breaking Changes 🛠

• refactor!: compose exact current-format readers by @Xuanwo in #8024
• refactor: centralize exact dataset write policy by @Xuanwo in #8025
• refactor: move file-version selectors to lance-file by @Xuanwo in #8026
• refactor: centralize dataset version policies by @Xuanwo in #8027
• refactor: make index file versions exact by @Xuanwo in #8028
• feat(mem_wal): return the sealed generation from force_seal_active by @hamersaw in #8051
• feat(mem_wal): expose whether an index type can be maintained by @hamersaw in #8095
• fix(cache): deduplicate deep size accounting by @app/lance-gatekeeper in #8159
• fix(encoding): return errors for malformed variable-width offsets by @app/lance-gatekeeper in #8172
• fix(index): align HNSW construction with reference by @app/lance-gatekeeper in #8188
• fix(core)!: stop reusing fragment ids across an overwrite by @wjones127 in #8206
• feat(compaction)!: support max_source_rows and max_source_bytes limits by @zhangyue19921010 in #8235
• fix(dataset)!: conflict a NOT NULL tightening with concurrent writes by @wkalt in #8347
• feat(mem_wal): validate maintained indexes against the writer's rules by @hamersaw in #8360
• fix(datafusion): use caller session for filter planning by @app/lance-gatefixer in #8407
• feat(format): add IndexMetadata.covering_fields for covering indexes by @vivek-bharathan in #8535

Critical Fixes ‼️

• fix(encoding): bounds-check the length prefix in VariableFullZipDecoder::unzip by @professor-moody in #8138
• fix(encoding): reject out-of-bounds variable-width offsets on decode by @Xuanwo in #8144

New Features 🎉

• feat(java): expose ArrowArrayStream export on LanceScanner by @sezruby in #7259
• feat(scanner): add external row-address mask prefilter by @JulianYG in #7288
• feat(cache): support register and switch pluggable cache backend by @yanghua in #7683
• feat: per value support for fixed-length packed structs by @morales-t-netflix in #7714
• feat(index): add FTS document granularity by @Xuanwo in #7788
• feat(python): expose file size from LanceFileWriter by @FANNG1 in #7876
• perf(dataset): spill large inline transaction out of manifest by @zhangyue19921010 in #7881
• feat(index): add zone map support for all data types by @westonpace in #8017
• feat: support shared session for fragment API by @zhangyue19921010 in #8034
• feat(index): stop HNSW greedy descent before level 0 by @u70b3 in #8035
• feat(java): add OpenTelemetry metrics bridge by @yanghua in #8064
• feat(java): expose fragment statics through java blocking dataset by @zhangyue19921010 in #8072
• feat(sql): add blob handling to query builder by @dentiny in #8087
• feat(java): expose index build progress callbacks by @hfutatzhanghb in #8090
• feat(fts): add composable scorer core for multimatch by @BubbleCal in #8092
• feat(fts): compose boolean and phrase scorers by @BubbleCal in #8093
• feat(fts): compose boosted compound scorers by @BubbleCal in #8094
• feat(object-store): add explicit IRSA and ECS credential storage options by @westonpace in #8103
• feat(python): expose max_source_fragments in compact_files by @zhangyue19921010 in #8116
• feat(fts): export CompoundQueryExec and support an injected base scorer by @LuQQiu in #8131
• feat(goosefs): implement safe commit with ConditionalPutCommitHandler by @XuQianJin-Stars in #8134
• feat(mem_wal): expose frozen memtable and backpressure stats by @hamersaw in #8241
• feat(rowids): validate stable row ids in Dataset::validate by @wjones127 in #8258
• feat(mem-wal): record index catch-up positions and withdraw them on index change by @xuanyu-z in #8263
• feat(java): add scanner performance and memory options by @everySympathy in #8288
• feat(dataset): per-fragment column writes that survive compaction by @wkalt in #8313
• feat(python): expose BlobFile.read_ranges by @keunhong in #8319
• feat(mem_wal): support delete against non-nullable base columns by @hamersaw in #8352
• feat(java): expose index segment sizes by @majin1102 in #8355
• feat(python): add RowIdSequence for constructing row id metadata by @wjones127 in #8356
• feat(fts): show tokenized queries in analyze plan by @BubbleCal in #8414
• feat(python): expose FTS query tokenization by @BubbleCal in #8415
• feat(merge_insert): add write_mode to select how merged rows are written by @LuciferYang in #8423
• feat(python): expose LanceFragment.validate by @keunhong in #8428
• feat(java): support cache backend register and switch by @yanghua in #8446
• feat(python): expose updated_fragment_offsets in the Update operation binding by @pengw0048 in #8447
• feat(java): expose manifest location metadata by @everySympathy in #8450
• feat(java): expose manifest writer version by @everySympathy in #8451
• feat(java): add efficient dataset version count by @everySympathy in #8453
• feat(fts): expose pure SHOULD MAXSCORE metrics by @BubbleCal in #8475
• feat(mem-wal): derive index catch-up from the version a commit read by @xuanyu-z in #8481
• feat: add Dataset::migrate_to_stable_row_ids migration method by @westonpace in #8521
• feat(dataset): add lightweight dataset version references by @dentiny in #8523
• feat(deps): prune unused dependencies by @dentiny in #8524
• feat(python): expose frozen memtable metrics in memtable_stats by @hamersaw in #8526
• feat(compaction): support excluding fragments from planning by @everySympathy in #8532
• feat: report the row ids deleted between two versions by @wkalt in #8589
• feat(mem_wal): observe flush latency through the metrics facade by @hamersaw in #8625
• feat(sql): expose batch size to sql builder by @dentiny in #8648
• feat(fts): add cross-column scorer composition by @BubbleCal in #8685

Bug Fixes 🐛

• fix(java): expose updatedFragmentOffsets on Update operation for RewriteColumns by @jerryjch in #6748
• fix(index): drop stale per-segment rows in vector search after in-place column update by @wombatu-kun in #7371
• fix: fall back to AVX2 f16 kernels on AVX-512 CPUs by @paramt in #7428
• fix: validate merge schema field bindings by @wkalt in #7703
• fix: recognize own commit on conflict and avoid deleting committed artifacts by @jackye1995 in #7722
• fix(bloomfilter): bound read/write to prevent BinaryArray i32 offset overflow by @zhangyue19921010 in #7753
• fix(ngram): recheck contains() with sub-trigram needle instead of dropping rows by @Ar-maan05 in #7845
• fix(index): don't report a ZoneMap value range that an overlay may have escaped by @wjones127 in #7924
• fix(datagen): preserve timestamp time zones by @xixiGoodLuck in #7946
• fix(index): align scalar index field type support by @ddupg in #796...

v11.0.0-beta.22

What's Changed

New Features 🎉

• feat(scanner): add external row-address mask prefilter by @JulianYG in #7288

Bug Fixes 🐛

• fix(fts): normalize token cursor before incremental merge by @app/lance-gatefixer in #8312
• fix(io): support GCS workload identity credentials by @app/lance-gatefixer in #8389
• fix(io): respect custom S3 endpoints from AWS profiles by @app/lance-gatefixer in #8390
• fix(index): search legacy truncated PQ indexes by @app/lance-gatefixer in #8396
• fix: avoid deadlock when opening scan fragments by @app/lance-gatefixer in #8397
• fix: filter stale physical row ids in take by @app/lance-gatefixer in #8401
• fix: validate substrait filters that need no field pruning by @1fanwang in #8484
• fix: conflict schema metadata updates with merges by @Xuanwo in #8707
• fix: delete data files when cleanup retains an older tag by @XuQianJin-Stars in #8708

Performance Improvements 🚀

• perf(index): sample PQ training data before sub-vector copies by @u70b3 in #7930

Full Changelog: v11.0.0-beta.21...v11.0.0-beta.22

v11.0.0-beta.21

What's Changed

New Features 🎉

• feat(java): expose ArrowArrayStream export on LanceScanner by @sezruby in #7259

Bug Fixes 🐛

• fix(io): stop serializing multipart uploads behind batch encoding by @jackye1995 in #8706

Full Changelog: v11.0.0-beta.20...v11.0.0-beta.21

v11.0.0-beta.20

What's Changed

Breaking Changes 🛠

• feat(format): add IndexMetadata.covering_fields for covering indexes by @vivek-bharathan in #8535

Full Changelog: v11.0.0-beta.19...v11.0.0-beta.20

v11.0.0-beta.19

What's Changed

New Features 🎉

• feat: report the row ids deleted between two versions by @wkalt in #8589

Full Changelog: v11.0.0-beta.18...v11.0.0-beta.19

v11.0.0-beta.18

What's Changed

Other Changes

• refactor(mem-wal): remove the index-catchup feature bit by @xuanyu-z in #8680

Full Changelog: v11.0.0-beta.17...v11.0.0-beta.18

v11.0.0-beta.17

What's Changed

New Features 🎉

• feat(fts): add cross-column scorer composition by @BubbleCal in #8685

Bug Fixes 🐛

• fix: preserve the row id high-water mark across restore by @wkalt in #8671
• fix(linalg): gate AMX FP16 import by target by @BubbleCal in #8690

Performance Improvements 🚀

• perf(fts): enable bounded cross-column search by @BubbleCal in #8689

Full Changelog: v11.0.0-beta.16...v11.0.0-beta.17

v11.0.0-beta.16

What's Changed

Bug Fixes 🐛

• fix(io): preserve Windows UNC share roots by @app/lance-gatefixer in #8378
• fix(encoding): validate variable-width Arrow offsets by @app/lance-gatefixer in #8382
• fix: support bitwise shift SQL expressions by @app/lance-gatefixer in #8388
• fix(fts): include must-not clauses in query planning by @ddupg in #8443
• fix(fsst): reject corrupt symbol tables and offsets on decode by @Xuanwo in #8588
• fix(java): preserve sessions on dataset checkout by @app/lance-gatefixer in #8650
• fix(fts): preserve exact wand score bounds by @BubbleCal in #8666
• fix(fts): chunk posting reads during segment merge by @BubbleCal in #8668
• fix: encode JSON string expressions during updates by @app/lance-gatefixer in #8669
• fix(io): add upload context to object store write failures by @jackye1995 in #8682
• fix(fts): derive empty segment codec from format by @BubbleCal in #8687

Documentation 📚

• docs: update Jeremy Leibs' affiliation by @Xuanwo in #8683

Performance Improvements 🚀

• perf(rowids): keep the scan position when reading a segment in order by @pengw0048 in #8534
• perf(index): route fp16 IVF partition assignment through an AMX-FP16 GEMM by @xtangxtang in #8540
• perf(ci): parallelize Python tests and shorten the macOS Rust job by @wjones127 in #8628
• perf(dataset): binary search the manifest in get_fragment by @zhangyue19921010 in #8636

Other Changes

• refactor(table): split transaction.rs into a module tree by @wjones127 in #8056
• refactor(fts): add exact posting load policies by @BubbleCal in #8667

Full Changelog: v11.0.0-beta.15...v11.0.0-beta.16

v11.0.0-beta.15

What's Changed

New Features 🎉

• feat: add Dataset::migrate_to_stable_row_ids migration method by @westonpace in #8521
• feat(mem_wal): observe flush latency through the metrics facade by @hamersaw in #8625

Bug Fixes 🐛

• fix: make MemWAL HNSW snapshots publication-consistent by @Xuanwo in #8591
• fix(linalg): validate distance input lengths by @Xuanwo in #8594
• fix(python): return a shared background executor by @Xuanwo in #8596
• fix(encoding): always validate Arrow conversions by @Xuanwo in #8597
• fix(index): initialize BQ distance buffers by @Xuanwo in #8598
• fix(encoding): initialize bitpacking output buffers by @Xuanwo in #8599
• fix(select): make row address iterators safe by @Xuanwo in #8600
• fix(encoding): reject bytepacking overflow by @Xuanwo in #8601
• fix(encoding): reject zero fixed-size-list dimension in repdef decimation by @westonpace in #8618
• fix(index): avoid fragment reuse cache deadlock by @app/lance-gatefixer in #8620
• fix(table): restore migration_next_row_id on ManifestBuildConfig by @wjones127 in #8629

Performance Improvements 🚀

• perf(index): skip unnecessary null tracking for BTree and Bitmap indexes by @everySympathy in #8577

Other Changes

• refactor(table): move transaction helpers down into lance-table by @wjones127 in #8053
• refactor(table): move transaction.rs into lance-table by @wjones127 in #8054
• fix(core): keep assumptions checked in release builds by @Xuanwo in #8592
• fix(linalg): validate slice lengths before SIMD loads by @Xuanwo in #8593
• fix: serialize BatchStore writers by @Xuanwo in #8595
• fix(deps): bump h2 to 0.4.16 for RUSTSEC-2026-0258 by @Xuanwo in #8613
• refactor(dataset): rename FileFragment::write_column by @Xuanwo in #8622

Full Changelog: v11.0.0-beta.14...v11.0.0-beta.15

v11.0.0-beta.14

What's Changed

New Features 🎉

• feat: per value support for fixed-length packed structs by @morales-t-netflix in #7714
• feat(python): expose frozen memtable metrics in memtable_stats by @hamersaw in #8526
• feat(compaction): support excluding fragments from planning by @everySympathy in #8532

Bug Fixes 🐛

• fix: validate merge schema field bindings by @wkalt in #7703
• fix(index): align scalar index field type support by @ddupg in #7966
• fix: handle reordered sources in indexed merge insert by @app/lance-gatefixer in #8509
• fix(index): make IVF buffer setter safe by @Xuanwo in #8602
• fix(java): narrow JNI byte slice casts by @Xuanwo in #8603
• fix(dataset): isolate wrapped additional base stores by @BubbleCal in #8609

Documentation 📚

• docs: avoid mint wording in stable row id docs by @Xuanwo in #8425

Performance Improvements 🚀

• perf(index): use dot distance for cosine in FlatFloatStorage by @clearlove10-c in #7777
• perf(filtered-read): prune non-candidate fragment metadata by @BubbleCal in #8587

Other Changes

• refactor: extract the shared manifest walk behind tracked_files by @LuciferYang in #8449

Full Changelog: v11.0.0-beta.13...v11.0.0-beta.14