v11.0.0-beta.1

What's Changed

Breaking Changes 🛠

• refactor!: compose exact current-format readers by @Xuanwo in #8024
• refactor: centralize exact dataset write policy by @Xuanwo in #8025
• refactor: move file-version selectors to lance-file by @Xuanwo in #8026
• fix(cache): deduplicate deep size accounting by @app/lance-gatekeeper in #8159
• fix(encoding): return errors for malformed variable-width offsets by @app/lance-gatekeeper in #8172
• fix(index): align HNSW construction with reference by @app/lance-gatekeeper in #8188
• fix(core)!: stop reusing fragment ids across an overwrite by @wjones127 in #8206

Critical Fixes ‼️

• fix(encoding): bounds-check the length prefix in VariableFullZipDecoder::unzip by @professor-moody in #8138
• fix(encoding): reject out-of-bounds variable-width offsets on decode by @Xuanwo in #8144

New Features 🎉

• feat(cache): support register and switch pluggable cache backend by @yanghua in #7683
• feat(index): add FTS document granularity by @Xuanwo in #7788
• feat(python): expose file size from LanceFileWriter by @FANNG1 in #7876
• feat(index): add zone map support for all data types by @westonpace in #8017
• feat(index): stop HNSW greedy descent before level 0 by @u70b3 in #8035
• feat(java): add OpenTelemetry metrics bridge by @yanghua in #8064
• feat(java): expose fragment statics through java blocking dataset by @zhangyue19921010 in #8072
• feat(sql): add blob handling to query builder by @dentiny in #8087
• feat(java): expose index build progress callbacks by @hfutatzhanghb in #8090
• feat(fts): add composable scorer core for multimatch by @BubbleCal in #8092
• feat(fts): compose boolean and phrase scorers by @BubbleCal in #8093
• feat(fts): compose boosted compound scorers by @BubbleCal in #8094
• feat(python): expose max_source_fragments in compact_files by @zhangyue19921010 in #8116
• feat(fts): export CompoundQueryExec and support an injected base scorer by @LuQQiu in #8131
• feat(goosefs): implement safe commit with ConditionalPutCommitHandler by @XuQianJin-Stars in #8134
• feat(java): add scanner performance and memory options by @everySympathy in #8288

Bug Fixes 🐛

• fix: fall back to AVX2 f16 kernels on AVX-512 CPUs by @paramt in #7428
• fix: recognize own commit on conflict and avoid deleting committed artifacts by @jackye1995 in #7722
• fix(bloomfilter): bound read/write to prevent BinaryArray i32 offset overflow by @zhangyue19921010 in #7753
• fix(ngram): recheck contains() with sub-trigram needle instead of dropping rows by @Ar-maan05 in #7845
• fix(index): don't report a ZoneMap value range that an overlay may have escaped by @wjones127 in #7924
• fix(datagen): preserve timestamp time zones by @xixiGoodLuck in #7946
• fix(lance-io): request vended credentials explicitly in namespace provider by @rpgreen in #7978
• fix(io): publish object store metrics for local reads and writes by @wjones127 in #7994
• fix(jni): align CommitBuilder.storageFormat() parsing with LanceFileVersion::from_str by @ivscheianu in #8063
• fix(index): keep address-domain indices off replaced fragments by @wjones127 in #8084
• fix(datagen): make default temporal ranges deterministic by @westonpace in #8121
• fix(index): compare Hamming IVF centroids during merge by @jackye1995 in #8132
• fix: preserve row address filters with stable row ids by @app/lance-gatekeeper in #8145
• fix(index): resolve nested vector index field paths by @justinrmiller in #8148
• fix: preserve schema metadata precedence on merge by @app/lance-gatekeeper in #8160
• fix(python): preserve commit handler exceptions by @app/lance-gatekeeper in #8162
• fix: type scoring columns in projection expressions by @app/lance-gatekeeper in #8163
• fix(io): return an error for short reads by @app/lance-gatekeeper in #8164
• fix(index): preserve row addresses in JSON index training by @app/lance-gatekeeper in #8165
• fix(index): replace same-name index segments across types by @app/lance-gatekeeper in #8166
• fix(index): count HNSW entry point at every level by @BubbleCal in #8167
• fix: reject missing merge insert update addresses by @app/lance-gatekeeper in #8168
• fix: bound commit conflict retry backoff by @app/lance-gatekeeper in #8169
• fix(encoding): reject invalid miniblock row counts by @app/lance-gatekeeper in #8173
• fix(storage): preserve multipart part identity across retries by @app/lance-gatekeeper in #8174
• fix(merge_insert): deduplicate indexed candidates by @app/lance-gatekeeper in #8176
• fix(ci): treat missing merge_insert base tag as not generated by @Xuanwo in #8177
• fix: preserve large unsigned SQL filter literals by @app/lance-gatekeeper in #8178
• fix: avoid Greek stemmer UTF-8 panic by @app/lance-gatekeeper in #8183
• fix(java): type scanner operation failures by @app/lance-gatekeeper in #8184
• fix(index): push down deeply nested scalar filters by @app/lance-gatekeeper in #8186
• fix: respect external base layout in tracked files by @app/lance-gatekeeper in #8187
• fix: align byte-sized batches across data files by @app/lance-gatekeeper in #8189
• fix(ci): enlarge mem pool only around merge_insert index build by @Xuanwo in #8191
• fix(index): initialize indices on nested fields by @app/lance-gatekeeper in #8192
• fix: deduplicate MemWAL IN point keys by @app/lance-gatekeeper in #8197
• fix: return error for divergent MemWAL schemas by @app/lance-gatekeeper in #8198
• fix: prevent update false negatives for address-domain indices by @app/lance-gatekeeper in #8203
• fix: retain empty vector index definitions in retain_relevant_indices by @wkalt in #8208
• fix: tolerate concurrent manifest cleanup by @app/lance-gatefixer in #8213
• fix(index): accept "RQ" when parsing QuantizationType by @jackylee-ch in #8214
• fix: validate GBO descriptors before metadata I/O by @app/lance-gatefixer in #8227
• fix(docs): stop make help from running the website build by @jackylee-ch in #8230
• fix: restore main CI after exact-format reader refactor by @Xuanwo in #8231
• fix(namespace): self-heal DirectoryNamespace read manifest cache by @rpgreen in #8233
• fix(table): reject corrupt row ID metadata by @app/lance-gatefixer in #8254
• fix: compute row offsets for unsorted addresses by @app/lance-gatefixer in #8255
• fix(lance): propagate deletion vector read errors by @app/lance-gatefixer in #8256
• fix: include highest fragment in legacy prefilter masks by @app/lance-gatefixer in #8257
• fix(rowids): validate serialized row ID metadata by @app/lance-gatefixer in #8259
• fix: remove duplicate LanceFileVersion import in lance-file tests by @wjones127 in #8261
• fix(ci): budget merge insert benchmark memory by @app/lance-gatefixer in #8271
• fix: validate top-level schema names in linear time by @app/lance-gatefixer in #8274
• fix(geo): preserve RTree bounds for empty batches by @app/lance-gatefixer in #8287
• fix(fts): sum all boolean must scores by @BubbleCal in #8292

Documentation 📚

• docs: require duplicate PR checks by @Xuanwo in #8265

Performance Improvements 🚀

• perf: skip index metadata deep clone and compare when no vector inference is needed by @LuQQiu in #8107
• perf(fts): pre-load doc lengths in parallel on cold deferred search path by @westonpace in #8119
• perf(index): skip the write-only distance column when loading HNSW partitions by @wombatu-kun in #8136
• perf(fts): bound compound score-floor tie buffering by @BubbleCal in #8161
• perf(transaction): avoid O(n*m) fragment compares in build_manifest for Update/Delete by @zhangyue19921010 in #8210

Other Changes

• refactor: activate exact current-format writers by @Xuanwo in #8023
• refactor: name blob v2 layouts by @Xuanwo in #8266

Full Changelog: release-root/11.0.0-beta.N...v11.0.0-beta.1