MB-62182: Avoid re-training vector indexes during merge by Thejas-bhat · Pull Request #2204 · blevesearch/bleve

Thejas-bhat · 2025-06-17T09:00:31Z

The main purpose of this PR is to avoid unnecessary re-training of the vector indexes during merge process.
Going by the numbers, we need roughly 156K vectors for a 1M dataset ((min_num_vectors_per_centroid) * num_centroids = 39 * 4 * sqrt(1M)) as per recommendation
The data ingestion is now split into 2 phases - the first phase involves creating a centroid index using the Train() API and the bolt is recorded with the progress in terms of samples trained upon. The second phase is just the normal indexing of data using the Batch() or the Index() APIs.
Later on, when the vector indexes are getting merged the merger will use the centroid index to merge the inverted lists (centroids) in a block-wise fashion without reconstructing the layout.
The feature can be enabled by passing a "vector_index_fast_merge": "true" key-value pair as part of kvconfig while creating/opening the index.

abhinavdangeti · 2026-03-19T21:26:01Z

@Thejas-bhat is this ready for review? Looks like it needs a rebase.

index.go

go.mod

index_alias_impl.go

index_impl.go

index.go

index/scorch/train_vector.go

coveralls · 2026-03-27T01:41:24Z

coverage: 52.616% (-0.08%) from 52.693%
when pulling 9d87598 on fastmerge
into 6b72a24 on master.

abhinavdangeti · 2026-03-27T17:16:34Z

@Thejas-bhat have you verified unit tests to work under -tags=vectors?

Thejas-bhat · 2026-03-27T17:42:06Z

@Thejas-bhat have you verified unit tests to work under -tags=vectors?

yeah it passes on my local system

...
?   	github.com/blevesearch/bleve/v2/search/highlight/highlighter/ansi	[no test files]
?   	github.com/blevesearch/bleve/v2/search/highlight/highlighter/html	[no test files]
ok  	github.com/blevesearch/bleve/v2/search/highlight/highlighter/simple	(cached)
ok  	github.com/blevesearch/bleve/v2/search/query	(cached)
ok  	github.com/blevesearch/bleve/v2/search/scorer	(cached)
ok  	github.com/blevesearch/bleve/v2/search/searcher	5.670s
?   	github.com/blevesearch/bleve/v2/size	[no test files]
ok  	github.com/blevesearch/bleve/v2/test	20.841s
?   	github.com/blevesearch/bleve/v2/util	[no test files]

in a vector index - First comes the sampling phase, where a faiss index is created using the vectors randomly sampled from KV. The coarse quantizer of this 'centroid index' is used in the later parts of the index lifecycle. - The data ingestion on all the bleveDests is blocked until this centroid index is generated and streamed to all the partitions on that node. - After the sampling phase, the feeds are unlocked at the batch worker level to ingest the data, which follows the existing codepath. - blevesearch/bleve#2204 Change-Id: I1f7e35d6b519a4cf5a502a7246f407c33c9dfae5 Reviewed-on: https://review.couchbase.org/c/cbft/+/235240 Tested-by: <thejas.orkombu@couchbase.com> Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: <thejas.orkombu@couchbase.com>

abhinavdangeti added this to the v2.6.0 milestone Jul 21, 2025

Thejas-bhat force-pushed the fastmerge branch from f12b089 to acce003 Compare November 26, 2025 19:17

Thejas-bhat force-pushed the fastmerge branch from 31c2cdb to 4e61bd8 Compare January 15, 2026 21:38

Thejas-bhat changed the title ~~WIP fast merge~~ [WIP] MB-62182: Avoid re-training vector indexes during merge Jan 15, 2026

Thejas-bhat force-pushed the fastmerge branch 3 times, most recently from c7a94a6 to 59a66df Compare January 29, 2026 19:16

Thejas-bhat marked this pull request as ready for review January 29, 2026 19:34

Thejas-bhat changed the title ~~[WIP] MB-62182: Avoid re-training vector indexes during merge~~ MB-62182: Avoid re-training vector indexes during merge Jan 29, 2026

Thejas-bhat force-pushed the fastmerge branch from e2edbd2 to 5cd9550 Compare January 29, 2026 22:47

CascadingRadium added this to Fast Merge Jan 30, 2026

github-project-automation bot moved this to Todo in Fast Merge Jan 30, 2026

Thejas-bhat moved this from Todo to In Progress in Fast Merge Jan 30, 2026

Thejas-bhat force-pushed the fastmerge branch 4 times, most recently from 74072d3 to ce43814 Compare February 6, 2026 00:24

Thejas-bhat force-pushed the fastmerge branch from ce43814 to cf0b034 Compare February 19, 2026 18:10

abhinavdangeti requested review from CascadingRadium, Likith101, abhinavdangeti, capemox and maneuvertomars March 19, 2026 21:32

Thejas-bhat force-pushed the fastmerge branch 4 times, most recently from 8ad6a31 to b0c71ba Compare March 23, 2026 23:05