perf: reduce lock contention

[usamoi]

create index on laion1m768_train using vchordrq (embedding vector_ip_ops) with (options = $$
build.pin = 2
build.internal.lists = [256, 4096]
build.internal.build_threads = 20
build.internal.spherical_centroids = true
degree_of_parallelism = 40
$$);

degree_of_parallelism is 32 by default. It reduces lock contention during insertion, especially, in a build where lists.len() != 1.

build.pin is -1 by default. build.pin = 2 caches almost everything in memory, which speeds build but spends more memory. build.pin = false and build.pin = true still works, and they're mapped to build.pin = -1 and build.pin = 1.

[VoVAllen]

What is vector_tapes? It should be a hidden config named like _vector_tapes

[usamoi]

What is vector_tapes? It should be a hidden config named like _vector_tapes

It's an experimental trick to speed parallel build, and won't be merged as-is.

[cutecutecat]

With vector_tapes = 32 and 16 workers, the assignment of LAION-100M can be accerated from 3300s to 1800s.

storage	worker	assign time	CPU util
1x3000G	32	1993.03s	60%
1x3000G	16	1848.54 s	54%
1x3000G	8	3104.53 s	42%

[usamoi]

What is vector_tapes? It should be a hidden config named like _vector_tapes

It's renamed to degree_of_parallelism now. It's hint that the index takes advantage of it, creates multiple vector linked lists, and randomly chooses one to insert, in order to reduce lock contention.

Its default is 32 and users don't need change it except if they have a very very large machine.

[VoVAllen]

Is there any downside to have degree_of_parallelism as 32? Shall we prefix it with _ ?

[usamoi]

Is there any downside to have degree_of_parallelism as 32?

I don't think so.

Shall we prefix it with _ ?

I don't think it's necessary. rerank_in_table is much more bugful and we don't make it prefixed with _.

[usamoi]

This feature modified the layout of storage, so it should be in 0.6.0. We could not pick it to 0.5.

[cutecutecat]

To be noticed, the 8w build time is slower than before, from 2400s -> 3100s .

storage	worker	assign time	CPU util
1x3000G	32	1993.03s	60%
1x3000G	16	1848.54 s	54%
1x3000G	8	3104.53 s	42%