build_index is very slow

[xiongqiangcs]

machine：

• cpu-machine：Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz
• mem: 32G
• cpu-cores: 16

code：

from autofaiss import build_index
import numpy as np

embeddings = np.float32(np.random.rand(1000000, 512))
index, index_infos = build_index(embeddings, save_on_disk=False)

log：

[rom1504]

Building an hnsw is indeed one of the slowest adding method, especially with random vectors.
This is calling faiss index.add

If you want to optimize for speed of building an index you can force the index key to not use hnsw

[xiongqiangcs]

Building an hnsw is indeed one of the slowest adding method, especially with random vectors. This is calling faiss index.add

If you want to optimize for speed of building an index you can force the index key to not use hnsw

autofaiss build_index cost time 49min，faiss index add cost time 34 min

from autofaiss import setup_logging, build_index
from autofaiss import Timeit
import numpy as np
import faiss
import logging
import logging.config
import multiprocessing

setup_logging(logging.INFO)
faiss.omp_set_num_threads(multiprocessing.cpu_count())


embeddings = np.float32(np.random.rand(1000000, 512))
with Timeit("build index"):
    index = faiss.index_factory(512, "HNSW32", faiss.METRIC_L2)
    index.add(embeddings)

[xiongqiangcs]

embedding_reader parameter max_piece_size and parallel_pieces need optimize?

autofaiss/autofaiss/indices/build.py

Lines 98 to 102 in d5c773f

	for batch_id, (vec_batch, ids_batch) in enumerate(embedding_reader(batch_size=batch_size)):
	if add_embeddings_with_ids:
	trained_index.add_with_ids(vec_batch, ids_batch["i"].to_numpy())
	else:
	trained_index.add(vec_batch)

[rom1504]

What kind of local disk do you have ?

[xiongqiangcs]

What kind of local disk do you have ?

SSD