Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Pre and post processing
Pre- and post-processing is used to: remap vector ids, apply transformations to the data, and re-rank search results with a better index.
Faiss ID mapping
By default Faiss assigns a sequential id to vectors added to the indexes. This page explains how to change this to arbitrary ids.
Index classes implement a
add_with_ids method, where 64-bit vector ids can be provided in addition to the the vectors.
At search time, the class will return the stored ids rather than the initial vectors.
This index encapsulates another index and translates ids when adding and searching. It maintains a table with the mapping.
index = faiss.IndexFlatL2(xb.shape) ids = np.arange(xb.shape) index.add_with_ids(xb, ids) # this will crash, because IndexFlatL2 does not support add_with_ids index2 = faiss.IndexIDMap(index) index2.add_with_ids(xb, ids) # works, the vectors are stored in the underlying index
IDs in the
IndexIVF sub-classes always store vector IDs. Therefore, the
IndexIDMap's additional table is a waste of space. The
Pre-transforming the data
It is often useful to transform data prior to indexing. Transformation classes inherit
VectorTransform applies a transformation to input vectors (of dimension
d_in) and outputs vectors of size
||useful to re-balance components of a vector before indexing in an
|remapping of dimensions||
||to reduce or increase the size of a vector because the index has a preferred dimension, or to apply a random permutation on dimensions.|
||for dimensionality reduction|
||OPQ applies a rotation to the input vectors to make them more amenable to PQ coding. See Optimized product quantization, Ge et al., CVPR'13 for more details.|
Transformations can be trained from a set of vectors if it makes sense, using the method
train. They can be applied to a set of vectors with
An index can be wrapped in a
IndexPreTransform index so that the mapping occurs transparently, and training is integrated with the Index training.
Example: apply a PCA to reduce the number of dimensions
See the next section
With an IndexPreTransform
For example, if the input vectors are 2048D, and have to be reduced to 16 bytes, it makes sense to reduce them with a PCA before.
# the IndexIVFPQ will be in 256D not 2048 coarse_quantizer = faiss.IndexFlatL2 (256) sub_index = faiss.IndexIVFPQ (coarse_quantizer, 256, ncoarse, 16, 8) # PCA 2048->256 # also does a random rotation after the reduction (the 4th argument) pca_matrix = faiss.PCAMatrix (2048, 256, 0, True) #- the wrapping index index = faiss.IndexPreTransform (pca_matrix, sub_index) # will also train the PCA index.train(...) # PCA will be applied prior to addition index.add(...)
Example: increase the number of dimensions
It is sometimes useful to increase the number of dimensions d by inserting zeros in the vectors. This can be useful for:
da multiple of 4, this is what the distance computations are optimized for
da multiple of
Mis the size of a PQ.
This can be done with
# input is in dimension d, but we want a multiple of M d2 = int((d + M - 1) / M) * M remapper = faiss.RemapDimensionsTransform (d, d2, true) # the index in d2 dimensions index_pq = faiss.IndexPQ(d2, M, 8) # the index that will be used for add and search index = faiss.IndexPreTransform (remapper, index_pq)
IndexRefineFlat: re-ranking search results
When querying a vector, it may be useful to re-rank the search results with real distance computations. The following example searches an index with an
IndexPQ, then reranks the first results by computing real distances:
q = faiss.IndexPQ (d, M, nbits_per_index) rq = faiss.IndexRefineFlat (q) rq.train (xt) rq.add (xb) rq.k_factor = 4 D, I = rq:search (xq, 10)
The search function will fetch 4 * 10 nearest neighbors from the IndexPQ, then compute the real distance for each of the results, and keep the 10 best ones. Note that the
IndexRefineFlat has to store the full vectors, so it is not memory-efficient.
IndexShards: combining results from several indexes
When the dataset is spread over several indexes, a query can be dispatched over them and the results can be combined with an
IndexShards. This is also useful if the index is spread over several GPUs and the queries can be done in parallel, see
shards set to true in