Skip to content
A library for efficient similarity search and clustering of dense vectors.
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github Update ISSUE_TEMPLATE.md Jun 25, 2018
.travis Add conda packages metadata + tests. (#769) Apr 5, 2019
acinclude Add conda packages metadata + tests. (#769) Apr 5, 2019
benchs Facebook sync (Mar 2019) (#756) Mar 29, 2019
build-aux Refactor makefiles and add configure script (#466) Jun 2, 2018
c_api [C API] Multi-GPU functions (#628) Dec 23, 2018
conda Add conda packages metadata + tests. (#769) Apr 5, 2019
demos Add conda packages metadata + tests. (#769) Apr 5, 2019
docs Facebook sync (Dec 2018). (#660) Dec 19, 2018
example_makefiles PYTHON executable location is missing in example makefile.inc (#626) Nov 23, 2018
gpu Merge branch 'master' of https://www.github.com/facebookresearch/faiss Apr 9, 2019
misc Refactor makefiles and add configure script (#466) Jun 2, 2018
python Add conda packages metadata + tests. (#769) Apr 5, 2019
tests Add conda packages metadata + tests. (#769) Apr 5, 2019
tutorial Update cpp tutorial makefile for GPU part (#774) Apr 11, 2019
.dockerignore Add Dockerfile (#55) Mar 23, 2017
.gitignore Refactor makefiles and add configure script (#466) Jun 2, 2018
.travis.yml Add conda packages metadata + tests. (#769) Apr 5, 2019
AutoTune.cpp Facebook sync (Mar 2019) (#756) Mar 29, 2019
AutoTune.h Facebook sync (Mar 2019) (#756) Mar 29, 2019
AuxIndexStructures.cpp Facebook sync (Mar 2019) (#756) Mar 29, 2019
AuxIndexStructures.h Facebook sync (Mar 2019) (#756) Mar 29, 2019
CODE_OF_CONDUCT.md OSS Automated Fix: Addition of Code of Conduct Mar 22, 2019
CONTRIBUTING.md Update CONTRIBUTING.md Mar 1, 2017
Clustering.cpp Facebook sync (Mar 2019) (#756) Mar 29, 2019
Clustering.h Facebook sync (Mar 2019) (#756) Mar 29, 2019
Dockerfile Configure install paths in Dockerfile. (#772) Apr 7, 2019
FaissAssert.h Facebook sync (#504) Jul 6, 2018
FaissException.cpp Facebook sync (#504) Jul 6, 2018
FaissException.h Facebook sync (#504) Jul 6, 2018
HNSW.cpp Facebook sync (Mar 2019) (#756) Mar 29, 2019
HNSW.h Facebook sync (Mar 2019) (#756) Mar 29, 2019
Heap.cpp Facebook sync (Dec 2018). (#660) Dec 19, 2018
Heap.h Facebook sync (#504) Jul 6, 2018
INSTALL.md Update INSTALL.md Apr 11, 2019
IVFlib.cpp Facebook sync (Mar 2019) (#756) Mar 29, 2019
IVFlib.h Facebook sync (#573) Aug 30, 2018
Index.cpp Facebook sync (#504) Jul 6, 2018
Index.h Add conda packages metadata + tests. (#769) Apr 5, 2019
IndexBinary.cpp Facebook sync (#504) Jul 6, 2018
IndexBinary.h Facebook sync (Mar 2019) (#756) Mar 29, 2019
IndexBinaryFlat.cpp Facebook sync (Dec 2018). (#660) Dec 19, 2018
IndexBinaryFlat.h Facebook sync (#504) Jul 6, 2018
IndexBinaryFromFloat.cpp Facebook sync (Dec 2018). (#660) Dec 19, 2018
IndexBinaryFromFloat.h Facebook sync (Dec 2018). (#660) Dec 19, 2018
IndexBinaryHNSW.cpp Facebook sync (Mar 2019) (#756) Mar 29, 2019
IndexBinaryHNSW.h Facebook sync (Mar 2019) (#756) Mar 29, 2019
IndexBinaryIVF.cpp Facebook sync (Mar 2019) (#756) Mar 29, 2019
IndexBinaryIVF.h Facebook sync (Mar 2019) (#756) Mar 29, 2019
IndexFlat.cpp add bench_all_ivf Dec 20, 2018
IndexFlat.h Facebook sync (#504) Jul 6, 2018
IndexHNSW.cpp Facebook sync (Mar 2019) (#756) Mar 29, 2019
IndexHNSW.h Facebook sync (Mar 2019) (#756) Mar 29, 2019
IndexIVF.cpp Facebook sync (Mar 2019) (#756) Mar 29, 2019
IndexIVF.h Facebook sync (Mar 2019) (#756) Mar 29, 2019
IndexIVFFlat.cpp Facebook sync (Mar 2019) (#756) Mar 29, 2019
IndexIVFFlat.h Facebook sync (Mar 2019) (#756) Mar 29, 2019
IndexIVFPQ.cpp add bench_all_ivf Dec 20, 2018
IndexIVFPQ.h Facebook sync (Dec 2018). (#660) Dec 19, 2018
IndexLSH.cpp Facebook sync (#504) Jul 6, 2018
IndexLSH.h Facebook sync (#504) Jul 6, 2018
IndexPQ.cpp Facebook sync (Mar 2019) (#756) Mar 29, 2019
IndexPQ.h Facebook sync (#504) Jul 6, 2018
IndexReplicas.cpp Facebook sync (Mar 2019) (#756) Mar 29, 2019
IndexReplicas.h Facebook sync (Mar 2019) (#756) Mar 29, 2019
IndexScalarQuantizer.cpp Facebook sync (Mar 2019) (#756) Mar 29, 2019
IndexScalarQuantizer.h Facebook sync (Mar 2019) (#756) Mar 29, 2019
IndexShards.cpp Facebook sync (Mar 2019) (#756) Mar 29, 2019
IndexShards.h Facebook sync (Mar 2019) (#756) Mar 29, 2019
InvertedLists.cpp Facebook sync (Mar 2019) (#756) Mar 29, 2019
InvertedLists.h Facebook sync (Mar 2019) (#756) Mar 29, 2019
LICENSE changed license Jul 30, 2017
Makefile Make clean target bourne shell compatible. (#783) Apr 10, 2019
MetaIndexes.cpp Facebook sync (Mar 2019) (#756) Mar 29, 2019
MetaIndexes.h Facebook sync (Mar 2019) (#756) Mar 29, 2019
OnDiskInvertedLists.cpp Facebook sync (#504) Jul 6, 2018
OnDiskInvertedLists.h Facebook sync (#504) Jul 6, 2018
PATENTS changed license Jul 30, 2017
PolysemousTraining.cpp Facebook sync (Dec 2018). (#660) Dec 19, 2018
PolysemousTraining.h Facebook sync (#504) Jul 6, 2018
ProductQuantizer.cpp Facebook sync (Mar 2019) (#756) Mar 29, 2019
ProductQuantizer.h Facebook sync (Mar 2019) (#756) Mar 29, 2019
README.md Update README.md Jan 15, 2019
VectorTransform.cpp Facebook sync (Mar 2019) (#756) Mar 29, 2019
VectorTransform.h Facebook sync (Mar 2019) (#756) Mar 29, 2019
WorkerThread.cpp Facebook sync (Mar 2019) (#756) Mar 29, 2019
WorkerThread.h Facebook sync (Mar 2019) (#756) Mar 29, 2019
configure Add conda packages metadata + tests. (#769) Apr 5, 2019
configure.ac Improve arm64 support. (#676) Feb 15, 2019
depend Add conda packages metadata + tests. (#769) Apr 5, 2019
hamming.cpp Facebook sync (Dec 2018). (#660) Dec 19, 2018
hamming.h Facebook sync (Dec 2018). (#660) Dec 19, 2018
index_io.cpp Facebook sync (Mar 2019) (#756) Mar 29, 2019
index_io.h Facebook sync (Mar 2019) (#756) Mar 29, 2019
makefile.inc.in Add conda packages metadata + tests. (#769) Apr 5, 2019
utils.cpp Facebook sync (Mar 2019) (#756) Mar 29, 2019
utils.h Facebook sync (Mar 2019) (#756) Mar 29, 2019
utils_simd.cpp Facebook sync (Dec 2018). (#660) Dec 19, 2018

README.md

Faiss

Faiss is a library for efficient similarity search and clustering of dense vectors. It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM. It also contains supporting code for evaluation and parameter tuning. Faiss is written in C++ with complete wrappers for Python/numpy. Some of the most useful algorithms are implemented on the GPU. It is developed by Facebook AI Research.

NEWS

NEW: version 1.5.0 (2018-12-19) GPU binary flat index and binary HNSW index

NEW: version 1.4.0 (2018-08-30) no more crashes in pure Python code

NEW: version 1.3.0 (2018-07-12) support for binary indexes

NEW: latest commit (2018-02-22) supports on-disk storage of inverted indexes, see demos/demo_ondisk_ivf.py

NEW: latest commit (2018-01-09) includes an implementation of the HNSW indexing method, see benchs/bench_hnsw.py

NEW: there is now a Facebook public discussion group for Faiss users at https://www.facebook.com/groups/faissusers/

NEW: on 2017-07-30, the license on Faiss was relaxed to BSD from CC-BY-NC. Read LICENSE for details.

Introduction

Faiss contains several methods for similarity search. It assumes that the instances are represented as vectors and are identified by an integer, and that the vectors can be compared with L2 distances or dot products. Vectors that are similar to a query vector are those that have the lowest L2 distance or the highest dot product with the query vector. It also supports cosine similarity, since this is a dot product on normalized vectors.

Most of the methods, like those based on binary vectors and compact quantization codes, solely use a compressed representation of the vectors and do not require to keep the original vectors. This generally comes at the cost of a less precise search but these methods can scale to billions of vectors in main memory on a single server.

The GPU implementation can accept input from either CPU or GPU memory. On a server with GPUs, the GPU indexes can be used a drop-in replacement for the CPU indexes (e.g., replace IndexFlatL2 with GpuIndexFlatL2) and copies to/from GPU memory are handled automatically. Results will be faster however if both input and output remain resident on the GPU. Both single and multi-GPU usage is supported.

Building

The library is mostly implemented in C++, with optional GPU support provided via CUDA, and an optional Python interface. The CPU version requires a BLAS library. It compiles with a Makefile and can be packaged in a docker image. See INSTALL.md for details.

How Faiss works

Faiss is built around an index type that stores a set of vectors, and provides a function to search in them with L2 and/or dot product vector comparison. Some index types are simple baselines, such as exact search. Most of the available indexing structures correspond to various trade-offs with respect to

  • search time
  • search quality
  • memory used per index vector
  • training time
  • need for external data for unsupervised training

The optional GPU implementation provides what is likely (as of March 2017) the fastest exact and approximate (compressed-domain) nearest neighbor search implementation for high-dimensional vectors, fastest Lloyd's k-means, and fastest small k-selection algorithm known. The implementation is detailed here.

Full documentation of Faiss

The following are entry points for documentation:

Authors

The main authors of Faiss are:

Reference

Reference to cite when you use Faiss in a research paper:

@article{JDH17,
  title={Billion-scale similarity search with GPUs},
  author={Johnson, Jeff and Douze, Matthijs and J{\'e}gou, Herv{\'e}},
  journal={arXiv preprint arXiv:1702.08734},
  year={2017}
}

Join the Faiss community

For public discussion of Faiss or for questions, there is a Facebook public discussion group at https://www.facebook.com/groups/faissusers/

We monitor the issues page of the repository. You can report bugs, ask questions, etc.

License

Faiss is BSD-licensed. We also provide an additional patent grant.

You can’t perform that action at this time.