make nbits configurable for graph indices based on PQ #3031
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This pull request was exported from Phabricator. Differential Revision: D48860659 |
…ch#3031) Summary: Pull Request resolved: facebookresearch#3031 As requested in facebookresearch#3027 Indeed, PQ sizes with nbits > 8 are good tradeoffs, so it is interesting to support them. Differential Revision: D48860659 fbshipit-source-id: 5cd679a4a62f3c9177962481679aa85e88827cc4
|
This pull request was exported from Phabricator. Differential Revision: D48860659 |
mdouze
force-pushed
the
export-D48860659
branch
from
0fed833
to
5222d49
Compare
|
Thanks @mdouze! I can run some tests on this if that would be helpful! |
I am merging this PR. LMK if you have feedback on the functionality (positive or negative). |
|
This pull request has been merged in 82352dd. |
abhinavdangeti
pushed a commit
to blevesearch/faiss
that referenced
this pull request
Jul 12, 2024
…ch#3031) Summary: Pull Request resolved: facebookresearch#3031 As requested in facebookresearch#3027 Indeed, PQ sizes with nbits > 8 are good tradeoffs, so it is interesting to support them. Reviewed By: pemazare Differential Revision: D48860659 fbshipit-source-id: 6f3c642e0902e1523bef36db6be3af3688d529a5
abhinavdangeti
added a commit
to blevesearch/faiss
that referenced
this pull request
Aug 13, 2024
…s@bleve * Merging faiss's [v1.8.0](https://github.com/facebookresearch/faiss/releases/tag/v1.8.0) release into bleve branch on fork. * Brings in: ``` |\ | * 943d08b Junjie Qi | Prepare for v.1.8.0 release (facebookresearch#3265) | * abff75e Jeff Johnson | faiss gpu: fix DeviceVector reallocations (facebookresearch#3256) | * b8d91d8 Gergely Szilvasy | fixes for D51074065 | * 27b1055 Tarang Jain | Integrate IVF-PQ from RAFT (facebookresearch#3044) | * 87d43b9 Jim Borden | Fix AVX2 build on Windows (facebookresearch#3238) | * c577f43 Gergely Szilvasy | fix perf test (facebookresearch#3255) | * 8400ece Richard Barnes | Remove unused variables in faiss/utils/partitioning.cpp | * 873b1bc Gergely Szilvasy | fix perf test (facebookresearch#3253) | * 6d00c41 Richard Barnes | Remove unused variables in faiss/utils/hamming_distance/neon-inl.h | * c0624d2 Richard Barnes | Remove unused variables in faiss/invlists/OnDiskInvertedLists.cpp | * a87b432 Richard Barnes | Remove unused variables in faiss/IndexIVFPQ.cpp | * 1338e0d Richard Barnes | Remove unused variables in faiss/IndexBinaryIVF.cpp | * 1b0d274 Richard Barnes | Remove unused variables in faiss/IndexIVF.cpp | * 53fc617 Richard Barnes | Remove unused variables in faiss/tests/test_threaded_index.cpp | * aaca1c2 Richard Barnes | Remove semicolon(s) from 3 files inc facer/engine/utils/UDPServer.h | * a187394 Gergely Szilvasy | libraft 24.02 is released, switching channel | * f262011 Gergely Szilvasy | fix omp parallelism in fast scan range search | * 8898eab yuhaijun999 | Add feature in IndexIDMap.cpp range_search with Parameters. (facebookresearch#3213) | * ebb5f84 Xiaozhong Pan | add a context parameter to InvertedLists and InvertedListsIterator (facebookresearch#3247) | * bfa46a3 Gergely Szilvasy | pin circleci windows image (facebookresearch#3248) | * ed3f6e5 Gergely Szilvasy | faiss-gpu to use 11.4.4 and be the only one supporting P100 (facebookresearch#3237) | * 31a29d0 xinhuitian | HNSW search use efSearch from params if provided (facebookresearch#3233) | * c1822a8 Gergely Szilvasy | cuda dependencies (facebookresearch#3230) | * 1d0e8d4 Gergely Szilvasy | index optimizer (facebookresearch#3154) | * 75ae0bf Richard Barnes | Remove unused variables in faiss/IndexScalarQuantizer.cpp | * 420d25f Maria Lomeli | Index pretransform support in search_preassigned (facebookresearch#3225) | * 31bddeb Gergely Szilvasy | add faiss_gpu lib (facebookresearch#3222) | * 2817344 Gergely Szilvasy | fix ACCESS VIOLATION error when searching using IDSelectorArray | * 67c6a19 Richard Barnes | Remove extra semi colon from faiss/impl/ProductAdditiveQuantizer.h | * 898ce35 Richard Barnes | Remove unused function from faiss/utils/distances_simd.cpp | * 1dcb5d3 Kota Yamaguchi | Add missing header files (facebookresearch#3218) | * 51b6083 Gergely Szilvasy | faiss on rocksdb demo (facebookresearch#3216) | * c4b91a5 Gergely Szilvasy | Replace pickle serialization to address security vulnerability | * a30fd74 Richard Barnes | Remove unused function from faiss/clone_index.cpp | * fc4dbd5 Richard Barnes | Remove extra semi colon from faiss/IndexIVFSpectralHash.cpp | * 6b2c79c Richard Barnes | Remove extra semi colon from faiss/index_factory.cpp | * a7b76a7 Richard Barnes | Fix shadowed variable in faiss/IndexAdditiveQuantizerFastScan.cpp | * ae25b1b Richard Barnes | Fix shadowed variable in faiss/impl/residual_quantizer_encode_steps.cpp | * 7c4fb6d Richard Barnes | Fix shadowed variable in faiss/impl/ResultHandler.h | * 12637a2 Richard Barnes | Fix shadowed variable in faiss/utils/utils.cpp | * 63edc32 Kota Yamaguchi | Fix AVX512 build targets in CMakeLists.txt (facebookresearch#3214) | * 99c0e2b Richard Barnes | Remove unused variables in faiss/utils/sorting.cpp | * e8494f6 Richard Barnes | Remove unused variables in faiss/IndexBinaryHash.cpp | * bbb6ec7 Richard Barnes | Remove unused variables in faiss/IndexShardsIVF.cpp | * 0716bde Richard Barnes | Fix shadowed variable in faiss/IndexNSG.cpp | * bffedda Richard Barnes | Fix shadowed variable in faiss/IndexIDMap.cpp | * a70a8a5 Richard Barnes | Remove unused variables in faiss/impl/HNSW.cpp | * b274cb4 Richard Barnes | Remove unused variables in faiss/IndexIVFFastScan.cpp | * e55a0ac Richard Barnes | Remove unused variables in faiss/impl/index_write.cpp | * a651069 Richard Barnes | Remove unused variables in faiss/impl/index_read.cpp | * e19de27 Richard Barnes | Remove extra semi colon from faiss/IndexIVFFastScan.cpp | * b10eb35 Richard Barnes | Remove extra semi colon from faiss/Clustering.cpp | * 683eadf Richard Barnes | Remove extra semi colon from faiss/utils/distances_simd.cpp | * 522452f Richard Barnes | Remove extra semi colon from faiss/python/python_callbacks.cpp | * 3e666ae Richard Barnes | Remove extra semi colon from faiss/invlists/InvertedLists.cpp | * 7e01b47 Gergely Szilvasy | fix raft build | * 7f3e0a3 Richard Barnes | Mismatch tags take 2 (facebookresearch#3211) | * aff812e Richard Barnes | Fix shadowed variable in faiss/clone_index.cpp | * c540e76 Kota Yamaguchi | Use packaging instead of deprecated distutils for version handling (facebookresearch#3191) | * 091f344 Gergely Szilvasy | add gpu to avx512 (facebookresearch#3210) | * 5e3eae4 Richard Barnes | Remove unused variables in faiss/utils/hamming.cpp | * 739ce13 Matthijs Douze | doc update to mention the paper | * 9a63a3c Richard Barnes | Fix shadowed variable in faiss/IndexFastScan.cpp | * 7dd06dd Richard Barnes | Fix shadowed variable in faiss/impl/NNDescent.cpp | * 0fc8456 Maria Lomeli | Offline IVF powered by faiss big batch search (facebookresearch#3202) | * 4150fb1 Richard Barnes | Fix shadowed variable in faiss/impl/PolysemousTraining.cpp | * 3973017 Richard Barnes | Fix shadowed variable in faiss/IndexBinaryIVF.cpp | * 46320e0 Richard Barnes | Fix shadowed variable in faiss/utils/hamming_distance/avx2-inl.h | * 1be1d32 Richard Barnes | Fix shadowed variable in faiss/impl/io.cpp | * 7442a54 Richard Barnes | Fix shadowed variable in faiss/impl/index_write.cpp | * cfcefc0 Richard Barnes | Remove unused variables in faiss/utils/sorting.cpp | * f884ba2 Richard Barnes | Remove unused variables in faiss/IndexHNSW.cpp | * b7efd3a Gergely Szilvasy | bump libraft version to fix nightly | * c3aa526 Naveen Tatikonda | Add SIMD NEON Optimization for QT_FP16 in Scalar Quantizer (facebookresearch#3166) | * 32f0e8c Matthijs Douze | Generalize ResultHanlder, support range search for HNSW and Fast Scan (facebookresearch#3190) | * 0013c70 Gergely Szilvasy | avx512 CI + conda packages (facebookresearch#3197) | * 0710cbd Richard Barnes | Fix shadowed variable in faiss/impl/ResultHandler.h | * 42b6216 Richard Barnes | Fix shadowed variable in faiss/impl/ScalarQuantizer.cpp | * c399b11 Richard Barnes | Fix shadowed variable in faiss/utils/hamming.cpp | * 952941b Richard Barnes | Fix shadowed variable in faiss/MatrixStats.cpp | * db09984 Richard Barnes | Fix shadowed variable in faiss/impl/simd_result_handlers.h | * beef610 Gergely Szilvasy | faiss paper benchmarks (facebookresearch#3189) | * b7681be Richard Barnes | Remove unused exception parameter from files inc facer/engine/utils/StatsD.cpp | * 449f02c Richard Barnes | Add missing fallthrough to fbcode/faiss/utils/hamming_distance/neon-inl.h | * 0727fff Eddy Li | Adding [[fallthrough]] annotation to case statements in faiss/utils/hamming_distance/neon-inl.h | * 9d74f09 Jason Sylka | Adding [[fallthrough]] annotation to case statements in faiss/utils/hamming_distance/generic-inl.h | * 77c28f8 Gergely Szilvasy | Back out "Offline IVF powered by faiss big batch search" | * 65b08ef Richard Barnes | Remove unused function from faiss/utils/quantize_lut.cpp | * 5621487 Richard Barnes | Remove unused function from faiss/impl/ProductQuantizer.cpp | * 9a8b34e Maria | Offline IVF powered by faiss big batch search (facebookresearch#3175) | * be12427 Maria Lomeli | Upstream changes to big batch search (facebookresearch#3170) | * 79f558f Xiaozhong Pan | set keep_max for InvertedListScanner child classes (facebookresearch#3169) | * 9f8f238 Bader-eddine Ouaich | Fix potential buffer overflow (facebookresearch#3146) | * 9107035 Jaepil Jeong | Fix compile error (facebookresearch#3162) | * d40b39a Alexandr Guzhva | Nit: fix variable types (facebookresearch#3147) | * aebc1d5 Richard Barnes | Remove extra semi colon from faiss/impl/IDSelector.h | * 3dab47c Richard Barnes | Remove extra semi colon from facer/engine/libsvm/svm.cpp | * d2cb016 Richard Barnes | Remove unused exception parameter from files inc faiss/tests/test_approx_topk.cpp | * 5b6c4b4 Matthijs Douze | Back out "printf -> fmt::print in files inc faiss/IndexBinaryHNSW.cpp" (facebookresearch#3164) | * 131adc5 Richard Barnes | printf -> fmt::print in files inc faiss/IndexBinaryHNSW.cpp | * eefa391 Alexandr Guzhva | Introduce avx512 optimization mode and FAISS_OPT_LEVEL env variable (facebookresearch#3150) | * 4c83965 Gergely Szilvasy | benchmark view results (facebookresearch#3144) | * 9519a19 Gergely Szilvasy | benchmark refactor | * a5b03cb Alexandr Guzhva | Fix build on Alpine Linux (facebookresearch#3148) | * 4bf8f93 Yuri Vanin | Add NegativeDistanceComputer::distances_batch_4 override (facebookresearch#3149) | * 90654d6 Gergely Szilvasy | benchmark core faiss prereqs | * 04bb0a8 Alexandr Guzhva | improve ScalarQuantizer performance, ESPECIALLY on old GCC (facebookresearch#3141) | * 43f8220 Matthijs Douze | fix scopedeleter diff | * d3692d2 Alexandr Guzhva | Deprecate ScopeDeleter and ScopeDeleter1 in favor of std::unique_ptr<[]> (facebookresearch#3108) | * eb071f8 luyuncheng | Fix is_trained in IndexNSGSQ (facebookresearch#3145) | * d643c41 Ben Frederickson | use precomputed norms for raft brute_force knn calls (facebookresearch#3089) | * b109d08 Matthijs Douze | Search and return codes (facebookresearch#3143) | * 467f70e Jeff Johnson | Consolidate GPU IVF query tile calculation + special handling for large query memory requirements | * 411c172 Robert Maynard | Add linker script to support large cuda cubin files (facebookresearch#3115) | * 09c7aac Jeff Johnson | Faiss GPU CUDA 12 fix: warp synchronous behavior | * 0c2243c Christopher Ponce de Leon | Revert D51029740: Namespace doesn't need to be followed by semicolon | * 438b519 Richard Barnes | Namespace doesn't need to be followed by semicolon | * 9a66532 Alexandr Guzhva | Add search parameters for IndexRefine::search() and IndexRefineFlat::search() (facebookresearch#3122) | * df7280b pe4eniks | Documentation fixes (facebookresearch#3092) | * 6b76150 chasingegg | Remove confusing comments in partitioning.cpp (facebookresearch#3104) | * 6c89c8b Gergely Szilvasy | fix faiss-gpu nightly | * 0c07a11 Gergely Szilvasy | fix raft contbuild and switch to libraft 23.12 (facebookresearch#3116) | * 9bb6b4b Gergely Szilvasy | fix test TestCrossCodebookComputations::test_precomp | * c3b9374 Gergely Szilvasy | bench_fw - fixes & nits for oss (facebookresearch#3102) | * 0a00d81 Gergely Szilvasy | offline index evaluation (facebookresearch#3097) | * f969d7a Matthijs Douze | better docs | * edcf743 Corey J. Nolet | Integrate IVF-Flat from RAFT (facebookresearch#2521) | * 458633c Robert Maynard | Remove uneeded PTX code generation from libfaiss builds (facebookresearch#3083) | * 2b48901 Matthijs Douze | Remove 1L and 1UL | * 3f3321c Alexandr Guzhva | Small refactoring of inverted lists (facebookresearch#3055) | * 834c543 Haijun Yu | Fix SingleBestResultHandler bug. if IndexFlatL2 empty data then search topk = 1 return label = 0 not -1. (facebookresearch#3075) | * 9db1824 Matthijs Douze | Relax IVFFlatDedup test (facebookresearch#3077) | * 0f18251 generatedunixname89002005325676 | Daily `arc lint --take CLANGFORMAT` | * cf90435 Matthijs Douze | fix flaky GPU test | * e18de23 Alexandr Guzhva | Fix chunk-based processing in ResidualCoarseQuantizer::search() (facebookresearch#3047) | * a1814be Alexandr Guzhva | Simplify dependency components chain (facebookresearch#3058) | * 56b1087 Alexandr Guzhva | move fvec_madd_* functions declarations to a right header (facebookresearch#3054) | * 0780a28 Alexandr Guzhva | Fix a couple of type mismatches (facebookresearch#3059) | * 592f301 Alexandr Guzhva | Unneeded field, exists in a baseclass (facebookresearch#3064) | * 6218111 chasingegg | Fix some typos (facebookresearch#3056) | * d85601d generatedunixname89002005287564 | fairring, faiss, fairness (4401366386162573988) | * 50be4ea generatedunixname89002005287564 | faiss, falcon (1203443027085661913) | * c8d6f7b Matthijs Douze | fix CI issues after cross-matrix diff (facebookresearch#3042) | * 4699365 Naveen Tatikonda | Fix is_trained in IndexHNSWSQ (facebookresearch#3034) | * 1d6db93 generatedunixname89002005325676 | Daily `arc lint --take CLANGFORMAT` | * 9dc75d0 Matthijs Douze | reduce cross table size (facebookresearch#3012) | * 039409d Matthijs Douze | split off RQ encoding steps to another file (facebookresearch#3011) | * 67d8727 Matthijs Douze | Clean up batch comments + obey IO_FLAG_SKIP_PRECOMPUTE_TABLE (facebookresearch#3013) | * 82352dd Matthijs Douze | make nbits configurable for graph indices based on PQ (facebookresearch#3031) | * 5c4bd3f Matthijs Douze | Cleanup clustering code (facebookresearch#3030) | * 3888f9b Corey J. Nolet | Using expanded distance forms in `RaftFlatIndex.cu` (facebookresearch#3021) | * fef49a6 Richard Barnes | Del `(object)` from 50 inc faic/experiments/blip_finetune/transform/randaugment.py | * c00fe25 Gergely Szilvasy | faiss-gpu-raft, fix dispatch test (facebookresearch#3017) | * a02b37d Gergely Szilvasy | relax test_lut rtol (facebookresearch#3016) | * 69cb877 Matthijs Douze | Fix memory leak for ParameterSpace objects (facebookresearch#3007) | * e3731f7 Gergely Szilvasy | faiss-gpu-raft, the missing bits (facebookresearch#3009) | * 88b7255 qmc20234 | fix argument error (facebookresearch#2965) | * 2768fb3 Gergely Szilvasy | faiss-gpu-raft package (facebookresearch#2992) | * c09992b Maria Lomeli | Back out "Better NaN handling" (facebookresearch#3006) | * e3deb71 Fernando Gasperi | Enable for faiss tests (facebookresearch#3002) | * ef7e945 Gergely Szilvasy | remove avx2 from raft cmake contbuild | * 687457b Matthijs Douze | Access graph structure for NSG (facebookresearch#2984) | * da16d9d Gergely Szilvasy | simplify raft build (facebookresearch#2983) | * a3fbf2d Matthijs Douze | Better NaN handling (facebookresearch#2986) | * a4ddb18 generatedunixname89002005325676 | Daily `arc lint --take CLANGFORMAT` | * 35dac92 Maria | Added version to nighly install (facebookresearch#2982) | * 5a95d47 Alexandr Guzhva | Upgrade AVX2 code for SQ8 (facebookresearch#2942) | * 0aae4d3 youcheng huang | fix hnsw shrink_neighbor_list comment (facebookresearch#2980) | * 7bf7149 Corey J. Nolet | Adding `libraft` dependency to speed up compile times with `USE_RAFT` (facebookresearch#2958) | * 726143d Gergely Szilvasy | install libraft for cmake build (facebookresearch#2968) | * 821a401 Gergely Szilvasy | CodeSet for deduping large datasets (facebookresearch#2949) | * 43d86e3 Matthijs Douze | Relax IVF AQ FastScan (facebookresearch#2940) | * a757806 Maria | added blas=1.0=mkl to INSTALL (facebookresearch#2939) | * d48e777 Sid Jha | Fix import (facebookresearch#2936) | * 1c1d5c8 Matthijs Douze | Make tests a little less verbose | * 4bfdd43 Richard Barnes | Parallelize kernel compilation in FAISS (facebookresearch#2922) | * a91a288 Matthijs Douze | use dispatcher function to call HammingComputer (facebookresearch#2918) | * a27036a Matthijs Douze | add small benchmark for hamming computers | * 391601d Gergely Szilvasy | relax test_ivf_train_2level threshold (facebookresearch#2927) | * 1d7c05d Gergely Szilvasy | raft nightly (facebookresearch#2926) | * 9126f86 Octavian Guzu | Prevent snprintf vulnerability | * 8ac4e41 Richard Barnes | Switch //faiss/gpu to use templates instead of macros (facebookresearch#2914) | * e0741ca Gergely Szilvasy | fix for lib/jvm/languages/python/bin/conda no such file (facebookresearch#2917) | * f69b1db Gergely Szilvasy | update installation instructions with notes about mkl and the nvidia channel | * 07fe2b6 Matthijs Douze | Binary cloning and GPU range search (facebookresearch#2916) | * e153cac Gergely Szilvasy | fix the osx nightly build (facebookresearch#2896) | * 092606b Gergely Szilvasy | bbs producer/consumer threading (facebookresearch#2901) | * d8a6350 I | Update docs (C++11 -> C++17) (facebookresearch#2907) | * 6951466 Gergely Szilvasy | raft enabled cmake build (facebookresearch#2898) | * 27ffd14 Richard Barnes | Use C++17 [[fallthrough]] in faiss/utils/distances_simd.cpp (facebookresearch#2913) | * 100beb8 Richard Barnes | Use C++17 [[fallthrough]] in faiss/utils/hamming_distance/avx2-inl.h | * 463ffd8 Richard Barnes | Indicate that fallthrough is intentional in faiss (facebookresearch#2897) | * 8ec166c Taras Tsugrii | Simplify non-optimal points removal. | * f82298f Taras Tsugrii | Remove unused unordered_map include. (facebookresearch#2900) | * 451f6cd Gergely Szilvasy | c++ 17 (facebookresearch#2899) | * 9c88422 I | Some changes to simdlib (facebookresearch#2885) | * bbc95b1 I | Fix windows CI (facebookresearch#2889) | * 90349f2 Matthijs Douze | Large two-level clustering (facebookresearch#2882) | * 6fd0cb6 Alexandr Guzhva | fix a typo (facebookresearch#2881) | * e8b7575 Alexandr Guzhva | AVX2 version of faiss::HNSW::MinimaxHeap::pop_min() (facebookresearch#2874) | * 6800ebe Matthijs Douze | Support independent IVF coarse quantizer | * a3296f4 Alexandr Guzhva | Use uint8_t instead of uint32_t for faiss::VisitedTable.visno (facebookresearch#2873) | * fd09e51 Matthijs Douze | move by_residual to IndexIVF (facebookresearch#2870) | * 1c1879b Gergely Szilvasy | tiling bfKnn (facebookresearch#2865) | * 5c221ed generatedunixname89002005325676 | Daily `arc lint --take CLANGFORMAT` | * a878c79 Matthijs Douze | Support RAFT from python (facebookresearch#2864) | * 48d48a3 Matthijs Douze | fix windows test (facebookresearch#2862) | * 615e3fc generatedunixname89002005325676 | Daily `arc lint --take CLANGFORMAT` | * b9ea339 Matthijs Douze | support range search from GPU (facebookresearch#2860) | * 54d331e Matthijs Douze | Update INSTALL.md (facebookresearch#2853) | * f809cf0 Gergely Szilvasy | fix osx openmp (facebookresearch#2857) | * eea4a48 Gergely Szilvasy | pin mkl to 2023 (facebookresearch#2856) | * 81f2709 Alexandr Guzhva | enable RAFT under the hood of GPU FAISS (facebookresearch#2840) | * 3219e3d Matthijs Douze | Support Selector for IDMap (facebookresearch#2848) | * 5b17225 Alexandr Guzhva | HNSW speedup + Distance 4 points (facebookresearch#2841) | * f276c47 Alexandr Guzhva | Minor speedup of HNSW (facebookresearch#2832) | * 29318b3 Alexandr Guzhva | improve PQ/IVFPQ code_distance() for avx2 for 4 and 8 subquantizers (facebookresearch#2831) | * bc12015 Alexandr Guzhva | Fix PR problems (facebookresearch#2839) | * 1fb6968 Alexandr Guzhva | upgrade raft version to 23.06 (facebookresearch#2847) | * 2d8886c Matthijs Douze | IVF sorting routine (facebookresearch#2846) | * 4f574d5 Corey J. Nolet | Integrate RAFT FlatIndex / Distances (facebookresearch#2707) | * cbff63e Gergely Szilvasy | addn_query_subset_with_ids float index bug (facebookresearch#2834) | * 155065c Gergely Szilvasy | stop expensive osx arm64 builds on every diff (facebookresearch#2835) | * 3704bbe Matthijs Douze | Add GIST1M to datasets | * 1cb1e54 Alexandr Guzhva | upgrade horizontal sum in distance_single_code for PQ/IVFPQ (facebookresearch#2830) | * d0ba4c0 Alexandr Guzhva | Faster versions of fvec_op_ny_Dx for AVX2 (facebookresearch#2811) ```
abhinavdangeti
added a commit
to blevesearch/faiss
that referenced
this pull request
Aug 14, 2024
…s@bleve * Merging faiss's [v1.8.0](https://github.com/facebookresearch/faiss/releases/tag/v1.8.0) release into bleve branch on fork. * Brings in: ``` |\ | * 943d08b Junjie Qi | Prepare for v.1.8.0 release (facebookresearch#3265) | * abff75e Jeff Johnson | faiss gpu: fix DeviceVector reallocations (facebookresearch#3256) | * b8d91d8 Gergely Szilvasy | fixes for D51074065 | * 27b1055 Tarang Jain | Integrate IVF-PQ from RAFT (facebookresearch#3044) | * 87d43b9 Jim Borden | Fix AVX2 build on Windows (facebookresearch#3238) | * c577f43 Gergely Szilvasy | fix perf test (facebookresearch#3255) | * 8400ece Richard Barnes | Remove unused variables in faiss/utils/partitioning.cpp | * 873b1bc Gergely Szilvasy | fix perf test (facebookresearch#3253) | * 6d00c41 Richard Barnes | Remove unused variables in faiss/utils/hamming_distance/neon-inl.h | * c0624d2 Richard Barnes | Remove unused variables in faiss/invlists/OnDiskInvertedLists.cpp | * a87b432 Richard Barnes | Remove unused variables in faiss/IndexIVFPQ.cpp | * 1338e0d Richard Barnes | Remove unused variables in faiss/IndexBinaryIVF.cpp | * 1b0d274 Richard Barnes | Remove unused variables in faiss/IndexIVF.cpp | * 53fc617 Richard Barnes | Remove unused variables in faiss/tests/test_threaded_index.cpp | * aaca1c2 Richard Barnes | Remove semicolon(s) from 3 files inc facer/engine/utils/UDPServer.h | * a187394 Gergely Szilvasy | libraft 24.02 is released, switching channel | * f262011 Gergely Szilvasy | fix omp parallelism in fast scan range search | * 8898eab yuhaijun999 | Add feature in IndexIDMap.cpp range_search with Parameters. (facebookresearch#3213) | * ebb5f84 Xiaozhong Pan | add a context parameter to InvertedLists and InvertedListsIterator (facebookresearch#3247) | * bfa46a3 Gergely Szilvasy | pin circleci windows image (facebookresearch#3248) | * ed3f6e5 Gergely Szilvasy | faiss-gpu to use 11.4.4 and be the only one supporting P100 (facebookresearch#3237) | * 31a29d0 xinhuitian | HNSW search use efSearch from params if provided (facebookresearch#3233) | * c1822a8 Gergely Szilvasy | cuda dependencies (facebookresearch#3230) | * 1d0e8d4 Gergely Szilvasy | index optimizer (facebookresearch#3154) | * 75ae0bf Richard Barnes | Remove unused variables in faiss/IndexScalarQuantizer.cpp | * 420d25f Maria Lomeli | Index pretransform support in search_preassigned (facebookresearch#3225) | * 31bddeb Gergely Szilvasy | add faiss_gpu lib (facebookresearch#3222) | * 2817344 Gergely Szilvasy | fix ACCESS VIOLATION error when searching using IDSelectorArray | * 67c6a19 Richard Barnes | Remove extra semi colon from faiss/impl/ProductAdditiveQuantizer.h | * 898ce35 Richard Barnes | Remove unused function from faiss/utils/distances_simd.cpp | * 1dcb5d3 Kota Yamaguchi | Add missing header files (facebookresearch#3218) | * 51b6083 Gergely Szilvasy | faiss on rocksdb demo (facebookresearch#3216) | * c4b91a5 Gergely Szilvasy | Replace pickle serialization to address security vulnerability | * a30fd74 Richard Barnes | Remove unused function from faiss/clone_index.cpp | * fc4dbd5 Richard Barnes | Remove extra semi colon from faiss/IndexIVFSpectralHash.cpp | * 6b2c79c Richard Barnes | Remove extra semi colon from faiss/index_factory.cpp | * a7b76a7 Richard Barnes | Fix shadowed variable in faiss/IndexAdditiveQuantizerFastScan.cpp | * ae25b1b Richard Barnes | Fix shadowed variable in faiss/impl/residual_quantizer_encode_steps.cpp | * 7c4fb6d Richard Barnes | Fix shadowed variable in faiss/impl/ResultHandler.h | * 12637a2 Richard Barnes | Fix shadowed variable in faiss/utils/utils.cpp | * 63edc32 Kota Yamaguchi | Fix AVX512 build targets in CMakeLists.txt (facebookresearch#3214) | * 99c0e2b Richard Barnes | Remove unused variables in faiss/utils/sorting.cpp | * e8494f6 Richard Barnes | Remove unused variables in faiss/IndexBinaryHash.cpp | * bbb6ec7 Richard Barnes | Remove unused variables in faiss/IndexShardsIVF.cpp | * 0716bde Richard Barnes | Fix shadowed variable in faiss/IndexNSG.cpp | * bffedda Richard Barnes | Fix shadowed variable in faiss/IndexIDMap.cpp | * a70a8a5 Richard Barnes | Remove unused variables in faiss/impl/HNSW.cpp | * b274cb4 Richard Barnes | Remove unused variables in faiss/IndexIVFFastScan.cpp | * e55a0ac Richard Barnes | Remove unused variables in faiss/impl/index_write.cpp | * a651069 Richard Barnes | Remove unused variables in faiss/impl/index_read.cpp | * e19de27 Richard Barnes | Remove extra semi colon from faiss/IndexIVFFastScan.cpp | * b10eb35 Richard Barnes | Remove extra semi colon from faiss/Clustering.cpp | * 683eadf Richard Barnes | Remove extra semi colon from faiss/utils/distances_simd.cpp | * 522452f Richard Barnes | Remove extra semi colon from faiss/python/python_callbacks.cpp | * 3e666ae Richard Barnes | Remove extra semi colon from faiss/invlists/InvertedLists.cpp | * 7e01b47 Gergely Szilvasy | fix raft build | * 7f3e0a3 Richard Barnes | Mismatch tags take 2 (facebookresearch#3211) | * aff812e Richard Barnes | Fix shadowed variable in faiss/clone_index.cpp | * c540e76 Kota Yamaguchi | Use packaging instead of deprecated distutils for version handling (facebookresearch#3191) | * 091f344 Gergely Szilvasy | add gpu to avx512 (facebookresearch#3210) | * 5e3eae4 Richard Barnes | Remove unused variables in faiss/utils/hamming.cpp | * 739ce13 Matthijs Douze | doc update to mention the paper | * 9a63a3c Richard Barnes | Fix shadowed variable in faiss/IndexFastScan.cpp | * 7dd06dd Richard Barnes | Fix shadowed variable in faiss/impl/NNDescent.cpp | * 0fc8456 Maria Lomeli | Offline IVF powered by faiss big batch search (facebookresearch#3202) | * 4150fb1 Richard Barnes | Fix shadowed variable in faiss/impl/PolysemousTraining.cpp | * 3973017 Richard Barnes | Fix shadowed variable in faiss/IndexBinaryIVF.cpp | * 46320e0 Richard Barnes | Fix shadowed variable in faiss/utils/hamming_distance/avx2-inl.h | * 1be1d32 Richard Barnes | Fix shadowed variable in faiss/impl/io.cpp | * 7442a54 Richard Barnes | Fix shadowed variable in faiss/impl/index_write.cpp | * cfcefc0 Richard Barnes | Remove unused variables in faiss/utils/sorting.cpp | * f884ba2 Richard Barnes | Remove unused variables in faiss/IndexHNSW.cpp | * b7efd3a Gergely Szilvasy | bump libraft version to fix nightly | * c3aa526 Naveen Tatikonda | Add SIMD NEON Optimization for QT_FP16 in Scalar Quantizer (facebookresearch#3166) | * 32f0e8c Matthijs Douze | Generalize ResultHanlder, support range search for HNSW and Fast Scan (facebookresearch#3190) | * 0013c70 Gergely Szilvasy | avx512 CI + conda packages (facebookresearch#3197) | * 0710cbd Richard Barnes | Fix shadowed variable in faiss/impl/ResultHandler.h | * 42b6216 Richard Barnes | Fix shadowed variable in faiss/impl/ScalarQuantizer.cpp | * c399b11 Richard Barnes | Fix shadowed variable in faiss/utils/hamming.cpp | * 952941b Richard Barnes | Fix shadowed variable in faiss/MatrixStats.cpp | * db09984 Richard Barnes | Fix shadowed variable in faiss/impl/simd_result_handlers.h | * beef610 Gergely Szilvasy | faiss paper benchmarks (facebookresearch#3189) | * b7681be Richard Barnes | Remove unused exception parameter from files inc facer/engine/utils/StatsD.cpp | * 449f02c Richard Barnes | Add missing fallthrough to fbcode/faiss/utils/hamming_distance/neon-inl.h | * 0727fff Eddy Li | Adding [[fallthrough]] annotation to case statements in faiss/utils/hamming_distance/neon-inl.h | * 9d74f09 Jason Sylka | Adding [[fallthrough]] annotation to case statements in faiss/utils/hamming_distance/generic-inl.h | * 77c28f8 Gergely Szilvasy | Back out "Offline IVF powered by faiss big batch search" | * 65b08ef Richard Barnes | Remove unused function from faiss/utils/quantize_lut.cpp | * 5621487 Richard Barnes | Remove unused function from faiss/impl/ProductQuantizer.cpp | * 9a8b34e Maria | Offline IVF powered by faiss big batch search (facebookresearch#3175) | * be12427 Maria Lomeli | Upstream changes to big batch search (facebookresearch#3170) | * 79f558f Xiaozhong Pan | set keep_max for InvertedListScanner child classes (facebookresearch#3169) | * 9f8f238 Bader-eddine Ouaich | Fix potential buffer overflow (facebookresearch#3146) | * 9107035 Jaepil Jeong | Fix compile error (facebookresearch#3162) | * d40b39a Alexandr Guzhva | Nit: fix variable types (facebookresearch#3147) | * aebc1d5 Richard Barnes | Remove extra semi colon from faiss/impl/IDSelector.h | * 3dab47c Richard Barnes | Remove extra semi colon from facer/engine/libsvm/svm.cpp | * d2cb016 Richard Barnes | Remove unused exception parameter from files inc faiss/tests/test_approx_topk.cpp | * 5b6c4b4 Matthijs Douze | Back out "printf -> fmt::print in files inc faiss/IndexBinaryHNSW.cpp" (facebookresearch#3164) | * 131adc5 Richard Barnes | printf -> fmt::print in files inc faiss/IndexBinaryHNSW.cpp | * eefa391 Alexandr Guzhva | Introduce avx512 optimization mode and FAISS_OPT_LEVEL env variable (facebookresearch#3150) | * 4c83965 Gergely Szilvasy | benchmark view results (facebookresearch#3144) | * 9519a19 Gergely Szilvasy | benchmark refactor | * a5b03cb Alexandr Guzhva | Fix build on Alpine Linux (facebookresearch#3148) | * 4bf8f93 Yuri Vanin | Add NegativeDistanceComputer::distances_batch_4 override (facebookresearch#3149) | * 90654d6 Gergely Szilvasy | benchmark core faiss prereqs | * 04bb0a8 Alexandr Guzhva | improve ScalarQuantizer performance, ESPECIALLY on old GCC (facebookresearch#3141) | * 43f8220 Matthijs Douze | fix scopedeleter diff | * d3692d2 Alexandr Guzhva | Deprecate ScopeDeleter and ScopeDeleter1 in favor of std::unique_ptr<[]> (facebookresearch#3108) | * eb071f8 luyuncheng | Fix is_trained in IndexNSGSQ (facebookresearch#3145) | * d643c41 Ben Frederickson | use precomputed norms for raft brute_force knn calls (facebookresearch#3089) | * b109d08 Matthijs Douze | Search and return codes (facebookresearch#3143) | * 467f70e Jeff Johnson | Consolidate GPU IVF query tile calculation + special handling for large query memory requirements | * 411c172 Robert Maynard | Add linker script to support large cuda cubin files (facebookresearch#3115) | * 09c7aac Jeff Johnson | Faiss GPU CUDA 12 fix: warp synchronous behavior | * 0c2243c Christopher Ponce de Leon | Revert D51029740: Namespace doesn't need to be followed by semicolon | * 438b519 Richard Barnes | Namespace doesn't need to be followed by semicolon | * 9a66532 Alexandr Guzhva | Add search parameters for IndexRefine::search() and IndexRefineFlat::search() (facebookresearch#3122) | * df7280b pe4eniks | Documentation fixes (facebookresearch#3092) | * 6b76150 chasingegg | Remove confusing comments in partitioning.cpp (facebookresearch#3104) | * 6c89c8b Gergely Szilvasy | fix faiss-gpu nightly | * 0c07a11 Gergely Szilvasy | fix raft contbuild and switch to libraft 23.12 (facebookresearch#3116) | * 9bb6b4b Gergely Szilvasy | fix test TestCrossCodebookComputations::test_precomp | * c3b9374 Gergely Szilvasy | bench_fw - fixes & nits for oss (facebookresearch#3102) | * 0a00d81 Gergely Szilvasy | offline index evaluation (facebookresearch#3097) | * f969d7a Matthijs Douze | better docs | * edcf743 Corey J. Nolet | Integrate IVF-Flat from RAFT (facebookresearch#2521) | * 458633c Robert Maynard | Remove uneeded PTX code generation from libfaiss builds (facebookresearch#3083) | * 2b48901 Matthijs Douze | Remove 1L and 1UL | * 3f3321c Alexandr Guzhva | Small refactoring of inverted lists (facebookresearch#3055) | * 834c543 Haijun Yu | Fix SingleBestResultHandler bug. if IndexFlatL2 empty data then search topk = 1 return label = 0 not -1. (facebookresearch#3075) | * 9db1824 Matthijs Douze | Relax IVFFlatDedup test (facebookresearch#3077) | * 0f18251 generatedunixname89002005325676 | Daily `arc lint --take CLANGFORMAT` | * cf90435 Matthijs Douze | fix flaky GPU test | * e18de23 Alexandr Guzhva | Fix chunk-based processing in ResidualCoarseQuantizer::search() (facebookresearch#3047) | * a1814be Alexandr Guzhva | Simplify dependency components chain (facebookresearch#3058) | * 56b1087 Alexandr Guzhva | move fvec_madd_* functions declarations to a right header (facebookresearch#3054) | * 0780a28 Alexandr Guzhva | Fix a couple of type mismatches (facebookresearch#3059) | * 592f301 Alexandr Guzhva | Unneeded field, exists in a baseclass (facebookresearch#3064) | * 6218111 chasingegg | Fix some typos (facebookresearch#3056) | * d85601d generatedunixname89002005287564 | fairring, faiss, fairness (4401366386162573988) | * 50be4ea generatedunixname89002005287564 | faiss, falcon (1203443027085661913) | * c8d6f7b Matthijs Douze | fix CI issues after cross-matrix diff (facebookresearch#3042) | * 4699365 Naveen Tatikonda | Fix is_trained in IndexHNSWSQ (facebookresearch#3034) | * 1d6db93 generatedunixname89002005325676 | Daily `arc lint --take CLANGFORMAT` | * 9dc75d0 Matthijs Douze | reduce cross table size (facebookresearch#3012) | * 039409d Matthijs Douze | split off RQ encoding steps to another file (facebookresearch#3011) | * 67d8727 Matthijs Douze | Clean up batch comments + obey IO_FLAG_SKIP_PRECOMPUTE_TABLE (facebookresearch#3013) | * 82352dd Matthijs Douze | make nbits configurable for graph indices based on PQ (facebookresearch#3031) | * 5c4bd3f Matthijs Douze | Cleanup clustering code (facebookresearch#3030) | * 3888f9b Corey J. Nolet | Using expanded distance forms in `RaftFlatIndex.cu` (facebookresearch#3021) | * fef49a6 Richard Barnes | Del `(object)` from 50 inc faic/experiments/blip_finetune/transform/randaugment.py | * c00fe25 Gergely Szilvasy | faiss-gpu-raft, fix dispatch test (facebookresearch#3017) | * a02b37d Gergely Szilvasy | relax test_lut rtol (facebookresearch#3016) | * 69cb877 Matthijs Douze | Fix memory leak for ParameterSpace objects (facebookresearch#3007) | * e3731f7 Gergely Szilvasy | faiss-gpu-raft, the missing bits (facebookresearch#3009) | * 88b7255 qmc20234 | fix argument error (facebookresearch#2965) | * 2768fb3 Gergely Szilvasy | faiss-gpu-raft package (facebookresearch#2992) | * c09992b Maria Lomeli | Back out "Better NaN handling" (facebookresearch#3006) | * e3deb71 Fernando Gasperi | Enable for faiss tests (facebookresearch#3002) | * ef7e945 Gergely Szilvasy | remove avx2 from raft cmake contbuild | * 687457b Matthijs Douze | Access graph structure for NSG (facebookresearch#2984) | * da16d9d Gergely Szilvasy | simplify raft build (facebookresearch#2983) | * a3fbf2d Matthijs Douze | Better NaN handling (facebookresearch#2986) | * a4ddb18 generatedunixname89002005325676 | Daily `arc lint --take CLANGFORMAT` | * 35dac92 Maria | Added version to nighly install (facebookresearch#2982) | * 5a95d47 Alexandr Guzhva | Upgrade AVX2 code for SQ8 (facebookresearch#2942) | * 0aae4d3 youcheng huang | fix hnsw shrink_neighbor_list comment (facebookresearch#2980) | * 7bf7149 Corey J. Nolet | Adding `libraft` dependency to speed up compile times with `USE_RAFT` (facebookresearch#2958) | * 726143d Gergely Szilvasy | install libraft for cmake build (facebookresearch#2968) | * 821a401 Gergely Szilvasy | CodeSet for deduping large datasets (facebookresearch#2949) | * 43d86e3 Matthijs Douze | Relax IVF AQ FastScan (facebookresearch#2940) | * a757806 Maria | added blas=1.0=mkl to INSTALL (facebookresearch#2939) | * d48e777 Sid Jha | Fix import (facebookresearch#2936) | * 1c1d5c8 Matthijs Douze | Make tests a little less verbose | * 4bfdd43 Richard Barnes | Parallelize kernel compilation in FAISS (facebookresearch#2922) | * a91a288 Matthijs Douze | use dispatcher function to call HammingComputer (facebookresearch#2918) | * a27036a Matthijs Douze | add small benchmark for hamming computers | * 391601d Gergely Szilvasy | relax test_ivf_train_2level threshold (facebookresearch#2927) | * 1d7c05d Gergely Szilvasy | raft nightly (facebookresearch#2926) | * 9126f86 Octavian Guzu | Prevent snprintf vulnerability | * 8ac4e41 Richard Barnes | Switch //faiss/gpu to use templates instead of macros (facebookresearch#2914) | * e0741ca Gergely Szilvasy | fix for lib/jvm/languages/python/bin/conda no such file (facebookresearch#2917) | * f69b1db Gergely Szilvasy | update installation instructions with notes about mkl and the nvidia channel | * 07fe2b6 Matthijs Douze | Binary cloning and GPU range search (facebookresearch#2916) | * e153cac Gergely Szilvasy | fix the osx nightly build (facebookresearch#2896) | * 092606b Gergely Szilvasy | bbs producer/consumer threading (facebookresearch#2901) | * d8a6350 I | Update docs (C++11 -> C++17) (facebookresearch#2907) | * 6951466 Gergely Szilvasy | raft enabled cmake build (facebookresearch#2898) | * 27ffd14 Richard Barnes | Use C++17 [[fallthrough]] in faiss/utils/distances_simd.cpp (facebookresearch#2913) | * 100beb8 Richard Barnes | Use C++17 [[fallthrough]] in faiss/utils/hamming_distance/avx2-inl.h | * 463ffd8 Richard Barnes | Indicate that fallthrough is intentional in faiss (facebookresearch#2897) | * 8ec166c Taras Tsugrii | Simplify non-optimal points removal. | * f82298f Taras Tsugrii | Remove unused unordered_map include. (facebookresearch#2900) | * 451f6cd Gergely Szilvasy | c++ 17 (facebookresearch#2899) | * 9c88422 I | Some changes to simdlib (facebookresearch#2885) | * bbc95b1 I | Fix windows CI (facebookresearch#2889) | * 90349f2 Matthijs Douze | Large two-level clustering (facebookresearch#2882) | * 6fd0cb6 Alexandr Guzhva | fix a typo (facebookresearch#2881) | * e8b7575 Alexandr Guzhva | AVX2 version of faiss::HNSW::MinimaxHeap::pop_min() (facebookresearch#2874) | * 6800ebe Matthijs Douze | Support independent IVF coarse quantizer | * a3296f4 Alexandr Guzhva | Use uint8_t instead of uint32_t for faiss::VisitedTable.visno (facebookresearch#2873) | * fd09e51 Matthijs Douze | move by_residual to IndexIVF (facebookresearch#2870) | * 1c1879b Gergely Szilvasy | tiling bfKnn (facebookresearch#2865) | * 5c221ed generatedunixname89002005325676 | Daily `arc lint --take CLANGFORMAT` | * a878c79 Matthijs Douze | Support RAFT from python (facebookresearch#2864) | * 48d48a3 Matthijs Douze | fix windows test (facebookresearch#2862) | * 615e3fc generatedunixname89002005325676 | Daily `arc lint --take CLANGFORMAT` | * b9ea339 Matthijs Douze | support range search from GPU (facebookresearch#2860) | * 54d331e Matthijs Douze | Update INSTALL.md (facebookresearch#2853) | * f809cf0 Gergely Szilvasy | fix osx openmp (facebookresearch#2857) | * eea4a48 Gergely Szilvasy | pin mkl to 2023 (facebookresearch#2856) | * 81f2709 Alexandr Guzhva | enable RAFT under the hood of GPU FAISS (facebookresearch#2840) | * 3219e3d Matthijs Douze | Support Selector for IDMap (facebookresearch#2848) | * 5b17225 Alexandr Guzhva | HNSW speedup + Distance 4 points (facebookresearch#2841) | * f276c47 Alexandr Guzhva | Minor speedup of HNSW (facebookresearch#2832) | * 29318b3 Alexandr Guzhva | improve PQ/IVFPQ code_distance() for avx2 for 4 and 8 subquantizers (facebookresearch#2831) | * bc12015 Alexandr Guzhva | Fix PR problems (facebookresearch#2839) | * 1fb6968 Alexandr Guzhva | upgrade raft version to 23.06 (facebookresearch#2847) | * 2d8886c Matthijs Douze | IVF sorting routine (facebookresearch#2846) | * 4f574d5 Corey J. Nolet | Integrate RAFT FlatIndex / Distances (facebookresearch#2707) | * cbff63e Gergely Szilvasy | addn_query_subset_with_ids float index bug (facebookresearch#2834) | * 155065c Gergely Szilvasy | stop expensive osx arm64 builds on every diff (facebookresearch#2835) | * 3704bbe Matthijs Douze | Add GIST1M to datasets | * 1cb1e54 Alexandr Guzhva | upgrade horizontal sum in distance_single_code for PQ/IVFPQ (facebookresearch#2830) | * d0ba4c0 Alexandr Guzhva | Faster versions of fvec_op_ny_Dx for AVX2 (facebookresearch#2811) ```
abhinavdangeti
added a commit
to blevesearch/faiss
that referenced
this pull request
Aug 16, 2024
…s@bleve * Merging faiss's [v1.8.0](https://github.com/facebookresearch/faiss/releases/tag/v1.8.0) release into bleve branch on fork. * Brings in: ``` |\ | * 943d08b Junjie Qi | Prepare for v.1.8.0 release (facebookresearch#3265) | * abff75e Jeff Johnson | faiss gpu: fix DeviceVector reallocations (facebookresearch#3256) | * b8d91d8 Gergely Szilvasy | fixes for D51074065 | * 27b1055 Tarang Jain | Integrate IVF-PQ from RAFT (facebookresearch#3044) | * 87d43b9 Jim Borden | Fix AVX2 build on Windows (facebookresearch#3238) | * c577f43 Gergely Szilvasy | fix perf test (facebookresearch#3255) | * 8400ece Richard Barnes | Remove unused variables in faiss/utils/partitioning.cpp | * 873b1bc Gergely Szilvasy | fix perf test (facebookresearch#3253) | * 6d00c41 Richard Barnes | Remove unused variables in faiss/utils/hamming_distance/neon-inl.h | * c0624d2 Richard Barnes | Remove unused variables in faiss/invlists/OnDiskInvertedLists.cpp | * a87b432 Richard Barnes | Remove unused variables in faiss/IndexIVFPQ.cpp | * 1338e0d Richard Barnes | Remove unused variables in faiss/IndexBinaryIVF.cpp | * 1b0d274 Richard Barnes | Remove unused variables in faiss/IndexIVF.cpp | * 53fc617 Richard Barnes | Remove unused variables in faiss/tests/test_threaded_index.cpp | * aaca1c2 Richard Barnes | Remove semicolon(s) from 3 files inc facer/engine/utils/UDPServer.h | * a187394 Gergely Szilvasy | libraft 24.02 is released, switching channel | * f262011 Gergely Szilvasy | fix omp parallelism in fast scan range search | * 8898eab yuhaijun999 | Add feature in IndexIDMap.cpp range_search with Parameters. (facebookresearch#3213) | * ebb5f84 Xiaozhong Pan | add a context parameter to InvertedLists and InvertedListsIterator (facebookresearch#3247) | * bfa46a3 Gergely Szilvasy | pin circleci windows image (facebookresearch#3248) | * ed3f6e5 Gergely Szilvasy | faiss-gpu to use 11.4.4 and be the only one supporting P100 (facebookresearch#3237) | * 31a29d0 xinhuitian | HNSW search use efSearch from params if provided (facebookresearch#3233) | * c1822a8 Gergely Szilvasy | cuda dependencies (facebookresearch#3230) | * 1d0e8d4 Gergely Szilvasy | index optimizer (facebookresearch#3154) | * 75ae0bf Richard Barnes | Remove unused variables in faiss/IndexScalarQuantizer.cpp | * 420d25f Maria Lomeli | Index pretransform support in search_preassigned (facebookresearch#3225) | * 31bddeb Gergely Szilvasy | add faiss_gpu lib (facebookresearch#3222) | * 2817344 Gergely Szilvasy | fix ACCESS VIOLATION error when searching using IDSelectorArray | * 67c6a19 Richard Barnes | Remove extra semi colon from faiss/impl/ProductAdditiveQuantizer.h | * 898ce35 Richard Barnes | Remove unused function from faiss/utils/distances_simd.cpp | * 1dcb5d3 Kota Yamaguchi | Add missing header files (facebookresearch#3218) | * 51b6083 Gergely Szilvasy | faiss on rocksdb demo (facebookresearch#3216) | * c4b91a5 Gergely Szilvasy | Replace pickle serialization to address security vulnerability | * a30fd74 Richard Barnes | Remove unused function from faiss/clone_index.cpp | * fc4dbd5 Richard Barnes | Remove extra semi colon from faiss/IndexIVFSpectralHash.cpp | * 6b2c79c Richard Barnes | Remove extra semi colon from faiss/index_factory.cpp | * a7b76a7 Richard Barnes | Fix shadowed variable in faiss/IndexAdditiveQuantizerFastScan.cpp | * ae25b1b Richard Barnes | Fix shadowed variable in faiss/impl/residual_quantizer_encode_steps.cpp | * 7c4fb6d Richard Barnes | Fix shadowed variable in faiss/impl/ResultHandler.h | * 12637a2 Richard Barnes | Fix shadowed variable in faiss/utils/utils.cpp | * 63edc32 Kota Yamaguchi | Fix AVX512 build targets in CMakeLists.txt (facebookresearch#3214) | * 99c0e2b Richard Barnes | Remove unused variables in faiss/utils/sorting.cpp | * e8494f6 Richard Barnes | Remove unused variables in faiss/IndexBinaryHash.cpp | * bbb6ec7 Richard Barnes | Remove unused variables in faiss/IndexShardsIVF.cpp | * 0716bde Richard Barnes | Fix shadowed variable in faiss/IndexNSG.cpp | * bffedda Richard Barnes | Fix shadowed variable in faiss/IndexIDMap.cpp | * a70a8a5 Richard Barnes | Remove unused variables in faiss/impl/HNSW.cpp | * b274cb4 Richard Barnes | Remove unused variables in faiss/IndexIVFFastScan.cpp | * e55a0ac Richard Barnes | Remove unused variables in faiss/impl/index_write.cpp | * a651069 Richard Barnes | Remove unused variables in faiss/impl/index_read.cpp | * e19de27 Richard Barnes | Remove extra semi colon from faiss/IndexIVFFastScan.cpp | * b10eb35 Richard Barnes | Remove extra semi colon from faiss/Clustering.cpp | * 683eadf Richard Barnes | Remove extra semi colon from faiss/utils/distances_simd.cpp | * 522452f Richard Barnes | Remove extra semi colon from faiss/python/python_callbacks.cpp | * 3e666ae Richard Barnes | Remove extra semi colon from faiss/invlists/InvertedLists.cpp | * 7e01b47 Gergely Szilvasy | fix raft build | * 7f3e0a3 Richard Barnes | Mismatch tags take 2 (facebookresearch#3211) | * aff812e Richard Barnes | Fix shadowed variable in faiss/clone_index.cpp | * c540e76 Kota Yamaguchi | Use packaging instead of deprecated distutils for version handling (facebookresearch#3191) | * 091f344 Gergely Szilvasy | add gpu to avx512 (facebookresearch#3210) | * 5e3eae4 Richard Barnes | Remove unused variables in faiss/utils/hamming.cpp | * 739ce13 Matthijs Douze | doc update to mention the paper | * 9a63a3c Richard Barnes | Fix shadowed variable in faiss/IndexFastScan.cpp | * 7dd06dd Richard Barnes | Fix shadowed variable in faiss/impl/NNDescent.cpp | * 0fc8456 Maria Lomeli | Offline IVF powered by faiss big batch search (facebookresearch#3202) | * 4150fb1 Richard Barnes | Fix shadowed variable in faiss/impl/PolysemousTraining.cpp | * 3973017 Richard Barnes | Fix shadowed variable in faiss/IndexBinaryIVF.cpp | * 46320e0 Richard Barnes | Fix shadowed variable in faiss/utils/hamming_distance/avx2-inl.h | * 1be1d32 Richard Barnes | Fix shadowed variable in faiss/impl/io.cpp | * 7442a54 Richard Barnes | Fix shadowed variable in faiss/impl/index_write.cpp | * cfcefc0 Richard Barnes | Remove unused variables in faiss/utils/sorting.cpp | * f884ba2 Richard Barnes | Remove unused variables in faiss/IndexHNSW.cpp | * b7efd3a Gergely Szilvasy | bump libraft version to fix nightly | * c3aa526 Naveen Tatikonda | Add SIMD NEON Optimization for QT_FP16 in Scalar Quantizer (facebookresearch#3166) | * 32f0e8c Matthijs Douze | Generalize ResultHanlder, support range search for HNSW and Fast Scan (facebookresearch#3190) | * 0013c70 Gergely Szilvasy | avx512 CI + conda packages (facebookresearch#3197) | * 0710cbd Richard Barnes | Fix shadowed variable in faiss/impl/ResultHandler.h | * 42b6216 Richard Barnes | Fix shadowed variable in faiss/impl/ScalarQuantizer.cpp | * c399b11 Richard Barnes | Fix shadowed variable in faiss/utils/hamming.cpp | * 952941b Richard Barnes | Fix shadowed variable in faiss/MatrixStats.cpp | * db09984 Richard Barnes | Fix shadowed variable in faiss/impl/simd_result_handlers.h | * beef610 Gergely Szilvasy | faiss paper benchmarks (facebookresearch#3189) | * b7681be Richard Barnes | Remove unused exception parameter from files inc facer/engine/utils/StatsD.cpp | * 449f02c Richard Barnes | Add missing fallthrough to fbcode/faiss/utils/hamming_distance/neon-inl.h | * 0727fff Eddy Li | Adding [[fallthrough]] annotation to case statements in faiss/utils/hamming_distance/neon-inl.h | * 9d74f09 Jason Sylka | Adding [[fallthrough]] annotation to case statements in faiss/utils/hamming_distance/generic-inl.h | * 77c28f8 Gergely Szilvasy | Back out "Offline IVF powered by faiss big batch search" | * 65b08ef Richard Barnes | Remove unused function from faiss/utils/quantize_lut.cpp | * 5621487 Richard Barnes | Remove unused function from faiss/impl/ProductQuantizer.cpp | * 9a8b34e Maria | Offline IVF powered by faiss big batch search (facebookresearch#3175) | * be12427 Maria Lomeli | Upstream changes to big batch search (facebookresearch#3170) | * 79f558f Xiaozhong Pan | set keep_max for InvertedListScanner child classes (facebookresearch#3169) | * 9f8f238 Bader-eddine Ouaich | Fix potential buffer overflow (facebookresearch#3146) | * 9107035 Jaepil Jeong | Fix compile error (facebookresearch#3162) | * d40b39a Alexandr Guzhva | Nit: fix variable types (facebookresearch#3147) | * aebc1d5 Richard Barnes | Remove extra semi colon from faiss/impl/IDSelector.h | * 3dab47c Richard Barnes | Remove extra semi colon from facer/engine/libsvm/svm.cpp | * d2cb016 Richard Barnes | Remove unused exception parameter from files inc faiss/tests/test_approx_topk.cpp | * 5b6c4b4 Matthijs Douze | Back out "printf -> fmt::print in files inc faiss/IndexBinaryHNSW.cpp" (facebookresearch#3164) | * 131adc5 Richard Barnes | printf -> fmt::print in files inc faiss/IndexBinaryHNSW.cpp | * eefa391 Alexandr Guzhva | Introduce avx512 optimization mode and FAISS_OPT_LEVEL env variable (facebookresearch#3150) | * 4c83965 Gergely Szilvasy | benchmark view results (facebookresearch#3144) | * 9519a19 Gergely Szilvasy | benchmark refactor | * a5b03cb Alexandr Guzhva | Fix build on Alpine Linux (facebookresearch#3148) | * 4bf8f93 Yuri Vanin | Add NegativeDistanceComputer::distances_batch_4 override (facebookresearch#3149) | * 90654d6 Gergely Szilvasy | benchmark core faiss prereqs | * 04bb0a8 Alexandr Guzhva | improve ScalarQuantizer performance, ESPECIALLY on old GCC (facebookresearch#3141) | * 43f8220 Matthijs Douze | fix scopedeleter diff | * d3692d2 Alexandr Guzhva | Deprecate ScopeDeleter and ScopeDeleter1 in favor of std::unique_ptr<[]> (facebookresearch#3108) | * eb071f8 luyuncheng | Fix is_trained in IndexNSGSQ (facebookresearch#3145) | * d643c41 Ben Frederickson | use precomputed norms for raft brute_force knn calls (facebookresearch#3089) | * b109d08 Matthijs Douze | Search and return codes (facebookresearch#3143) | * 467f70e Jeff Johnson | Consolidate GPU IVF query tile calculation + special handling for large query memory requirements | * 411c172 Robert Maynard | Add linker script to support large cuda cubin files (facebookresearch#3115) | * 09c7aac Jeff Johnson | Faiss GPU CUDA 12 fix: warp synchronous behavior | * 0c2243c Christopher Ponce de Leon | Revert D51029740: Namespace doesn't need to be followed by semicolon | * 438b519 Richard Barnes | Namespace doesn't need to be followed by semicolon | * 9a66532 Alexandr Guzhva | Add search parameters for IndexRefine::search() and IndexRefineFlat::search() (facebookresearch#3122) | * df7280b pe4eniks | Documentation fixes (facebookresearch#3092) | * 6b76150 chasingegg | Remove confusing comments in partitioning.cpp (facebookresearch#3104) | * 6c89c8b Gergely Szilvasy | fix faiss-gpu nightly | * 0c07a11 Gergely Szilvasy | fix raft contbuild and switch to libraft 23.12 (facebookresearch#3116) | * 9bb6b4b Gergely Szilvasy | fix test TestCrossCodebookComputations::test_precomp | * c3b9374 Gergely Szilvasy | bench_fw - fixes & nits for oss (facebookresearch#3102) | * 0a00d81 Gergely Szilvasy | offline index evaluation (facebookresearch#3097) | * f969d7a Matthijs Douze | better docs | * edcf743 Corey J. Nolet | Integrate IVF-Flat from RAFT (facebookresearch#2521) | * 458633c Robert Maynard | Remove uneeded PTX code generation from libfaiss builds (facebookresearch#3083) | * 2b48901 Matthijs Douze | Remove 1L and 1UL | * 3f3321c Alexandr Guzhva | Small refactoring of inverted lists (facebookresearch#3055) | * 834c543 Haijun Yu | Fix SingleBestResultHandler bug. if IndexFlatL2 empty data then search topk = 1 return label = 0 not -1. (facebookresearch#3075) | * 9db1824 Matthijs Douze | Relax IVFFlatDedup test (facebookresearch#3077) | * 0f18251 generatedunixname89002005325676 | Daily `arc lint --take CLANGFORMAT` | * cf90435 Matthijs Douze | fix flaky GPU test | * e18de23 Alexandr Guzhva | Fix chunk-based processing in ResidualCoarseQuantizer::search() (facebookresearch#3047) | * a1814be Alexandr Guzhva | Simplify dependency components chain (facebookresearch#3058) | * 56b1087 Alexandr Guzhva | move fvec_madd_* functions declarations to a right header (facebookresearch#3054) | * 0780a28 Alexandr Guzhva | Fix a couple of type mismatches (facebookresearch#3059) | * 592f301 Alexandr Guzhva | Unneeded field, exists in a baseclass (facebookresearch#3064) | * 6218111 chasingegg | Fix some typos (facebookresearch#3056) | * d85601d generatedunixname89002005287564 | fairring, faiss, fairness (4401366386162573988) | * 50be4ea generatedunixname89002005287564 | faiss, falcon (1203443027085661913) | * c8d6f7b Matthijs Douze | fix CI issues after cross-matrix diff (facebookresearch#3042) | * 4699365 Naveen Tatikonda | Fix is_trained in IndexHNSWSQ (facebookresearch#3034) | * 1d6db93 generatedunixname89002005325676 | Daily `arc lint --take CLANGFORMAT` | * 9dc75d0 Matthijs Douze | reduce cross table size (facebookresearch#3012) | * 039409d Matthijs Douze | split off RQ encoding steps to another file (facebookresearch#3011) | * 67d8727 Matthijs Douze | Clean up batch comments + obey IO_FLAG_SKIP_PRECOMPUTE_TABLE (facebookresearch#3013) | * 82352dd Matthijs Douze | make nbits configurable for graph indices based on PQ (facebookresearch#3031) | * 5c4bd3f Matthijs Douze | Cleanup clustering code (facebookresearch#3030) | * 3888f9b Corey J. Nolet | Using expanded distance forms in `RaftFlatIndex.cu` (facebookresearch#3021) | * fef49a6 Richard Barnes | Del `(object)` from 50 inc faic/experiments/blip_finetune/transform/randaugment.py | * c00fe25 Gergely Szilvasy | faiss-gpu-raft, fix dispatch test (facebookresearch#3017) | * a02b37d Gergely Szilvasy | relax test_lut rtol (facebookresearch#3016) | * 69cb877 Matthijs Douze | Fix memory leak for ParameterSpace objects (facebookresearch#3007) | * e3731f7 Gergely Szilvasy | faiss-gpu-raft, the missing bits (facebookresearch#3009) | * 88b7255 qmc20234 | fix argument error (facebookresearch#2965) | * 2768fb3 Gergely Szilvasy | faiss-gpu-raft package (facebookresearch#2992) | * c09992b Maria Lomeli | Back out "Better NaN handling" (facebookresearch#3006) | * e3deb71 Fernando Gasperi | Enable for faiss tests (facebookresearch#3002) | * ef7e945 Gergely Szilvasy | remove avx2 from raft cmake contbuild | * 687457b Matthijs Douze | Access graph structure for NSG (facebookresearch#2984) | * da16d9d Gergely Szilvasy | simplify raft build (facebookresearch#2983) | * a3fbf2d Matthijs Douze | Better NaN handling (facebookresearch#2986) | * a4ddb18 generatedunixname89002005325676 | Daily `arc lint --take CLANGFORMAT` | * 35dac92 Maria | Added version to nighly install (facebookresearch#2982) | * 5a95d47 Alexandr Guzhva | Upgrade AVX2 code for SQ8 (facebookresearch#2942) | * 0aae4d3 youcheng huang | fix hnsw shrink_neighbor_list comment (facebookresearch#2980) | * 7bf7149 Corey J. Nolet | Adding `libraft` dependency to speed up compile times with `USE_RAFT` (facebookresearch#2958) | * 726143d Gergely Szilvasy | install libraft for cmake build (facebookresearch#2968) | * 821a401 Gergely Szilvasy | CodeSet for deduping large datasets (facebookresearch#2949) | * 43d86e3 Matthijs Douze | Relax IVF AQ FastScan (facebookresearch#2940) | * a757806 Maria | added blas=1.0=mkl to INSTALL (facebookresearch#2939) | * d48e777 Sid Jha | Fix import (facebookresearch#2936) | * 1c1d5c8 Matthijs Douze | Make tests a little less verbose | * 4bfdd43 Richard Barnes | Parallelize kernel compilation in FAISS (facebookresearch#2922) | * a91a288 Matthijs Douze | use dispatcher function to call HammingComputer (facebookresearch#2918) | * a27036a Matthijs Douze | add small benchmark for hamming computers | * 391601d Gergely Szilvasy | relax test_ivf_train_2level threshold (facebookresearch#2927) | * 1d7c05d Gergely Szilvasy | raft nightly (facebookresearch#2926) | * 9126f86 Octavian Guzu | Prevent snprintf vulnerability | * 8ac4e41 Richard Barnes | Switch //faiss/gpu to use templates instead of macros (facebookresearch#2914) | * e0741ca Gergely Szilvasy | fix for lib/jvm/languages/python/bin/conda no such file (facebookresearch#2917) | * f69b1db Gergely Szilvasy | update installation instructions with notes about mkl and the nvidia channel | * 07fe2b6 Matthijs Douze | Binary cloning and GPU range search (facebookresearch#2916) | * e153cac Gergely Szilvasy | fix the osx nightly build (facebookresearch#2896) | * 092606b Gergely Szilvasy | bbs producer/consumer threading (facebookresearch#2901) | * d8a6350 I | Update docs (C++11 -> C++17) (facebookresearch#2907) | * 6951466 Gergely Szilvasy | raft enabled cmake build (facebookresearch#2898) | * 27ffd14 Richard Barnes | Use C++17 [[fallthrough]] in faiss/utils/distances_simd.cpp (facebookresearch#2913) | * 100beb8 Richard Barnes | Use C++17 [[fallthrough]] in faiss/utils/hamming_distance/avx2-inl.h | * 463ffd8 Richard Barnes | Indicate that fallthrough is intentional in faiss (facebookresearch#2897) | * 8ec166c Taras Tsugrii | Simplify non-optimal points removal. | * f82298f Taras Tsugrii | Remove unused unordered_map include. (facebookresearch#2900) | * 451f6cd Gergely Szilvasy | c++ 17 (facebookresearch#2899) | * 9c88422 I | Some changes to simdlib (facebookresearch#2885) | * bbc95b1 I | Fix windows CI (facebookresearch#2889) | * 90349f2 Matthijs Douze | Large two-level clustering (facebookresearch#2882) | * 6fd0cb6 Alexandr Guzhva | fix a typo (facebookresearch#2881) | * e8b7575 Alexandr Guzhva | AVX2 version of faiss::HNSW::MinimaxHeap::pop_min() (facebookresearch#2874) | * 6800ebe Matthijs Douze | Support independent IVF coarse quantizer | * a3296f4 Alexandr Guzhva | Use uint8_t instead of uint32_t for faiss::VisitedTable.visno (facebookresearch#2873) | * fd09e51 Matthijs Douze | move by_residual to IndexIVF (facebookresearch#2870) | * 1c1879b Gergely Szilvasy | tiling bfKnn (facebookresearch#2865) | * 5c221ed generatedunixname89002005325676 | Daily `arc lint --take CLANGFORMAT` | * a878c79 Matthijs Douze | Support RAFT from python (facebookresearch#2864) | * 48d48a3 Matthijs Douze | fix windows test (facebookresearch#2862) | * 615e3fc generatedunixname89002005325676 | Daily `arc lint --take CLANGFORMAT` | * b9ea339 Matthijs Douze | support range search from GPU (facebookresearch#2860) | * 54d331e Matthijs Douze | Update INSTALL.md (facebookresearch#2853) | * f809cf0 Gergely Szilvasy | fix osx openmp (facebookresearch#2857) | * eea4a48 Gergely Szilvasy | pin mkl to 2023 (facebookresearch#2856) | * 81f2709 Alexandr Guzhva | enable RAFT under the hood of GPU FAISS (facebookresearch#2840) | * 3219e3d Matthijs Douze | Support Selector for IDMap (facebookresearch#2848) | * 5b17225 Alexandr Guzhva | HNSW speedup + Distance 4 points (facebookresearch#2841) | * f276c47 Alexandr Guzhva | Minor speedup of HNSW (facebookresearch#2832) | * 29318b3 Alexandr Guzhva | improve PQ/IVFPQ code_distance() for avx2 for 4 and 8 subquantizers (facebookresearch#2831) | * bc12015 Alexandr Guzhva | Fix PR problems (facebookresearch#2839) | * 1fb6968 Alexandr Guzhva | upgrade raft version to 23.06 (facebookresearch#2847) | * 2d8886c Matthijs Douze | IVF sorting routine (facebookresearch#2846) | * 4f574d5 Corey J. Nolet | Integrate RAFT FlatIndex / Distances (facebookresearch#2707) | * cbff63e Gergely Szilvasy | addn_query_subset_with_ids float index bug (facebookresearch#2834) | * 155065c Gergely Szilvasy | stop expensive osx arm64 builds on every diff (facebookresearch#2835) | * 3704bbe Matthijs Douze | Add GIST1M to datasets | * 1cb1e54 Alexandr Guzhva | upgrade horizontal sum in distance_single_code for PQ/IVFPQ (facebookresearch#2830) | * d0ba4c0 Alexandr Guzhva | Faster versions of fvec_op_ny_Dx for AVX2 (facebookresearch#2811) ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
As requested in
#3027
Indeed, PQ sizes with nbits > 8 are good tradeoffs, so it is interesting to support them.
Differential Revision: D48860659