Choice of “vector similarity search engine”: amazon-elasticsearch-knn, faiss or milvus?

[zhfkt]

Hi,

I know it is wierd to ask this question on the milvus comunity because the author will finally recommend/advertise milvus. But I still would like to ask because I could not find a place to discuss this question on the internet.

I am building a personal system to provide the similarity result among large scale (1 to 10 millions) vectors. I searched and found there are lots of libraries to provide/service this functionality.

• Faiss - https://github.com/facebookresearch/faiss

• Amazon-elasticsearch-knn - https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/knn.html

• Milvus - https://github.com/milvus-io/milvus

It seems that Amazon-elasticsearch-knn will occupy more memory compared with the other 2 libraries from :

• https://medium.com/@kumon/how-to-realize-similarity-search-with-elasticsearch-3dd5641b9adb

• https://medium.com/@kumon/similarity-search-and-similar-image-search-in-elasticsearch-14552a8a8dea

My hope is to save the expenses on the cloud infrasture and spend less maintainence on the code/cloud because it is only a personal project. But I am not sure because I didn't used these tools/services myself. So I could not compare them and get which library is better and I should choose: amazon-elasticsearch-knn, faiss or milvus ?

Put the same on the https://stackoverflow.com/questions/64493655/the-choice-of-vector-similarity-search-engine-amazon-elasticsearch-knn-faiss

[yhmo]

@zhfkt
Could you tell me the dimension of your vectors?
10 millions vectors is not a large dataset, a single milvus server can easily handle it, except your vector dimension is extremely high.
Milvus is a service built upon similarity-search libraries, try define unified interface to wrap them. With Milvus, you don't need to worry about how to integrate these libraries to your application, just focus on your business.

[zhfkt]

Thank your for replying. The dimension may not be larger than 1000 I think.