Introduction
This Issue provides a high level overview on Efficient Filtering support for K-NN native engines. Due to the limited support filtering in Native Engines we will be enabling Efficient filtering in Faiss only.
Background
The OpenSearch K-NN plugin supports 3 different type of Engines to perform the Approximate Nearest Neighbor Search. Engines is just an abstraction provided by the plugin over what downstream libraries we will use to do the Nearest Neighbor Search. Currently we have Lucene(Java Implementation), Faiss(C++ implementation) and Nmslib(C++ implementation) as 3 different engines.
Every engine supports various algorithms do the Search. On high level we support:
- Lucene: HNSW algorithm
- Nmslib: HNSW algorithm
- Faiss: HNSW and IVF.
For more details you can read this documentation: https://opensearch.org/docs/latest/search-plugins/knn/knn-index/
K-NN Architecture For Native Engines(Indexing and Search)
On a very high level, an OpenSearch index data is stored in Shards. Shards are nothing but Lucene Indexes. Each shard contains segments. These segments are immutable once they are created. For indices that have K-NN fields in it the architecture is same. K-NN plugin uses the same architecture to support the ANN search. At a very high level during the segment creation, apart from creating all the different data structures(Like FST, BKDs, DocValues etc) needed for different fields, for a K-NN field for HNSW algorithm K-NN plugin creates the HNSW graph files per K-NN Vector Field. These files are written down as segment files.
While performing the ANN Search, we load these HNSW files from disk into Memory(not JVM Heap) if not present already and then perform the Search using respective Libraries.
Filtering
As of OpenSearch 2.8 version K-NN plugin support various types of filters(ref doc). But the type of filtering that we are proposing as part of this document is only present with Lucene engine. None of the Native Engines(Faiss and NmsLib) doesn’t support it.
What is filtering and why it is important?
Filtering is a way that lets users to restrict their search to a certain part of their data. In terms of vector search: The goal is to return the nearest neighbor for a query(containing query vector and filter) among the data points that satisfy the filter specified with query. Lets try to understand this with an example specific to vector search:
Let’s say we have an index where we are storing the product catalog, where images are represented as vectors, along with the same index we are storing the ratings, date when it was uploaded, total review etc. The customer of that application is trying to search for similar products(providing that as a vector) but wants only that products which has Rating >= 4.
So, to honor such queries we need filtering along with Vector Search.
What is Efficient Filtering?
In Filtering world, there are basically 2 types of filtering, lets try to understand then in terms of Vector Search:
- Pre-Filtering: Pre filtering is mainly means, for a given query with filter you first apply the filters on your complete corpus to get the filtered documents and now on those filtered documents you do the Vector search. In general vector search on those filtered documents can be either be done via Exact Search or creating a new HNSW graph with these filtered Ids during runtime and then do a search.
- Post Filtering: Post filtering is mainly means, for a given query with filter you first do the Vector Search and then what ever documents you get apply filter on top of it. This has obvious problems because now the total number of results can be < k if we apply filters.
Efficient Filtering: Efficient filtering is an improvement over pre-filtering especially, where main idea
- we apply the filters first to get the filterIds and while doing the ANN on the whole corpus we only consider the docIds which are present in the filterIds set.
- We intelligently figure out when do ANN search on whole corpus with FilterIds and when do the exact search. The main idea behind doing this is in worst case you need to look at at-least K docs while doing ANN, but if the filterIds < K doing an exact search is efficient both in terms of latency and recall.
Note: The ideas discussed in this section around efficient filtering is not the only things we are considering in efficient filtering. Detail explanations are provided in the doc over the same, which are specific to OpenSearch K-NN Plugin.
Why we need Filtering in Native Engines?
There are couple of limitation with Lucene because of which we want add Filtering in Native engines:
- Lucene has limitation on the number of dimensions which 1024. With current boom in LLM space, the vector dimensions > 1024(Open AI has 1500 dimensions). So customer who want to use OpenSearch K-NN plugin won’t be able to take full advantage of K-NN plugin. There has been issues cut to lucene to increase the dimensions size to 2048 but seems like Lucene community is bit divided there. Currently native engines support 16K dimensions.
- Lucene doesn’t allow us to change the ef_search parameter while doing the search. This parameter has huge impact on both latency and recall. For filtering this will be more visible. Exp5 shows that.
Requirement
Implement Efficient Filtering for Native Engines(along with different engines algorithms) of OpenSearch K-NN plugins.
Out Of Scope
- Based on the deep-dive and POCs(refer POC section of this document) done, NMSLib in current form doesn’t have support for doing filtering with ANN Search. Hence filtering will be added only for Faiss Engine. Please refer Future Enhancements section around how we can support filtering on NmsLib engine.
Success Criteria
- Native engine should support all the different types of filter clauses currently supported in OpenSearch and by Lucene Engine.
- The Recall and Latency of the filtered query(both Relaxed and Restrictive Filters) should be in acceptable range.
High Level Flow
Below is a simplified flow/logic on how filtered search will work.
Filtered Search Flow: Users create a k-NN query with filter and calls the OpenSearch _search api to do the query with filter. The query follows the standard Query Route at the Coordinator node and routed to different shards(can be present on different or same Data Nodes). On the data node, the query is written into Lucene based Query Interface for every shard after performing validation checks(like does filter is supported by this KNN engine or not.). On every shard this query gets executed via IndexSearchers. The IndexSearcher runs the K-NN Query on each Shard, where each K-NN Query will build the a filterQuery(Lucene Constant Score Query) and run the query to return weight of the query. This filter weight is passed to KNNWeight where:
- It first resolves the FilterWeight to get FilterDocIds.
- Check if exact search needs to be done or not. If no, it loads the graphs which are not loaded in cache and then call the JNI Layer, which further calls the Faiss Library to do the Search.
- If exact Search is required, we do exact search on the filterIds and then return the results back to IndexSearcher.
Once the results are returned standard OpenSearch Query flows happens, where a list of docIds of length “size”, is send from each shard to coordinator, which then compile the top docs Ids (of length == size parameter) and runs the fetch phase to get more information on these docIds.
Example
Create Index
PUT /hotels-index
{
"settings": {
"index": {
"knn": true,
"knn.algo_param.ef_search": 100,
"number_of_shards": 1,
"number_of_replicas": 0,
"refresh_interval": "1s"
}
},
"mappings": {
"properties": {
"location": {
"type": "knn_vector",
"dimension": 2,
"method": {
"name": "hnsw",
"space_type": "l2",
"engine": "faiss"
}
}
}
}
}
Ingest Documents
POST /_bulk?refresh
{ "index": { "_index": "hotels-index", "_id": "1" } }
{ "location": [5.2, 4.4], "parking" : "true", "rating" : 5 }
{ "index": { "_index": "hotels-index", "_id": "2" } }
{ "location": [5.2, 3.9], "parking" : "false", "rating" : 4 }
{ "index": { "_index": "hotels-index", "_id": "3" } }
{ "location": [4.9, 3.4], "parking" : "true", "rating" : 9 }
{ "index": { "_index": "hotels-index", "_id": "4" } }
{ "location": [4.2, 4.6], "parking" : "false", "rating" : 6}
{ "index": { "_index": "hotels-index", "_id": "5" } }
{ "location": [3.3, 4.5], "parking" : "true", "rating" : 8 }
{ "index": { "_index": "hotels-index", "_id": "6" } }
{ "location": [6.4, 3.4], "parking" : "true", "rating" : 9 }
{ "index": { "_index": "hotels-index", "_id": "7" } }
{ "location": [4.2, 6.2], "parking" : "true", "rating" : 5 }
{ "index": { "_index": "hotels-index", "_id": "8" } }
{ "location": [2.4, 4.0], "parking" : "true", "rating" : 8 }
{ "index": { "_index": "hotels-index", "_id": "9" } }
{ "location": [1.4, 3.2], "parking" : "false", "rating" : 5 }
{ "index": { "_index": "hotels-index", "_id": "10" } }
{ "location": [7.0, 9.9], "parking" : "true", "rating" : 9 }
{ "index": { "_index": "hotels-index", "_id": "11" } }
{ "location": [3.0, 2.3], "parking" : "false", "rating" : 6 }
{ "index": { "_index": "hotels-index", "_id": "12" } }
{ "location": [5.0, 1.0], "parking" : "true", "rating" : 3 }
Query With Filters
POST /hotels-index/_search
{
"size": 10,
"query": {
"knn": {
"location": {
"vector": [5,4],
"k": 10,
"filter": {
"bool": {
"must": [
{
"range": {
"rating": {
"gte": 7,
"lte": 10
}
}
},
{
"term": {
"parking": "true"
}
}
]
}
}
}
}
}
}
Query Without Filters
POST /hotels-index/_search
{
"size": 10,
"query": {
"knn": {
"location": {
"vector": [
5,
4
],
"k": 3
}
}
}
}
Task:
Introduction
This Issue provides a high level overview on Efficient Filtering support for K-NN native engines. Due to the limited support filtering in Native Engines we will be enabling Efficient filtering in Faiss only.
Background
The OpenSearch K-NN plugin supports 3 different type of Engines to perform the Approximate Nearest Neighbor Search. Engines is just an abstraction provided by the plugin over what downstream libraries we will use to do the Nearest Neighbor Search. Currently we have Lucene(Java Implementation), Faiss(C++ implementation) and Nmslib(C++ implementation) as 3 different engines.
Every engine supports various algorithms do the Search. On high level we support:
For more details you can read this documentation: https://opensearch.org/docs/latest/search-plugins/knn/knn-index/
K-NN Architecture For Native Engines(Indexing and Search)
On a very high level, an OpenSearch index data is stored in Shards. Shards are nothing but Lucene Indexes. Each shard contains segments. These segments are immutable once they are created. For indices that have K-NN fields in it the architecture is same. K-NN plugin uses the same architecture to support the ANN search. At a very high level during the segment creation, apart from creating all the different data structures(Like FST, BKDs, DocValues etc) needed for different fields, for a K-NN field for HNSW algorithm K-NN plugin creates the HNSW graph files per K-NN Vector Field. These files are written down as segment files.
While performing the ANN Search, we load these HNSW files from disk into Memory(not JVM Heap) if not present already and then perform the Search using respective Libraries.
Filtering
As of OpenSearch 2.8 version K-NN plugin support various types of filters(ref doc). But the type of filtering that we are proposing as part of this document is only present with Lucene engine. None of the Native Engines(Faiss and NmsLib) doesn’t support it.
What is filtering and why it is important?
Filtering is a way that lets users to restrict their search to a certain part of their data. In terms of vector search: The goal is to return the nearest neighbor for a query(containing query vector and filter) among the data points that satisfy the filter specified with query. Lets try to understand this with an example specific to vector search:
Let’s say we have an index where we are storing the product catalog, where images are represented as vectors, along with the same index we are storing the ratings, date when it was uploaded, total review etc. The customer of that application is trying to search for similar products(providing that as a vector) but wants only that products which has Rating >= 4.
So, to honor such queries we need filtering along with Vector Search.
What is Efficient Filtering?
In Filtering world, there are basically 2 types of filtering, lets try to understand then in terms of Vector Search:
Efficient Filtering: Efficient filtering is an improvement over pre-filtering especially, where main idea
Note: The ideas discussed in this section around efficient filtering is not the only things we are considering in efficient filtering. Detail explanations are provided in the doc over the same, which are specific to OpenSearch K-NN Plugin.
Why we need Filtering in Native Engines?
There are couple of limitation with Lucene because of which we want add Filtering in Native engines:
Requirement
Implement Efficient Filtering for Native Engines(along with different engines algorithms) of OpenSearch K-NN plugins.
Out Of Scope
Success Criteria
High Level Flow
Below is a simplified flow/logic on how filtered search will work.
Filtered Search Flow: Users create a k-NN query with filter and calls the OpenSearch _search api to do the query with filter. The query follows the standard Query Route at the Coordinator node and routed to different shards(can be present on different or same Data Nodes). On the data node, the query is written into Lucene based Query Interface for every shard after performing validation checks(like does filter is supported by this KNN engine or not.). On every shard this query gets executed via IndexSearchers. The IndexSearcher runs the K-NN Query on each Shard, where each K-NN Query will build the a filterQuery(Lucene Constant Score Query) and run the query to return weight of the query. This filter weight is passed to KNNWeight where:
Once the results are returned standard OpenSearch Query flows happens, where a list of docIds of length “size”, is send from each shard to coordinator, which then compile the top docs Ids (of length == size parameter) and runs the fetch phase to get more information on these docIds.
Example
Create Index
Ingest Documents
Query With Filters
Query Without Filters
Task: