Add k-NN Faiss filtering documentation #4476

kolchfa-aws · 2023-07-03T17:47:43Z

Description

Fixes #4350

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

navneet1v · 2023-07-17T02:27:20Z

_search-plugins/knn/filter-search-knn.md

@@ -11,12 +11,24 @@ has_math: true

 To refine k-NN results, you can filter a k-NN search using one of the following methods:

- [Scoring script filter](#scoring-script-filter): This approach involves pre-filtering a document set and then running an exact k-NN search on the filtered subset. It does not scale for large filtered subsets.
+- [Efficient k-NN filtering](#efficient-k-nn-filtering): This approach applies filtering _during_ the k-NN search, as opposed to before or after the k-NN search, which ensures that `k` results are returned. This approach is supported by the following search engines:


Suggested change

- [Efficient k-NN filtering](#efficient-k-nn-filtering): This approach applies filtering _during_ the k-NN search, as opposed to before or after the k-NN search, which ensures that `k` results are returned. This approach is supported by the following search engines:

- [Efficient k-NN filtering](#efficient-k-nn-filtering): This approach applies filtering _during_ the k-NN search, as opposed to before or after the k-NN search, which ensures that `k` results are returned(if present). This approach is supported by the following engines:

navneet1v · 2023-07-17T02:27:36Z

_search-plugins/knn/filter-search-knn.md

@@ -11,12 +11,24 @@ has_math: true

 To refine k-NN results, you can filter a k-NN search using one of the following methods:

- [Scoring script filter](#scoring-script-filter): This approach involves pre-filtering a document set and then running an exact k-NN search on the filtered subset. It does not scale for large filtered subsets.
+- [Efficient k-NN filtering](#efficient-k-nn-filtering): This approach applies filtering _during_ the k-NN search, as opposed to before or after the k-NN search, which ensures that `k` results are returned. This approach is supported by the following search engines:
+  - Lucene search engine with a Hierarchical Navigable Small World (HNSW) algorithm (k-NN plugin versions 2.4 and later) 


Suggested change

- Lucene search engine with a Hierarchical Navigable Small World (HNSW) algorithm (k-NN plugin versions 2.4 and later)

- Lucene engine with a Hierarchical Navigable Small World (HNSW) algorithm (k-NN plugin versions 2.4 and later)

navneet1v · 2023-07-17T02:27:49Z

_search-plugins/knn/filter-search-knn.md

- [Scoring script filter](#scoring-script-filter): This approach involves pre-filtering a document set and then running an exact k-NN search on the filtered subset. It does not scale for large filtered subsets.
+- [Efficient k-NN filtering](#efficient-k-nn-filtering): This approach applies filtering _during_ the k-NN search, as opposed to before or after the k-NN search, which ensures that `k` results are returned. This approach is supported by the following search engines:
+  - Lucene search engine with a Hierarchical Navigable Small World (HNSW) algorithm (k-NN plugin versions 2.4 and later) 
+  - Faiss search engine with a Hierarchical Navigable Small World (HNSW) algorithm (k-NN plugin versions 2.9 or later) 


Suggested change

- Faiss search engine with a Hierarchical Navigable Small World (HNSW) algorithm (k-NN plugin versions 2.9 or later)

- Faiss engine with a Hierarchical Navigable Small World (HNSW) algorithm (k-NN plugin versions 2.9 or later)

navneet1v · 2023-07-17T02:29:24Z

_search-plugins/knn/filter-search-knn.md


- [Boolean filter](#boolean-filter-with-ann-search): This approach runs an [approximate nearest neighbor (ANN)]({{site.url}}{{site.baseurl}}/search-plugins/knn/approximate-knn) search and then applies a filter to the results. Because of post-filtering, it may return significantly fewer than `k` results for a restrictive filter.
+-  [Post filtering](#post-filtering): Because it is performed after the k-NN search, this approach may return significantly fewer than `k` results for a restrictive filter.
+    - [Boolean post filter](#boolean-filter-with-ann-search): This approach runs an [approximate nearest neighbor (ANN)]({{site.url}}{{site.baseurl}}/search-plugins/knn/approximate-knn/) search and then applies a filter to the results. The two query parts are executed independently and then the intersection of their result sets is taken. 


Suggested change

- [Boolean post filter](#boolean-filter-with-ann-search): This approach runs an [approximate nearest neighbor (ANN)]({{site.url}}{{site.baseurl}}/search-plugins/knn/approximate-knn/) search and then applies a filter to the results. The two query parts are executed independently and then the intersection of their result sets is taken.

- [Boolean post filter](#boolean-filter-with-ann-search): This approach runs an [approximate nearest neighbor (ANN)]({{site.url}}{{site.baseurl}}/search-plugins/knn/approximate-knn/) search and then applies a filter to the results. The two query parts are executed independently and then the results are combined based on the query operator(should/must etc) provided in the query.

navneet1v · 2023-07-17T02:29:58Z

_search-plugins/knn/filter-search-knn.md


- [Lucene k-NN filter](#using-a-lucene-k-nn-filter): This approach applies filtering _during_ the k-NN search, as opposed to before or after the k-NN search, which ensures that `k` results are returned. You can only use this method with the Hierarchical Navigable Small World (HNSW) algorithm implemented by the Lucene search engine in k-NN plugin versions 2.4 and later.
+- [Scoring script filter](#scoring-script-filter): This approach involves pre-filtering a document set and then running an exact k-NN search on the filtered subset. It does not scale for large filtered subsets. 


Lets also add that, latencies can be high for this query.

navneet1v · 2023-07-17T02:30:40Z

_search-plugins/knn/filter-search-knn.md

+
+Filter | When the filter is applied | Type of search | Supported engines and methods | Where to place the `filter` clause
+:--- | :--- | :--- | :---
+Efficient k-NN filtering | During search (a hybrid of pre- and post-filtering) | Approximate | - `lucene` (`hnsw`) <br> - `faiss` (`hnsw`, `ivf`) | Inside the k-NN query clause.


lets remove ivf from here.

navneet1v · 2023-07-17T02:32:47Z

_search-plugins/knn/filter-search-knn.md

+| 10M | 80 | 100 | Scoring script |Efficient k-NN filtering |
+| 1M | 2.5 | 100 |Efficient k-NN filtering | Scoring script |
+| 1M | 38 | 100 |Efficient k-NN filtering |Efficient k-NN filtering/scoring script |
+| 1M | 80 | 100 | Boolean filter |Efficient k-NN filtering |


Suggested change

| 1M | 80 | 100 | Boolean filter |Efficient k-NN filtering |

| 1M | 80 | 100 | Efficient k-NN filtering |Boolean filter |

navneet1v · 2023-07-17T02:34:07Z

_search-plugins/knn/filter-search-knn.md


-A scoring script filter first filters the documents and then uses a brute-force exact k-NN search on the results. For example, the following query searches for hotels with a rating between 8 and 10, inclusive, that provide parking and then performs a k-NN search to return the 3 hotels that are closest to the specified `location`:
+You can perform efficient k-NN filtering with the `lucene` or `faiss` search engines. 


Suggested change

You can perform efficient k-NN filtering with the `lucene` or `faiss` search engines.

You can perform efficient k-NN filtering with the `lucene` or `faiss` engines.

navneet1v · 2023-07-17T02:36:50Z

_search-plugins/knn/filter-search-knn.md

+
+**Step 3: Search your data with a filter**
+
+Now you can create a k-NN search with filters. <!-- TODO: add details -->


The same thing which we wrote for lucene above is true for filters here too.

Note that there are multiple ways to construct a filter that returns hotels that provide parking, for example: A term query clause in the should clause A wildcard query clause in the should clause A regexp query clause in the should clause A must_not clause to eliminate hotels with parking set to false.

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

vagimeli

LGTM. Minimal edits

_search-plugins/knn/filter-search-knn.md

Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>

natebower

@kolchfa-aws Please see my comments and changes and let me know if you have any questions. Thanks!

_search-plugins/knn/filter-search-knn.md

natebower · 2023-07-17T16:59:57Z

_search-plugins/knn/filter-search-knn.md


- [Boolean filter](#boolean-filter-with-ann-search): This approach runs an [approximate nearest neighbor (ANN)]({{site.url}}{{site.baseurl}}/search-plugins/knn/approximate-knn) search and then applies a filter to the results. Because of post-filtering, it may return significantly fewer than `k` results for a restrictive filter.
+-  [Post filtering](#post-filtering): Because it is performed after the k-NN search, this approach may return significantly fewer than `k` results for a restrictive filter.


The sub-bullets here need to be introduced using a colon.

_search-plugins/knn/filter-search-knn.md

natebower · 2023-07-17T17:04:41Z

_search-plugins/knn/filter-search-knn.md


- [Boolean filter](#boolean-filter-with-ann-search): This approach runs an [approximate nearest neighbor (ANN)]({{site.url}}{{site.baseurl}}/search-plugins/knn/approximate-knn) search and then applies a filter to the results. Because of post-filtering, it may return significantly fewer than `k` results for a restrictive filter.
+-  [Post filtering](#post-filtering): Because it is performed after the k-NN search, this approach may return significantly fewer than `k` results for a restrictive filter.
+    - [Boolean post filter](#boolean-filter-with-ann-search): This approach runs an [approximate nearest neighbor (ANN)]({{site.url}}{{site.baseurl}}/search-plugins/knn/approximate-knn/) search and then applies a filter to the results. The two query parts are executed independently and then the results are combined based on the query operator (`should`, `must`, and so on) provided in the query. 


"Boolean post-filter"? "elements" instead of "parts"? "run" instead of "executed"?

_search-plugins/knn/filter-search-knn.md

natebower · 2023-07-17T17:18:03Z

_search-plugins/knn/filter-search-knn.md

+
+### Using a Faiss efficient filter
+
+Consider an index that holds information about shirts for a clothing store. You want to find the top-rated shirts that are similar to the one you have but would like to restrict the results by shirt size.


Suggested change

Consider an index that holds information about shirts for a clothing store. You want to find the top-rated shirts that are similar to the one you have but would like to restrict the results by shirt size.

Consider an index containing information about a particular brand of shirt. You want to find the top-rated shirts that are similar to one you already have but would like to restrict the results by shirt size.

natebower · 2023-07-17T17:21:23Z

_search-plugins/knn/filter-search-knn.md

@@ -466,4 +507,196 @@ POST /hotels-index/_search
  }
 } 
 ```
+{% include copy-curl.html %}
+
+## Post filtering


This is hyphenated elsewhere. Ensure consistency across docs.

natebower · 2023-07-17T17:21:32Z

_search-plugins/knn/filter-search-knn.md

+
+## Post filtering
+
+You can achieve post filtering with a Boolean filter or by providing the `post_filter` parameter.


natebower · 2023-07-17T17:22:12Z

_search-plugins/knn/filter-search-knn.md

+}
+```
+
+### Post filter parameter


"Post-filter?

_search-plugins/knn/filter-search-knn.md

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Add k-NN Faiss filtering documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Move the note Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add faiss and a filter table Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Refactor boolean filtering section Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Clarified that Faiss works with hnsw only Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add more Faiss filtering information Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Update _search-plugins/knn/filter-search-knn.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Implemented editorial comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Implemented one more editorial comment Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> --------- Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Co-authored-by: Nathan Bower <nbower@amazon.com>

Add k-NN Faiss filtering documentation

a6a1f83

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

kolchfa-aws self-assigned this Jul 3, 2023

kolchfa-aws added the v2.9.0 label Jul 3, 2023

kolchfa-aws added 4 commits July 6, 2023 16:41

Merge branch 'main' into knn-filter-update

467cbb6

Move the note

a76c5cb

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

Add faiss and a filter table

1d0eb6c

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

Refactor boolean filtering section

6686582

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

hdhalter added the release-notes PR: Include this PR in the automated release notes label Jul 13, 2023

Clarified that Faiss works with hnsw only

01dc37c

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

navneet1v reviewed Jul 17, 2023

View reviewed changes

Add more Faiss filtering information

03c1d01

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

kolchfa-aws marked this pull request as ready for review July 17, 2023 16:02

kolchfa-aws requested review from cwillum, hdhalter, Naarcha-AWS, vagimeli, ananzh, seanneumann, AMoo-Miki and natebower as code owners July 17, 2023 16:02

vagimeli approved these changes Jul 17, 2023

View reviewed changes

Apply suggestions from code review

f13f7c8

Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>

navneet1v approved these changes Jul 17, 2023

View reviewed changes

natebower reviewed Jul 17, 2023

View reviewed changes

kolchfa-aws and others added 4 commits July 18, 2023 10:25

Apply suggestions from code review

5ee7ccc

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>

Update _search-plugins/knn/filter-search-knn.md

0b0779a

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>

Implemented editorial comments

85375ee

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

Implemented one more editorial comment

b77debd

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

kolchfa-aws merged commit 6c83dfd into main Jul 18, 2023
4 checks passed

hdhalter mentioned this pull request Sep 25, 2023

[DOC] Feedback on Search topics #4326

Open

29 tasks

prudhvigodithi added release-notes PR: Include this PR in the automated release notes and removed release-notes PR: Include this PR in the automated release notes labels Oct 3, 2023

Naarcha-AWS deleted the knn-filter-update branch March 28, 2024 23:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add k-NN Faiss filtering documentation #4476

Add k-NN Faiss filtering documentation #4476

kolchfa-aws commented Jul 3, 2023 •

edited

Loading

navneet1v Jul 17, 2023

kolchfa-aws Jul 17, 2023

navneet1v Jul 17, 2023

navneet1v Jul 17, 2023

navneet1v Jul 17, 2023

navneet1v Jul 17, 2023

navneet1v Jul 17, 2023

navneet1v Jul 17, 2023

navneet1v Jul 17, 2023

navneet1v Jul 17, 2023

vagimeli left a comment

natebower left a comment

natebower Jul 17, 2023

natebower Jul 17, 2023

natebower Jul 17, 2023

natebower Jul 17, 2023

natebower Jul 17, 2023

natebower Jul 17, 2023

	- [Efficient k-NN filtering](#efficient-k-nn-filtering): This approach applies filtering _during_ the k-NN search, as opposed to before or after the k-NN search, which ensures that `k` results are returned. This approach is supported by the following search engines:
	- [Efficient k-NN filtering](#efficient-k-nn-filtering): This approach applies filtering _during_ the k-NN search, as opposed to before or after the k-NN search, which ensures that `k` results are returned(if present). This approach is supported by the following engines:

	- Lucene search engine with a Hierarchical Navigable Small World (HNSW) algorithm (k-NN plugin versions 2.4 and later)
	- Lucene engine with a Hierarchical Navigable Small World (HNSW) algorithm (k-NN plugin versions 2.4 and later)

	- Faiss search engine with a Hierarchical Navigable Small World (HNSW) algorithm (k-NN plugin versions 2.9 or later)
	- Faiss engine with a Hierarchical Navigable Small World (HNSW) algorithm (k-NN plugin versions 2.9 or later)

	- [Boolean post filter](#boolean-filter-with-ann-search): This approach runs an [approximate nearest neighbor (ANN)]({{site.url}}{{site.baseurl}}/search-plugins/knn/approximate-knn/) search and then applies a filter to the results. The two query parts are executed independently and then the intersection of their result sets is taken.
	- [Boolean post filter](#boolean-filter-with-ann-search): This approach runs an [approximate nearest neighbor (ANN)]({{site.url}}{{site.baseurl}}/search-plugins/knn/approximate-knn/) search and then applies a filter to the results. The two query parts are executed independently and then the results are combined based on the query operator(should/must etc) provided in the query.


		- [Lucene k-NN filter](#using-a-lucene-k-nn-filter): This approach applies filtering _during_ the k-NN search, as opposed to before or after the k-NN search, which ensures that `k` results are returned. You can only use this method with the Hierarchical Navigable Small World (HNSW) algorithm implemented by the Lucene search engine in k-NN plugin versions 2.4 and later.
		- [Scoring script filter](#scoring-script-filter): This approach involves pre-filtering a document set and then running an exact k-NN search on the filtered subset. It does not scale for large filtered subsets.

	\| 1M \| 80 \| 100 \| Boolean filter \|Efficient k-NN filtering \|
	\| 1M \| 80 \| 100 \| Efficient k-NN filtering \|Boolean filter \|


		A scoring script filter first filters the documents and then uses a brute-force exact k-NN search on the results. For example, the following query searches for hotels with a rating between 8 and 10, inclusive, that provide parking and then performs a k-NN search to return the 3 hotels that are closest to the specified `location`:
		You can perform efficient k-NN filtering with the `lucene` or `faiss` search engines.


		Step 3: Search your data with a filter

		Now you can create a k-NN search with filters. <!-- TODO: add details -->


		- [Boolean filter](#boolean-filter-with-ann-search): This approach runs an [approximate nearest neighbor (ANN)]({{site.url}}{{site.baseurl}}/search-plugins/knn/approximate-knn) search and then applies a filter to the results. Because of post-filtering, it may return significantly fewer than `k` results for a restrictive filter.
		- [Post filtering](#post-filtering): Because it is performed after the k-NN search, this approach may return significantly fewer than `k` results for a restrictive filter.


		### Using a Faiss efficient filter

		Consider an index that holds information about shirts for a clothing store. You want to find the top-rated shirts that are similar to the one you have but would like to restrict the results by shirt size.

	Consider an index that holds information about shirts for a clothing store. You want to find the top-rated shirts that are similar to the one you have but would like to restrict the results by shirt size.
	Consider an index containing information about a particular brand of shirt. You want to find the top-rated shirts that are similar to one you already have but would like to restrict the results by shirt size.


		## Post filtering

		You can achieve post filtering with a Boolean filter or by providing the `post_filter` parameter.

Add k-NN Faiss filtering documentation #4476

Add k-NN Faiss filtering documentation #4476

Conversation

kolchfa-aws commented Jul 3, 2023 • edited Loading

Description

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vagimeli left a comment

Choose a reason for hiding this comment

natebower left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kolchfa-aws commented Jul 3, 2023 •

edited

Loading