Add documentation about setting a default model for neural search #5121

kolchfa-aws · 2023-09-29T22:12:31Z

Fixes #5060

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

vibrantvarun · 2023-10-03T17:01:00Z

_search-plugins/neural-search.md

-                  "query_text": "Hello world",
-                  "model_id": "xzy76xswsd",
+                "passage_embedding": {
+                  "query_text": "Hi world",


I believe here "Hi planet" would come.

Actually, I didn't want to search for the exact text here because vector search would match both documents anyway. Do you think it makes more sense to search for the exact text?

I think we should always give user a sucessful case so he goes in right direction

Both GET requests return results. I have added the responses to show the results.

vibrantvarun · 2023-10-03T17:01:24Z

_search-plugins/neural-search.md

-              "match": { "passage_text": "Hello world" }
+              "match": {
+                "passage_text": "Hi world"
+              }


"Hi planet"

vibrantvarun · 2023-10-03T17:03:34Z

_search-plugins/neural-search.md

+
+To eliminate passing the model ID with each neural query request, you can set a default model on a k-NN index or a field. 
+
+First, create a [search pipeline]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/index/) with a [`neural_query_enricher`]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/neural-query-enricher/) request processor. To set a default model on an index, provide the model ID in the `default_model_id` parameter. To set a default model on a specific field, provide the field name and the corresponding model ID in the `neural_field_default_id` map:


In this sentence, can you please mention that if both default_model_id and neural_field_default_id is provided then priority would be given to neural_field_default_id .

vibrantvarun · 2023-10-03T17:05:38Z

_search-plugins/neural-search.md

+  "query": {
+    "neural": {
+      "passage_embedding": {
+        "query_text": "Hi world",


vibrantvarun · 2023-10-03T17:06:53Z

_search-plugins/search-pipelines/neural-query-enricher.md

+Field | Data type | Description
+:--- | :--- | :---
+`default_model_id` | String | The model ID of the default model for an index. Optional. You must specify at least one of `default_model_id` and `neural_field_default_id`.
+`neural_field_default_id` | Object | A map of key-value pairs representing document field names and their associated default model IDs. Optional. You must specify at least one of `default_model_id` and `neural_field_default_id`.


If both are given then neural field default Id would be prioritized

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

vagimeli

Great writing! Few comments that should be quick to address/review.

vagimeli · 2023-10-04T15:22:02Z

_search-plugins/search-pipelines/creating-search-pipeline.md

+
+## Updating a search pipeline
+
+To update a search pipeline dynamically, replace the search pipeline using the Search Pipeline API. 


Should we include a link to the search pipeline API documentation?

There's really not a separate API page for this and I think in this case I'm just referring to the /_search/pipeline endpoint so it should be understandable.

vagimeli · 2023-10-04T15:24:19Z

_search-plugins/search-pipelines/index.md

-#### Example request
-
-The following request creates a search pipeline with a `filter_query` request processor that uses a term query to return only public messages and a response processor that renames the field `message` to `notification`:
+To create a search pipeline, send a request to the search pipeline endpoint, specifying an ordered list of processors, which will be applied sequentially:


Great job writing this sentence succinctly. Your wording is perfect.

vagimeli · 2023-10-04T15:26:03Z

_search-plugins/search-pipelines/neural-query-enricher.md

+grand_parent: Search pipelines
+---
+
+# Neural query enrich processor


Should "enrich" be "enricher"?

Technically, yes. I went back and forth on this and decided to go with a more English-like version :) Let's ask @natebower's opinion.

Let's err on the side of technical accuracy and go with "enricher".

vagimeli · 2023-10-04T15:26:34Z

_search-plugins/search-pipelines/neural-query-enricher.md

@@ -0,0 +1,47 @@
+---
+layout: default
+title: Neural query enrich


Should title mirror the wording used in line 10?

Since this page is under the "Search processors" heading in the left nav, I'm just omitting "processor" here so the string is shorter. I do the same with the other processors.

_search-plugins/search-pipelines/neural-query-enricher.md

vagimeli · 2023-10-04T15:32:27Z

_search-plugins/search-pipelines/retrieving-search-pipeline.md

+
+# Retrieving search pipelines
+
+To retrieve the details of an existing search pipeline, use the Search Pipeline API. 


Include link to API documentation?

Same as above: not really anything to link to :)

Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>

natebower

@kolchfa-aws Please see my comments and changes and let me know if you have any questions. Thanks!

_search-plugins/neural-search.md

natebower · 2023-10-04T17:10:49Z

_search-plugins/neural-search.md

-OpenSearch responds with information about your new index:
+## Step 3: Ingest documents into the index
+
+To ingest documents into the index created in the previous section, send a POST request for each document:


"step" instead of "section"?

_search-plugins/neural-search.md

natebower · 2023-10-04T17:14:42Z

_search-plugins/neural-search.md


+First, create a [search pipeline]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/index/) with a [`neural_query_enricher`]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/neural-query-enricher/) request processor. To set a default model on an index, provide the model ID in the `default_model_id` parameter. To set a default model on a specific field, provide the field name and the corresponding model ID in the `neural_field_default_id` map. If you provide both `default_model_id` and `neural_field_default_id`, `neural_field_default_id` takes precedence:


To set a default model "for" an index? To set a default model "for" a specific field?

_search-plugins/search-pipelines/creating-search-pipeline.md

_search-plugins/search-pipelines/neural-query-enricher.md

_search-plugins/search-pipelines/filter-query-processor.md

_search-plugins/search-pipelines/rename-field-processor.md

_search-plugins/search-pipelines/script-processor.md

_search-plugins/search-pipelines/search-processors.md

_search-plugins/neural-search.md

_search-plugins/search-pipelines/neural-query-enricher.md

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>

) * Add documentation about setting a default model for neural search Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add new processor to the processor list Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * More tweaks Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Refactor search pipeline documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Refactor retrieving search pipelines Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add working examples Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Implement tech review comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add responses to documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Update _search-plugins/search-pipelines/neural-query-enricher.md Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> --------- Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

…ensearch-project#5121) * Add documentation about setting a default model for neural search Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add new processor to the processor list Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * More tweaks Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Refactor search pipeline documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Refactor retrieving search pipelines Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add working examples Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Implement tech review comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add responses to documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Update _search-plugins/search-pipelines/neural-query-enricher.md Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> --------- Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Co-authored-by: Nathan Bower <nbower@amazon.com>

) * Add documentation about setting a default model for neural search Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add new processor to the processor list Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * More tweaks Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Refactor search pipeline documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Refactor retrieving search pipelines Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add working examples Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Implement tech review comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add responses to documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Update _search-plugins/search-pipelines/neural-query-enricher.md Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> --------- Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Co-authored-by: Nathan Bower <nbower@amazon.com>

Add documentation about setting a default model for neural search

f220b27

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

kolchfa-aws requested review from cwillum, hdhalter, Naarcha-AWS, vagimeli, ananzh, seanneumann, AMoo-Miki and natebower as code owners September 29, 2023 22:12

Add new processor to the processor list

2d22bd6

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

kolchfa-aws self-assigned this Sep 29, 2023

kolchfa-aws added release-notes PR: Include this PR in the automated release notes v2.11.0 labels Sep 29, 2023

kolchfa-aws added 4 commits September 29, 2023 22:26

More tweaks

bf9d4b5

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

Refactor search pipeline documentation

b717f28

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

Refactor retrieving search pipelines

543ce70

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

Add working examples

225446b

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

vibrantvarun reviewed Oct 3, 2023

View reviewed changes

kolchfa-aws added 2 commits October 3, 2023 13:21

Implement tech review comments

1420ff1

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

Add responses to documentation

e18b296

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

vibrantvarun approved these changes Oct 3, 2023

View reviewed changes

Merge branch 'main' into default-neural-model

b5d9fc6

vagimeli approved these changes Oct 4, 2023

View reviewed changes

Update _search-plugins/search-pipelines/neural-query-enricher.md

6e974ab

Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>

natebower reviewed Oct 4, 2023

View reviewed changes

kolchfa-aws commented Oct 4, 2023

View reviewed changes

_search-plugins/neural-search.md Outdated Show resolved Hide resolved

kolchfa-aws commented Oct 4, 2023

View reviewed changes

_search-plugins/neural-search.md Outdated Show resolved Hide resolved

kolchfa-aws commented Oct 4, 2023

View reviewed changes

_search-plugins/search-pipelines/neural-query-enricher.md Outdated Show resolved Hide resolved

kolchfa-aws commented Oct 4, 2023

View reviewed changes

_search-plugins/search-pipelines/neural-query-enricher.md Outdated Show resolved Hide resolved

Apply suggestions from code review

1cc6a4b

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>

kolchfa-aws merged commit 06527a2 into main Oct 4, 2023
3 checks passed

zhichao-aws mentioned this pull request Mar 12, 2024

[DOC] Default Model Id for Neural Sparse Search Query clause #6652

Closed

4 tasks

kolchfa-aws deleted the default-neural-model branch March 28, 2024 21:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add documentation about setting a default model for neural search #5121

Add documentation about setting a default model for neural search #5121

kolchfa-aws commented Sep 29, 2023

vibrantvarun Oct 3, 2023

kolchfa-aws Oct 3, 2023

vibrantvarun Oct 3, 2023

kolchfa-aws Oct 3, 2023

vibrantvarun Oct 3, 2023

vibrantvarun Oct 3, 2023

vibrantvarun Oct 3, 2023

vibrantvarun Oct 3, 2023

vagimeli left a comment

vagimeli Oct 4, 2023

kolchfa-aws Oct 4, 2023

vagimeli Oct 4, 2023

vagimeli Oct 4, 2023

kolchfa-aws Oct 4, 2023

natebower Oct 4, 2023

vagimeli Oct 4, 2023

kolchfa-aws Oct 4, 2023

vagimeli Oct 4, 2023

kolchfa-aws Oct 4, 2023

natebower left a comment

natebower Oct 4, 2023

natebower Oct 4, 2023


		To eliminate passing the model ID with each neural query request, you can set a default model on a k-NN index or a field.

		First, create a [search pipeline]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/index/) with a [`neural_query_enricher`]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/neural-query-enricher/) request processor. To set a default model on an index, provide the model ID in the `default_model_id` parameter. To set a default model on a specific field, provide the field name and the corresponding model ID in the `neural_field_default_id` map:


		## Updating a search pipeline

		To update a search pipeline dynamically, replace the search pipeline using the Search Pipeline API.


		# Retrieving search pipelines

		To retrieve the details of an existing search pipeline, use the Search Pipeline API.

Add documentation about setting a default model for neural search #5121

Add documentation about setting a default model for neural search #5121

Conversation

kolchfa-aws commented Sep 29, 2023

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vagimeli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

natebower left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment