add docs for RPS #328

HenryL27 · 2024-04-01T17:23:40Z

There are a couple missing links still

Signed-off-by: Henry Lindeman <hmlindeman@yahoo.com>

jonfritz

Good start. Would sync with Alex on his dedup section - I might include that as it's own doc vs. going deeper on this page.

docs/source/querying_data/remote_processors.md

jonfritz · 2024-04-01T19:42:09Z

docs/source/querying_data/remote_processors.md

+These search processors include:
+
+- `debug`: prints the search response to stdout. Useful for debugging.
+- `dedup`: works in conjunction with the `Sketcher` ingest transform to deduplicate search results at query-time. See the [sketch](../data_ingestion_and_preparation/transforms/sketch.md) transform for more details.


Can we put deup first? We also want to link to Alex's docs for this. We should not point to sketch - instead, point to Alex's docs.

We can say it works in conjunction with Sketcher though.

can bump it. was going with alphabetical scheme but can go by importance scheme. I couldn't find anywhere to point to other than sketch. alex confirmed this

Yes - it's because Alex hasn't written it yet. That's where all the info is. Can you please add this in today once Alex's docs for query time NDD are written? We want users going to that page, not sketcher.

Alphabetical is fine, I guess.

will re-point with a new target. Actually will make alex re-point this.

jonfritz · 2024-04-01T19:43:54Z

docs/source/querying_data/remote_processors.md

+- `debug`: prints the search response to stdout. Useful for debugging.
+- `dedup`: works in conjunction with the `Sketcher` ingest transform to deduplicate search results at query-time. See the [sketch](../data_ingestion_and_preparation/transforms/sketch.md) transform for more details.
+
+These processors run in a microservice in the sycamore ecosystem called RPS. The processors running by default are configured in a config file: [remote-processor-service/config/pipelines.yml](https://github.com/aryn-ai/sycamore/blob/main/apps/remote-processor-service/config/pipelines.yml)


Merge the first sentence ino the first paragraph - this is repetitive.

"The default hybrid search and RAG pipelines (link to each in the docs) include remote processors:"

jonfritz · 2024-04-01T19:44:37Z

docs/source/querying_data/remote_processors.md

+          threshold: 0.4
+```
+
+You can add a remote processor to a search pipeline using standard OpenSearch search pipeline creation syntax:


Should we make it clear you can only use the ones we provide in Sycamore?

jonfritz · 2024-04-01T19:45:10Z

docs/source/querying_data/remote_processors.md

+
+### Configuring RPS
+
+You can change the config file for RPS. Simply declare a processor, which processors are part of it, and their individual parameters. So, maybe I want a processor that dedupes and prints the search response before and after so I can compare them manually. I can declare this configuration in a new `my-pipelines.yml`:


I don't know if we want people doing this right now. I might remove it - unless there is a clear use case.

Can I put in a

Warning: This is super experimental. Don't do this unless you really need to.

? Also happy to delete

If there isn't a use case we know, I'd leave this out. Our docs are already complex, let's try to keep it simple. I'd keep this text on your machine, in case we need to add it back. Unless - do you have a use case we think customers would do here?

k, will remove.

Signed-off-by: Henry Lindeman <hmlindeman@yahoo.com>

jonfritz

A few more comments. Thx for the iteration.

jonfritz · 2024-04-01T22:29:44Z

docs/source/querying_data/remote_processors.md

@@ -0,0 +1,83 @@
+# Remote Search Processors
+
+> 👉 This page assumes you’re familiar with OpenSearch’s concept of a search pipeline and search processor


This is not a friendly customer experience. Instead, write a note guiding somewhere were to go to learn.

jonfritz · 2024-04-01T22:31:42Z

docs/source/querying_data/remote_processors.md

+These search processors include:
+
+- `debug`: prints the search response to stdout. Useful for debugging.
+- `dedup`: works in conjunction with the `Sketcher` ingest transform to deduplicate search results at query-time. See the [sketch](../data_ingestion_and_preparation/transforms/sketch.md) transform for more details.


Yes - it's because Alex hasn't written it yet. That's where all the info is. Can you please add this in today once Alex's docs for query time NDD are written? We want users going to that page, not sketcher.

jonfritz · 2024-04-01T22:32:01Z

docs/source/querying_data/remote_processors.md

+These search processors include:
+
+- `debug`: prints the search response to stdout. Useful for debugging.
+- `dedup`: works in conjunction with the `Sketcher` ingest transform to deduplicate search results at query-time. See the [sketch](../data_ingestion_and_preparation/transforms/sketch.md) transform for more details.


Alphabetical is fine, I guess.

jonfritz · 2024-04-01T22:33:01Z

docs/source/querying_data/remote_processors.md

+
+### Configuring RPS
+
+You can change the config file for RPS. Simply declare a processor, which processors are part of it, and their individual parameters. So, maybe I want a processor that dedupes and prints the search response before and after so I can compare them manually. I can declare this configuration in a new `my-pipelines.yml`:


If there isn't a use case we know, I'd leave this out. Our docs are already complex, let's try to keep it simple. I'd keep this text on your machine, in case we need to add it back. Unless - do you have a use case we think customers would do here?

jonfritz

LGTM, pending a nicer note at the top. Thx!

Signed-off-by: Henry Lindeman <hmlindeman@yahoo.com>

HenryL27 requested a review from jonfritz April 1, 2024 17:23

add docs for RPS. some missing links

f957baa

Signed-off-by: Henry Lindeman <hmlindeman@yahoo.com>

HenryL27 force-pushed the rps-docs branch from 184448b to f957baa Compare April 1, 2024 17:24

HenryL27 added 2 commits April 1, 2024 11:28

fix links

1d4573a

Signed-off-by: Henry Lindeman <hmlindeman@yahoo.com>

make http requests not hideous

68dcc41

Signed-off-by: Henry Lindeman <hmlindeman@yahoo.com>

jonfritz requested changes Apr 1, 2024

View reviewed changes

HenryL27 added 2 commits April 1, 2024 14:37

address PR comments. ish

ead36f6

Signed-off-by: Henry Lindeman <hmlindeman@yahoo.com>

fix a punctuation

0f46023

Signed-off-by: Henry Lindeman <hmlindeman@yahoo.com>

HenryL27 requested a review from jonfritz April 1, 2024 21:56

jonfritz reviewed Apr 1, 2024

View reviewed changes

jonfritz approved these changes Apr 1, 2024

View reviewed changes

dont insult the customer

4ca244b

Signed-off-by: Henry Lindeman <hmlindeman@yahoo.com>

HenryL27 merged commit 5da99d4 into main Apr 1, 2024
12 checks passed

HenryL27 deleted the rps-docs branch April 1, 2024 23:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add docs for RPS #328

add docs for RPS #328

HenryL27 commented Apr 1, 2024

jonfritz left a comment

jonfritz Apr 1, 2024

jonfritz Apr 1, 2024

HenryL27 Apr 1, 2024

jonfritz Apr 1, 2024

jonfritz Apr 1, 2024

HenryL27 Apr 1, 2024

jonfritz Apr 1, 2024

jonfritz Apr 1, 2024

jonfritz Apr 1, 2024

HenryL27 Apr 1, 2024

jonfritz Apr 1, 2024

HenryL27 Apr 1, 2024

jonfritz left a comment

jonfritz Apr 1, 2024

jonfritz Apr 1, 2024

jonfritz Apr 1, 2024

jonfritz Apr 1, 2024

jonfritz left a comment


		### Configuring RPS

		You can change the config file for RPS. Simply declare a processor, which processors are part of it, and their individual parameters. So, maybe I want a processor that dedupes and prints the search response before and after so I can compare them manually. I can declare this configuration in a new `my-pipelines.yml`:

		@@ -0,0 +1,83 @@
		# Remote Search Processors

		> 👉 This page assumes you’re familiar with OpenSearch’s concept of a search pipeline and search processor

add docs for RPS #328

add docs for RPS #328

Conversation

HenryL27 commented Apr 1, 2024

jonfritz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonfritz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonfritz left a comment

Choose a reason for hiding this comment