-
Notifications
You must be signed in to change notification settings - Fork 31
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add docs for RPS #328
add docs for RPS #328
Conversation
Signed-off-by: Henry Lindeman <hmlindeman@yahoo.com>
Signed-off-by: Henry Lindeman <hmlindeman@yahoo.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good start. Would sync with Alex on his dedup section - I might include that as it's own doc vs. going deeper on this page.
These search processors include: | ||
|
||
- `debug`: prints the search response to stdout. Useful for debugging. | ||
- `dedup`: works in conjunction with the `Sketcher` ingest transform to deduplicate search results at query-time. See the [sketch](../data_ingestion_and_preparation/transforms/sketch.md) transform for more details. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we put deup first? We also want to link to Alex's docs for this. We should not point to sketch - instead, point to Alex's docs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can say it works in conjunction with Sketcher though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can bump it. was going with alphabetical scheme but can go by importance scheme. I couldn't find anywhere to point to other than sketch. alex confirmed this
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes - it's because Alex hasn't written it yet. That's where all the info is. Can you please add this in today once Alex's docs for query time NDD are written? We want users going to that page, not sketcher.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Alphabetical is fine, I guess.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
will re-point with a new target. Actually will make alex re-point this.
- `debug`: prints the search response to stdout. Useful for debugging. | ||
- `dedup`: works in conjunction with the `Sketcher` ingest transform to deduplicate search results at query-time. See the [sketch](../data_ingestion_and_preparation/transforms/sketch.md) transform for more details. | ||
|
||
These processors run in a microservice in the sycamore ecosystem called RPS. The processors running by default are configured in a config file: [remote-processor-service/config/pipelines.yml](https://github.com/aryn-ai/sycamore/blob/main/apps/remote-processor-service/config/pipelines.yml) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Merge the first sentence ino the first paragraph - this is repetitive.
"The default hybrid search and RAG pipelines (link to each in the docs) include remote processors:"
threshold: 0.4 | ||
``` | ||
|
||
You can add a remote processor to a search pipeline using standard OpenSearch search pipeline creation syntax: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we make it clear you can only use the ones we provide in Sycamore?
|
||
### Configuring RPS | ||
|
||
You can change the config file for RPS. Simply declare a processor, which processors are part of it, and their individual parameters. So, maybe I want a processor that dedupes and prints the search response before and after so I can compare them manually. I can declare this configuration in a new `my-pipelines.yml`: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't know if we want people doing this right now. I might remove it - unless there is a clear use case.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can I put in a
Warning: This is super experimental. Don't do this unless you really need to.
? Also happy to delete
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If there isn't a use case we know, I'd leave this out. Our docs are already complex, let's try to keep it simple. I'd keep this text on your machine, in case we need to add it back. Unless - do you have a use case we think customers would do here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
k, will remove.
Signed-off-by: Henry Lindeman <hmlindeman@yahoo.com>
Signed-off-by: Henry Lindeman <hmlindeman@yahoo.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A few more comments. Thx for the iteration.
@@ -0,0 +1,83 @@ | |||
# Remote Search Processors | |||
|
|||
> 👉 This page assumes you’re familiar with OpenSearch’s concept of a search pipeline and search processor |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is not a friendly customer experience. Instead, write a note guiding somewhere were to go to learn.
These search processors include: | ||
|
||
- `debug`: prints the search response to stdout. Useful for debugging. | ||
- `dedup`: works in conjunction with the `Sketcher` ingest transform to deduplicate search results at query-time. See the [sketch](../data_ingestion_and_preparation/transforms/sketch.md) transform for more details. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes - it's because Alex hasn't written it yet. That's where all the info is. Can you please add this in today once Alex's docs for query time NDD are written? We want users going to that page, not sketcher.
These search processors include: | ||
|
||
- `debug`: prints the search response to stdout. Useful for debugging. | ||
- `dedup`: works in conjunction with the `Sketcher` ingest transform to deduplicate search results at query-time. See the [sketch](../data_ingestion_and_preparation/transforms/sketch.md) transform for more details. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Alphabetical is fine, I guess.
|
||
### Configuring RPS | ||
|
||
You can change the config file for RPS. Simply declare a processor, which processors are part of it, and their individual parameters. So, maybe I want a processor that dedupes and prints the search response before and after so I can compare them manually. I can declare this configuration in a new `my-pipelines.yml`: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If there isn't a use case we know, I'd leave this out. Our docs are already complex, let's try to keep it simple. I'd keep this text on your machine, in case we need to add it back. Unless - do you have a use case we think customers would do here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, pending a nicer note at the top. Thx!
Signed-off-by: Henry Lindeman <hmlindeman@yahoo.com>
There are a couple missing links still