Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Interface for remote search index #1524

Closed
3coins opened this issue Mar 8, 2023 · 3 comments
Closed

Interface for remote search index #1524

3coins opened this issue Mar 8, 2023 · 3 comments

Comments

@3coins
Copy link
Contributor

3coins commented Mar 8, 2023 •
edited

Documentation for Indexes covers local indexes that use local loaders and indexes.
https://langchain.readthedocs.io/en/latest/modules/indexes/getting_started.html

Is there an appropriate interface for connecting with a remote search index? Is DocStore the correct interface to implement?
https://github.com/hwchase17/langchain/blob/4f41e20f0970df42d40907ed91f1c6d58a613541/langchain/docstore/base.py#L8

Also, DocStore is missing the top-k search parameter which is needed for it to be usable. Is this something we can add to this interface?
@3coins
Copy link
Contributor Author

3coins commented Mar 8, 2023

cc @ellisonbg
cc @abhjaw

@3coins
Copy link
Contributor Author

3coins commented Mar 9, 2023

@hwchase17
Adding some more context here. I want to integrate AWS Kendra Service with Langchain.
https://aws.amazon.com/kendra/

It is not clear which interface to use for building applications with langchain and Kendra. Kendra is mainly a search index, so applications might want to integrate it in data augmented retrieval and question answering use cases using Langchain. It seems like there are 2 interfaces of interest here.

  1. Docstore, which has a single search api.
  2. VectorsStore which has add_documents interface to augment the index.

We might need a common interface that has these 2 interfaces, but need your advice on the next steps. Thanks for your help.

@ravwojdyla ravwojdyla mentioned this issue Apr 29, 2023
Merged
hwchase17 pushed a commit that referenced this issue Apr 29, 2023
@ravwojdyla
Add DocstoreFn - lookup doc via arbitrary function (#3760)
160bfae 
This **partially** addresses
#1524, but it's also useful
for some of our use cases.

This `DocstoreFn` allows to lookup a document given a function that
accepts the `search` string without the need to implement a custom
`Docstore`.

This could be useful when:
* you don't want to implement a `Docstore` just to provide a custom
`search`
 * it's expensive to construct an `InMemoryDocstore`/dict
 * you retrieve documents from remote sources
 * you just want to reuse existing objects
samching pushed a commit to samching/langchain that referenced this issue May 1, 2023
@ravwojdyla @samching
Add DocstoreFn - lookup doc via arbitrary function (langchain-ai#3760)
bdcf956 
This **partially** addresses
langchain-ai#1524, but it's also useful
for some of our use cases.

This `DocstoreFn` allows to lookup a document given a function that
accepts the `search` string without the need to implement a custom
`Docstore`.

This could be useful when:
* you don't want to implement a `Docstore` just to provide a custom
`search`
 * it's expensive to construct an `InMemoryDocstore`/dict
 * you retrieve documents from remote sources
 * you just want to reuse existing objects
@3coins
Copy link
Contributor Author

3coins commented Jun 27, 2023

Closing, as completed with #6616

@3coins 3coins closed this as completed Jun 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@3coins