New feat: Self Query Retriever #1266
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…anslator, but a basic translator that can be used with pinecone and chroma is included and can be imported
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
dqbd
reviewed
May 16, 2023
langchain/src/output_parsers/expression_type_handlers/string_literal_handler.ts
Show resolved
Hide resolved
langchain/src/output_parsers/expression_type_handlers/call_expression_handler.ts
Outdated
Show resolved
Hide resolved
dqbd
added
lgtm
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit introduces the SelfQueryRetriever, a new class of object translated directly from the python version (https://github.com/hwchase17/langchain/tree/master/langchain/retrievers/self_query) that retrieves documents from vector stores by using a query-constructing LLM chain to write a structured query. This is achieved through the development of a query translator that translates LLM-generated output into a query that the vector stores can understand. I included basic query translators that works with Chroma and Pinecone, though it shouldn't be hard to create new query translators, as long as the retriever/vector store can also accept filter.
To support this functionality, three new parsers have been added:
ExpressionParser: Parses javascript expressions outputted by the LLM. This is achieved by using meriyah package to parse the output. This parser is general enough that it can parse any javascript call expressions.
AsymmetricOutputParser: This is like StructuredOutputParser, but can handle different input and output shape.
StructuredQueryOutputParser: Parses output from query-constructing LLM into structured query that can be used by retrievers as filter. This is a child of the new AsymmetricOutputParser class. The output from this class still needs to be translated by a Translator which then can be used as filter.
Additionally, this commit includes new unit and integration tests, examples, and docs for Pinecone and Chroma self query retrievers.