Add New Retriever Interface with Callbacks #5962

vowelparrot · 2023-06-09T21:54:37Z

Handle the new retriever events in a way that (I think) is entirely backwards compatible? Needs more testing for some of the chain changes and all.

This creates an entire new run type, however. We could also just treat this as an event within a chain run presumably (same with memory)

Adds a subclass initializer that upgrades old retriever implementations to the new schema, along with tests to ensure they work.

First commit doesn't upgrade any of our retriever implementations (to show that we can pass the tests along with additional ones testing the upgrade logic).

Second commit upgrades the known universe of retrievers in langchain.

Add callback handling methods for retriever start/end/error (open to renaming to 'retrieval' if you want that)
Update BaseRetriever schema to support callbacks
Tests for upgrading old "v1" retrievers for backwards compatibility
Update existing retriever implementations to implement the new interface
Update calls within chains to .{a]get_relevant_documents to pass the child callback manager
Update the notebooks/docs to reflect the new interface
Test notebooks thoroughly

Not handled:

Memory pass throughs: retrieval memory doesn't have a parent callback manager passed through the method

vowelparrot · 2023-06-09T21:56:54Z

langchain/schema.py

+    _new_arg_supported: bool = False
+    _expects_other_args: bool = False
+
+    def __init_subclass__(cls, **kwargs: Any) -> None:


This is where the magic is injected @hwchase17 @agola11 @nfcampos

vercel · 2023-06-30T15:20:53Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchain	⬜️ Ignored (Inspect)			Jun 30, 2023 9:39pm

nfcampos

lgtm

hwchase17 · 2023-06-30T15:46:04Z

langchain/chains/conversational_retrieval/base.py

@@ -107,7 +114,13 @@ def _call(
            )
        else:
            new_question = question
-        docs = self._get_docs(new_question, inputs)
+        accepts_run_manager = (


imo this may be overkill. this only matters if someone has subclassed this and overridden _get_docs right?

Yeah, though _get_docs is abstract so it would be anyone who subclasses this outside our repo. I don't know if it is the case

…trievers, adding tag arg to get_child()

baskaryan · 2023-06-30T23:21:31Z

langchain/schema.py

-        """Get documents relevant for a query.
-
+    def _get_relevant_documents(
+        self, query: str, *, run_manager: CallbackManagerForRetrieverRun, **kwargs: Any


ahhh are we sure we wanna introduce arbitrary kwargs here? something we've previously discussed and ik @hwchase17 wanted to keep the interface tight/minimal

rlancemartin · 2023-07-03T03:37:12Z

This introduced unexpected behavior for some retrievers.

Running the MultiQueryRetriever notebook.

I no longer see expected logging when running:

unique_docs = retriever_from_llm.get_relevant_documents(query="What does the course say about regression?")
len(unique_docs)

For MultiQueryRetriever, get_relevant_documents does a few things (PR here).

E.g., it will run queries = self.generate_queries(query, run_manager) and log the queries.

It appears the method name has been changed to _get_relevant_documents?

And it now requires some additional args (e.g., run_manager)?

AFAICT, get_relevant_documents currently will just do retrieval w/o any of the MultiQueryRetriever logic.

If this is expected, then documentation will need to be updated for MultiQueryRetriever here to use _get_relevant_documents and supply run_manager and explain what run_manager is.

Also this exposes that fact that we need a test for MultiQueryRetriever. Will add.

* Add an easier-to-run example. * Add logging per #6891. * Updated params per #5962. --------- Co-authored-by: R. Lance Martin <rlm@Rs-MacBook-Pro.local> Co-authored-by: Lance Martin <lance@langchain.dev>

Handle the new retriever events in a way that (I think) is entirely backwards compatible? Needs more testing for some of the chain changes and all. This creates an entire new run type, however. We could also just treat this as an event within a chain run presumably (same with memory) Adds a subclass initializer that upgrades old retriever implementations to the new schema, along with tests to ensure they work. First commit doesn't upgrade any of our retriever implementations (to show that we can pass the tests along with additional ones testing the upgrade logic). Second commit upgrades the known universe of retrievers in langchain. - [X] Add callback handling methods for retriever start/end/error (open to renaming to 'retrieval' if you want that) - [X] Update BaseRetriever schema to support callbacks - [X] Tests for upgrading old "v1" retrievers for backwards compatibility - [X] Update existing retriever implementations to implement the new interface - [X] Update calls within chains to .{a]get_relevant_documents to pass the child callback manager - [X] Update the notebooks/docs to reflect the new interface - [X] Test notebooks thoroughly Not handled: - Memory pass throughs: retrieval memory doesn't have a parent callback manager passed through the method --------- Co-authored-by: Nuno Campos <nuno@boringbits.io> Co-authored-by: William Fu-Hinthorn <13333726+hinthornw@users.noreply.github.com>

vowelparrot commented Jun 9, 2023

View reviewed changes

vowelparrot changed the title ~~Add New Retriever Interface~~ [RFC2] Add New Retriever Interface Jun 9, 2023

vowelparrot marked this pull request as draft June 9, 2023 22:01

Add New Retriever Interface

808cf7e

vowelparrot force-pushed the vwp/retrieval_callbacks_v3 branch from 5735a14 to 808cf7e Compare June 9, 2023 22:08

Update Implementations

e5fa935

vowelparrot force-pushed the vwp/retrieval_callbacks_v3 branch from adca72e to e5fa935 Compare June 10, 2023 00:01

vowelparrot added 2 commits June 9, 2023 17:10

Update nested retrievers

33d1403

Update _get_docs in Chains

29b00fb

vowelparrot requested review from agola11, hwchase17, dev2049 and nfcampos and removed request for hwchase17 June 10, 2023 00:36

nfcampos added 2 commits June 30, 2023 16:20

Merge branch 'master' into vwp/retrieval_callbacks_v3

ec45a64

Add tag arg to get_child

a197db3

nfcampos approved these changes Jun 30, 2023

View reviewed changes

hwchase17 reviewed Jun 30, 2023

View reviewed changes

nfcampos added 2 commits June 30, 2023 17:43

Passing callbacks to document compressors, patching recently added re…

be0f301

…trievers, adding tag arg to get_child()

Format

95e789f

hinthornw marked this pull request as ready for review June 30, 2023 17:14

hinthornw added 3 commits June 30, 2023 10:29

future

eddbc0e

lint type fix

2d11738

Update docs

bc9257a

langchain-ai deleted a comment from hinthornw Jun 30, 2023

vowelparrot force-pushed the vwp/retrieval_callbacks_v3 branch from 9e67686 to 6f8fbf1 Compare June 30, 2023 19:21

Add doc and import update

4b7d641

vowelparrot force-pushed the vwp/retrieval_callbacks_v3 branch from 6f8fbf1 to 4b7d641 Compare June 30, 2023 19:24

hinthornw changed the title ~~[RFC2] Add New Retriever Interface~~ Add New Retriever Interface with Callbacks Jun 30, 2023

switch class

d2c968c

hinthornw merged commit b0859c9 into master Jun 30, 2023

hinthornw deleted the vwp/retrieval_callbacks_v3 branch June 30, 2023 21:44

baskaryan reviewed Jun 30, 2023

View reviewed changes

This was referenced Jul 3, 2023

MultiQueryRetriever broken on master and needs tests #7063

Closed

Minor updates to notebook for MultiQueryRetriever #7102

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add New Retriever Interface with Callbacks #5962

Add New Retriever Interface with Callbacks #5962

vowelparrot commented Jun 9, 2023 •

edited by hinthornw

Loading

vowelparrot Jun 9, 2023 •

edited

Loading

vercel bot commented Jun 30, 2023 •

edited

Loading

nfcampos left a comment

hwchase17 Jun 30, 2023

hinthornw Jun 30, 2023

baskaryan Jun 30, 2023

rlancemartin commented Jul 3, 2023 •

edited

Loading

Add New Retriever Interface with Callbacks #5962

Add New Retriever Interface with Callbacks #5962

Conversation

vowelparrot commented Jun 9, 2023 • edited by hinthornw Loading

vowelparrot Jun 9, 2023 • edited Loading

Choose a reason for hiding this comment

vercel bot commented Jun 30, 2023 • edited Loading

nfcampos left a comment

Choose a reason for hiding this comment

hwchase17 Jun 30, 2023

Choose a reason for hiding this comment

hinthornw Jun 30, 2023

Choose a reason for hiding this comment

baskaryan Jun 30, 2023

Choose a reason for hiding this comment

rlancemartin commented Jul 3, 2023 • edited Loading

vowelparrot commented Jun 9, 2023 •

edited by hinthornw

Loading

vowelparrot Jun 9, 2023 •

edited

Loading

vercel bot commented Jun 30, 2023 •

edited

Loading

rlancemartin commented Jul 3, 2023 •

edited

Loading