StreamlitCallbackHandler

[tconkling]

A new implementation of StreamlitCallbackHandler. It formats Agent thoughts into Streamlit expanders.

You can see the handler in action here: https://langchain-mrkl.streamlit.app/

Per a discussion with Harrison, we'll be adding a StreamlitCallbackHandler implementation to an upcoming Streamlit release as well, and will be updating it as we add new LLM- and LangChain-specific features to Streamlit.

The idea with this PR is that the LangChain StreamlitCallbackHandler will "auto-update" in a way that keeps it forward- (and backward-) compatible with Streamlit. If the user has an older Streamlit version installed, the LangChain StreamlitCallbackHandler will be used; if they have a newer Streamlit version that has an updated StreamlitCallbackHandler, that implementation will be used instead.

(I'm opening this as a draft to get the conversation going and make sure we're on the same page. We're really excited to land this into LangChain!)

Who can review?

@agola11, @hwchase17

[vercel]

@tconkling is attempting to deploy a commit to the LangChain Team on Vercel.

A member of the Team first needs to authorize it.

[vowelparrot]

I like this on a high level! Some things that come to mind as I skim this:

• It's explicitly targeting agents (or rather the agent executor state machine). That's great, I think, especially at first, but I wonder if we want to indicate this in any way via the callback naming since it doesn't support arbitrary "chain" or other workflows.
• I may be misreading, but it doesn't seem like it supports at most one thought at a time, right? I think that's fine to start, just want to understand the limits a bit here.

Re: testing, an integration test would be great. If we could mock a couple of the imports, that would be even better, but I'm not sure we are quite that rigorous on the callback integrations yet.

cc @agola11

[vowelparrot]

I think we'll have to lazy import these

[sfc-gh-jcarroll]

Thanks @vowelparrot! I have been working on this with Tim. On the high level feedback:

It's explicitly targeting agents (or rather the agent executor state machine). That's great, I think, especially at first, but I wonder if we want to indicate this in any way via the callback naming since it doesn't support arbitrary "chain" or other workflows.

Yep, we mentioned that in the docstring and would want to call it out in the docs & examples we share too.

• Only concern about changing the name is that we DO want to make it more general later so would that be an annoying name change to deprecate / break in a later version?
• BTW, any ideas on what the right flow / UI would be for this in a non-agent Chain? (like a raw SQLChain or something?) We wracked our brains a bit and weren't totally sure.

I may be misreading, but it doesn't seem like it supports at most one thought at a time, right? I think that's fine to start, just want to understand the limits a bit here.

I'm not sure I understand what you mean by this, could you share a little more detail or an example of what you mean? The callback integration should support one thought or arbitrarily many thoughts (as seen in the example app). Although one thought / no tool use would render fine but be a bit awkward - related to the chain comment above - very open to ideas on how to support this.

Thank you!

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchain	⬜️ Ignored (Inspect)			Jun 22, 2023 5:15pm

[tconkling]

Thanks for the review, @vowelparrot! I'll take care of the nits today, and get started on integration tests.

I'm not able to add Streamlit as an optional dependency because our supported Python version range excludes Python 3.9.7 due to some issue we had with it, and Poetry won't resolve Streamlit because of that 3.9.7 exclusion.

[tconkling]

@vowelparrot (and cc @sfc-gh-jcarroll) - I'm removing the draft status from this PR, and would love another review if you have a chance.

Some notes:

• There's a few unit tests for the "auto-upgrading callback handler" feature that pulls StreamlitCallbackHandler from the streamlit package itself, if it has a newer version of the callback handler
• There's a very basic integration test for the handler. It just runs the "Who is Olivia Wilde's boyfriend" agent query that I saw used in other tests. Obviously, this test requires network access, but:
  • I wrote a (very simple!) tool for testing CallbackHandlers that saves an Agent's callback invocations to a pickle file, and lets you "replay" those callbacks against any set of CallbackHandlers without hitting the network (or spending your token budget). If this would be useful, I'd be happy to add it to the PR and use it in the integration test. (It's just a single small Python file.)
• Re: naming - happy to rename this to StreamlitAgentCallbackHandler or whatever you think is appropriate; just let me know and I'll push the change.

Thanks!

[vowelparrot]

Will review in an hour!

[vowelparrot]

@sfc-gh-jcarroll I think this looks good as is, and with the current name.

In terms of other non-agent flows that are common - I'd say:

• Some fixed retrieval -> response synthesis for Q&A or summarization, so map reduce chain, SQL DB chain
• Information extraction
• Some of the experimental agents (BabyAGI, Generative agents, etc.)

You know your users better than I, so maybe the first two are less important for visualizing in streamlit. Perhaps verifying this works reasonably well with custom agents that don't use the existing agent executor would be good, even if it doesn't impact this PR directly

[sfc-gh-jcarroll]

Awesome!! Anything else needed on our end before this can be merged and released?

We will give those other workflows a try and see what viz makes sense with this callback handler in Streamlit (future work). Thank you!!