Implement Chat* composable components by ahuang11 · Pull Request #5333 · holoviz/panel

ahuang11 · 2023-07-27T14:04:51Z

https://github.com/ahuang11/panel-chat-examples

Diagram:

Overview:

ChatMessage is essentially a dataclass, and holds the content of the users data and metadata
ChatEntry is the rendering of the ChatMessage (pane?)
ChatFeed is the container to hold all the ChatEntry(s) (layout?) with the ability to attach a callback (e.g. AI response)
ChatInterface is the highest level interface, composing the ChatFeed and Tabs of Widget(s)

Todo:

codecov · 2023-07-27T14:15:44Z

Codecov Report

Merging #5333 (b088ed4) into main (497ad69) will increase coverage by 0.43%.
The diff coverage is 94.18%.

@@            Coverage Diff             @@
##             main    #5333      +/-   ##
==========================================
+ Coverage   83.50%   83.93%   +0.43%     
==========================================
  Files         275      278       +3     
  Lines       39549    41128    +1579     
==========================================
+ Hits        33024    34522    +1498     
- Misses       6525     6606      +81

Flag	Coverage Δ
ui-tests	`40.16% <23.12%> (-0.65%)`	⬇️
unitexamples-tests	`74.26% <94.18%> (+0.82%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
panel/layout/grid.py	`74.53% <100.00%> (ø)`
panel/pane/base.py	`85.06% <100.00%> (+0.57%)`	⬆️
panel/pane/holoviews.py	`79.80% <100.00%> (+0.03%)`	⬆️
panel/param.py	`85.67% <100.00%> (+0.12%)`	⬆️
panel/pipeline.py	`88.80% <100.00%> (+0.05%)`	⬆️
panel/reactive.py	`80.09% <100.00%> (+0.01%)`	⬆️
panel/tests/widgets/test_chat.py	`100.00% <100.00%> (ø)`
panel/widgets/__init__.py	`100.00% <100.00%> (ø)`
panel/widgets/button.py	`87.06% <ø> (ø)`
panel/widgets/indicators.py	`74.21% <100.00%> (+0.44%)`	⬆️
... and 2 more

... and 4 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

ahuang11 · 2023-08-02T21:40:21Z

I think I have most of the functionality down:

import re
from typing import Any

from panel.io.mime_render import exec_with_return
import pandas as pd
import panel as pn
import openai

DATAFRAME_PROMPT = """
    Here are the columns in your DataFrame: {columns}.
    Create  a plot with hvplot that highlights an interesting
    relationship between the columns with hvplot groupby kwarg.
"""

CODE_REGEX = re.compile(r"```python(.*?)```", re.DOTALL)


async def respond_with_openai(contents: Any):
    # extract the DataFrame
    if isinstance(contents, pd.DataFrame):
        global df
        df = contents
        columns = contents.columns
        message = DATAFRAME_PROMPT.format(columns=columns)
    else:
        message = contents

    # ask OpenAI to plot
    response = await openai.ChatCompletion.acreate(
        model="gpt-3.5-turbo",
        messages=[{"role": "user", "content": message}],
        temperature=0,
        max_tokens=500,
        stream=True,
    )
    message = ""
    async for chunk in response:
        message += chunk["choices"][0]["delta"].get("content", "")
        yield {"user": "OpenAI", "value": message}


async def respond_with_executor(code: str):
    return {
        "user": "Executor",
        "value": exec_with_return(code=code, global_context=globals()),
    }


async def response_callback(
    contents: Any,
    name: str,
    chat_interface: pn.widgets.ChatInterface,
):
    if name == "You":
        async for chunk in respond_with_openai(contents):
            yield chunk
    elif CODE_REGEX.search(contents):
        yield await respond_with_executor(CODE_REGEX.search(contents).group(1))


chat_card = pn.widgets.ChatCard(callback=response_callback)
chat_interface = pn.widgets.ChatInterface(
    value=chat_card, widgets=[pn.widgets.TextInput(), pn.widgets.FileInput()]
)
chat_interface.servable()

Screen.Recording.2023-08-02.at.5.38.54.PM.mov

However, one issue I have no idea how to solve is the flickering; I thought it was because I was using ReactiveHTML, but apparently not.

I also tried using param.update but it didn't stream.

ahuang11 · 2023-08-02T23:51:51Z

It seems like it also happens in the old ChatBox implementation too #5317

Screen.Recording.2023-08-02.at.7.48.24.PM.mov

However, if I downgrade to panel 1.2.0, the flickering disappears.

Screen.Recording.2023-08-02.at.7.51.07.PM.mov

ahuang11 · 2023-08-03T01:47:59Z

Added icons + repeat, undo, clear buttons.

Screen.Recording.2023-08-02.at.9.46.18.PM.mov

import re
from typing import Any

from panel.io.mime_render import exec_with_return
import pandas as pd
import panel as pn
import openai

DATAFRAME_PROMPT = """
    Here are the columns in your DataFrame: {columns}.
    Create  a plot with hvplot that highlights an interesting
    relationship between the columns with hvplot groupby kwarg.
"""

CODE_REGEX = re.compile(r"```python(.*?)```", re.DOTALL)


async def respond_with_openai(contents: Any):
    # extract the DataFrame
    if isinstance(contents, pd.DataFrame):
        global df
        df = contents
        columns = contents.columns
        message = DATAFRAME_PROMPT.format(columns=columns)
    else:
        message = contents

    # ask OpenAI to plot
    response = await openai.ChatCompletion.acreate(
        model="gpt-3.5-turbo",
        messages=[{"role": "user", "content": message}],
        temperature=0,
        max_tokens=500,
        stream=True,
    )
    message = ""
    async for chunk in response:
        message += chunk["choices"][0]["delta"].get("content", "")
        yield {"user": "OpenAI", "value": message}


async def respond_with_executor(code: str):
    return {
        "user": "Executor",
        "value": exec_with_return(code=code, global_context=globals()),
    }


async def response_callback(
    contents: Any,
    name: str,
    chat_interface: pn.widgets.ChatInterface,
):
    print(contents, "PASSED")
    if name == "You":
        async for chunk in respond_with_openai(contents):
            yield chunk
    elif CODE_REGEX.search(contents):
        yield await respond_with_executor(CODE_REGEX.search(contents).group(1))


chat_card = pn.widgets.ChatCard(callback=response_callback)
chat_interface = pn.widgets.ChatInterface(
    value=chat_card, widgets=[pn.widgets.TextInput(), pn.widgets.FileInput()]
)
chat_interface.servable()

ahuang11 · 2023-08-04T01:42:15Z

Some sample langchain code with chat interface

from langchain.chains import LLMChain
from langchain.memory import ConversationBufferMemory
from langchain.chat_models import ChatOpenAI
from langchain.schema import SystemMessage
from langchain.prompts import (
    ChatPromptTemplate,
    HumanMessagePromptTemplate,
    MessagesPlaceholder,
)
import panel as pn

pn.extension()


async def langchain_callback(contents, user, chat_interface):
    yield await chat_llm_chain.apredict(human_input=contents)

prompt = ChatPromptTemplate.from_messages([
    SystemMessage(content="You are a chatbot having a conversation with a human."),
    MessagesPlaceholder(variable_name="chat_history"),
    HumanMessagePromptTemplate.from_template("{human_input}"),
])
memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True)
llm = ChatOpenAI()
chat_llm_chain = LLMChain(
    llm=llm,
    prompt=prompt,
    verbose=True,
    memory=memory,
)
chat_feed = pn.widgets.ChatFeed(callback=langchain_callback)
chat_interface = pn.widgets.ChatInterface(value=chat_feed)
chat_interface.servable()

Screen.Recording.2023-08-03.at.9.38.59.PM.mov

Need to figure out how streaming + agent works with this.

ahuang11 · 2023-08-04T13:53:50Z

The latest diagram

ahuang11 · 2023-08-04T18:27:14Z

Starting to think that ChatInterface should simply inherit ChatFeed rather than compose it as a value. Feels a tad tedious typing this all out every time I need a callback:

chat_feed = pn.widgets.ChatFeed(callback=callback)
chat_interface = pn.widgets.ChatInterface(value=chat_feed)

The other solution is ChatInterface re-implements the same methods as ChatFeed.

ahuang11 · 2023-08-08T21:26:09Z

LangChain is now supported too!

from langchain.agents import initialize_agent, AgentType, load_tools
from langchain.llms import OpenAI


def callback(contents, name, chat_interface):
    agent.run(contents, callbacks=[PanelCallbackHandler(chat_interface=chat_interface)])
    yield system_entry.clone(value="That was fun, ask me more!")


system_entry = pn.widgets.ChatEntry(user="System", avatar="⚙️")
chat_interface = pn.widgets.ChatInterface(
    value=[system_entry.clone(value="Let's do math!")],
    callback=callback,
)
llm = OpenAI(streaming=True)
tools = load_tools(["pal-math"], llm=llm)
agent = initialize_agent(tools, llm)
pn.template.FastListTemplate(
    main=[chat_interface],
    title="MathGPT"
).servable()

Screen.Recording.2023-08-08.at.5.19.59.PM.mov

**with the following Callback (PR to LangChain repo)?

class PanelCallbackHandler(BaseCallbackHandler):
    def __init__(
        self,
        chat_interface: pn.widgets.ChatInterface,
        user: str = "LangChain",
        avatar: str = "🦜️",
    ):
        self.chat_interface = chat_interface
        self._entry = None
        self._active_user = user
        self._active_avatar = avatar
        self._disabled_state = self.chat_interface.disabled

        self._input_user = user
        self._input_avatar = avatar

    def on_llm_start(self, serialized: Dict[str, Any], *args, **kwargs):
        model = kwargs.get("invocation_params", {}).get("model_name", "")
        if self._active_user and model not in self._active_user:
            self._active_user = f"{self._active_user} ({model})"
        return super().on_llm_start(serialized, *args, **kwargs)

    def on_llm_new_token(self, token: str, **kwargs) -> None:
        self._entry = self.chat_interface.stream(
            token.replace("\n", "<br>"),
            user=self._active_user,
            avatar=self._active_avatar,
            entry=self._entry,
        )
        return super().on_llm_new_token(token, **kwargs)

    def on_llm_end(self, response: LLMResult, *args, **kwargs):
        return super().on_llm_end(response, *args, **kwargs)

    def on_llm_error(self, error: Union[Exception, KeyboardInterrupt], *args, **kwargs):
        return super().on_llm_error(error, *args, **kwargs)

    def on_agent_action(self, action: AgentAction, *args, **kwargs: Any) -> Any:
        return super().on_agent_action(action, *args, **kwargs)

    def on_agent_finish(self, finish: AgentFinish, *args, **kwargs: Any) -> Any:
        return super().on_agent_finish(finish, *args, **kwargs)

    def on_tool_start(
        self, serialized: Dict[str, Any], input_str: str, *args, **kwargs
    ):
        self._active_avatar = "🛠️"
        self._active_user = f"{self._active_user} - {serialized['name']}"
        return super().on_tool_start(serialized, input_str, *args, **kwargs)

    def on_tool_end(self, output, *args, **kwargs):
        self._active_user = self._input_user
        self._active_avatar = self._input_avatar
        return super().on_tool_end(output, *args, **kwargs)

    def on_tool_error(
        self, error: Union[Exception, KeyboardInterrupt], *args, **kwargs
    ):
        return super().on_tool_error(error, *args, **kwargs)

    def on_chain_start(
        self, serialized: Dict[str, Any], inputs: Dict[str, Any], *args, **kwargs
    ):
        self.chat_interface.disabled = True
        return super().on_chain_start(serialized, inputs, *args, **kwargs)

    def on_chain_end(self, outputs: Dict[str, Any], *args, **kwargs):
        self._entry = None
        self.chat_interface.disabled = self._disabled_state
        return super().on_chain_end(outputs, *args, **kwargs)

ahuang11 · 2023-08-09T13:54:17Z

Here's the latest ways to use with OpenAI

no async no stream

import openai
import panel as pn
pn.extension(sizing_mode="stretch_width")


def callback(contents: str, user: str, instance: pn.widgets.ChatInterface):
    response = openai.ChatCompletion.create(
        model="gpt-3.5-turbo",
        messages=[{"role": "user", "content": contents}],
    )
    yield response.choices[0]["value"]["content"]

ci = pn.widgets.ChatInterface(callback=callback)
ci.servable()

no async explicit stream (no response)

import openai
import panel as pn
pn.extension(sizing_mode="stretch_width")


def callback(contents: str, user: str, instance: pn.widgets.ChatInterface):
    response = openai.ChatCompletion.create(
        model="gpt-3.5-turbo",
        messages=[{"role": "user", "content": contents}],
        stream=True
    )
    entry = None
    for chunk in response:
        value = chunk["choices"][0]["delta"].get("content", "")
        entry = instance.stream(value=value, user="GPT3.5", avatar="🤖", entry=entry)

ci = pn.widgets.ChatInterface(callback=callback)
ci.servable()

(half) async generator stream

import openai
import panel as pn
pn.extension(sizing_mode="stretch_width")


async def callback(contents: str, user: str, instance: pn.widgets.ChatInterface):
    response = openai.ChatCompletion.create(
        model="gpt-3.5-turbo",
        messages=[{"role": "user", "content": contents}],
        stream=True
    )
    value = ""
    for chunk in response:
        value += chunk["choices"][0]["delta"].get("content", "")
        yield {"value": value, "user": "GPT3.5", "avatar": "🤖"}

ci = pn.widgets.ChatInterface(callback=callback)
ci.servable()

async no stream

import openai
import panel as pn
pn.extension(sizing_mode="stretch_width")


async def callback(contents: str, user: str, instance: pn.widgets.ChatInterface):
    response = await openai.ChatCompletion.acreate(
        model="gpt-3.5-turbo",
        messages=[{"role": "user", "content": contents}],
    )
    yield response.choices[0]["value"]["content"]

ci = pn.widgets.ChatInterface(callback=callback)
ci.servable()

async generator stream

async def callback(contents: str, user: str, instance: pn.widgets.ChatInterface):
    response = await openai.ChatCompletion.acreate(
        model="gpt-3.5-turbo",
        messages=[{"role": "user", "content": contents}],
        stream=True
    )
    value = ""
    async for chunk in response:
        value += chunk["choices"][0]["delta"].get("content", "")
        yield {"value": value, "user": "GPT3.5", "avatar": "🤖"}

ci = pn.widgets.ChatInterface(callback=callback)
ci.servable()

MarcSkovMadsen · 2023-08-09T19:29:43Z

How do I panel serve these examples such that the css is used @ahuang11 ?

It looks like below for me

ahuang11 · 2023-08-09T19:30:26Z

I think you may need to run panel build panel inside the panel repo dir once.

Once it's released, this will not be necessary.

MarcSkovMadsen · 2023-08-09T19:32:46Z

I have run panel build panel and it does not work. Why/ How does it work for you @ahuang11 ?

ahuang11 · 2023-08-09T19:34:42Z

Do you see any console errors?

ahuang11 · 2023-08-09T19:36:04Z

Maybe try running in incognito mode or hard refresh, rerun your panel serve, clear your cache

MarcSkovMadsen · 2023-08-09T19:43:54Z

Yes. Its trying to find the unreleased css via the CDN

I've tried setting the environment variable BOKEH_RESOURCES=inline. But it did not change anything. Is there some other configuration or environment variable I need to set?

ahuang11 · 2023-08-09T19:49:42Z

Have you tried incognito / clearing your cache?

Co-authored-by: Philipp Rudiger <prudiger@anaconda.com>

MarcSkovMadsen · 2023-10-10T02:36:50Z

@@ -0,0 +1,1646 @@
+"""The chat module provides components for building and using chat interfaces


When I run the automated tests to generate videos of chat_memory.py I can see that the ChatFeed does not scroll to the end. I.e. the user ends up not being able to see what is written. (1600x900px, zoom=1.5)

6facf4c1fa752e60f478ebf366720e4d.webm

""" Demonstrates how to use the ChatInterface widget to create a chatbot using OpenAI's GPT-3 API with LangChain. """ import panel as pn from langchain.chains import ConversationChain from langchain.chat_models import ChatOpenAI from langchain.memory import ConversationBufferMemory pn.extension(design="material") async def callback(contents: str, user: str, instance: pn.widgets.ChatInterface): await chain.apredict(input=contents) chat_interface = pn.widgets.ChatInterface(callback=callback, callback_user="ChatGPT") chat_interface.send( "Send a message to get a reply from ChatGPT!", user="System", respond=False ) callback_handler = pn.widgets.langchain.PanelCallbackHandler( chat_interface=chat_interface ) llm = ChatOpenAI(streaming=True, callbacks=[callback_handler]) memory = ConversationBufferMemory() chain = ConversationChain(llm=llm, memory=memory) chat_interface.servable()

Maybe try tweaking auto_scroll_limit?

philippjfr

Really excellent work @ahuang11! This PR has grown huge and we've iterated a bunch. Overall I think it's in a good state and we can resolve remaining issues in subsequent PRs.

github-actions · 2025-09-10T15:24:05Z

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

ahuang11 changed the title ~~Rough outline of Chat* components~~ Implement Chat* composable components Jul 27, 2023

ahuang11 requested a review from MarcSkovMadsen July 27, 2023 14:07

ahuang11 force-pushed the chat_components branch from c6942e2 to f8a2847 Compare August 2, 2023 02:32

ahuang11 force-pushed the chat_components branch from 8a8439a to 9672f82 Compare August 4, 2023 00:54

ahuang11 force-pushed the chat_components branch from e0daca0 to 6f3e1e9 Compare August 7, 2023 22:18

MarcSkovMadsen reviewed Aug 9, 2023

View reviewed changes

Comment thread panel/widgets/chat.py

MarcSkovMadsen reviewed Aug 9, 2023

View reviewed changes

Comment thread panel/widgets/chat.py Outdated

MarcSkovMadsen reviewed Aug 9, 2023

View reviewed changes

Comment thread panel/widgets/chat.py Outdated

MarcSkovMadsen reviewed Aug 9, 2023

View reviewed changes

Comment thread panel/widgets/chat.py Outdated

ahuang11 and others added 19 commits October 9, 2023 17:25

Fix tests

a00db2a

Reorganize parameter definitions

73dbe72

Revert overriding

d4249e0

Add back stylesheets and add llama logo

3788feb

Embed PDF

5d6a93c

Remove ignored_refs

4ad7ebf

Apply suggestions from code review

e89bfd5

Co-authored-by: Philipp Rudiger <prudiger@anaconda.com>

Missing }

e7a2364

Fix tests

8715662

Fix CSS (#5574)

cbe195a

Optimize string tokens

59b3a1c

Add copy to clipboard

5c7c1bd

Prevent horizontal scrollbar and add hover over reaction

bf44da9

Fix broken tests from copy icon

1e773a9

Streamline and reorganize

7748001

Refactor ChatEntry update and stream

39067b2

Fix user when passed in as dict

a8ef5b5

Add test

25a6a79

Add missing doc, do not have copy for placeholder, and add test for none

f9a2198

ahuang11 force-pushed the chat_components branch from e935f8b to f9a2198 Compare October 10, 2023 00:26

Add wait to prevent tests from failing

b088ed4

MarcSkovMadsen reviewed Oct 10, 2023

View reviewed changes

Comment thread panel/widgets/chat.py

ahuang11 mentioned this pull request Oct 10, 2023

ChatEntry copy icon should show animation upon copy #5610

Closed

philippjfr approved these changes Oct 11, 2023

View reviewed changes

philippjfr merged commit af6262b into main Oct 11, 2023

philippjfr deleted the chat_components branch October 11, 2023 18:20

ahuang11 mentioned this pull request Oct 13, 2023

Make ChatBox usage easy #5277

Closed

github-actions Bot locked as resolved and limited conversation to collaborators Sep 10, 2025

		@@ -0,0 +1,1646 @@
		"""The chat module provides components for building and using chat interfaces

Uh oh!

Uh oh!

Conversation

ahuang11 commented Jul 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Jul 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ahuang11 commented Aug 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ahuang11 commented Aug 2, 2023

Uh oh!

ahuang11 commented Aug 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ahuang11 commented Aug 4, 2023

Uh oh!

ahuang11 commented Aug 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ahuang11 commented Aug 4, 2023

Uh oh!

ahuang11 commented Aug 8, 2023

Uh oh!

ahuang11 commented Aug 9, 2023

Uh oh!

Uh oh!

MarcSkovMadsen commented Aug 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ahuang11 commented Aug 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MarcSkovMadsen commented Aug 9, 2023

Uh oh!

ahuang11 commented Aug 9, 2023

Uh oh!

ahuang11 commented Aug 9, 2023

Uh oh!

MarcSkovMadsen commented Aug 9, 2023

Uh oh!

Uh oh!

ahuang11 commented Aug 9, 2023

Uh oh!

Uh oh!

Uh oh!

MarcSkovMadsen Oct 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ahuang11 Oct 10, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

philippjfr left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ahuang11 commented Jul 27, 2023 •

edited

Loading

codecov Bot commented Jul 27, 2023 •

edited

Loading

ahuang11 commented Aug 2, 2023 •

edited

Loading

ahuang11 commented Aug 3, 2023 •

edited

Loading

ahuang11 commented Aug 4, 2023 •

edited

Loading

MarcSkovMadsen commented Aug 9, 2023 •

edited

Loading

ahuang11 commented Aug 9, 2023 •

edited

Loading

MarcSkovMadsen Oct 10, 2023 •

edited

Loading