Simpler Chatbot Component / API

[abidlabs]

Building a Chatbot is a very common use case that is currently fairly complex to do with Gradio. Here's all the code that's needed for the simple Chatbot:

import gradio as gr
import random
import time

with gr.Blocks() as demo:
    chatbot = gr.Chatbot()
    msg = gr.Textbox()
    clear = gr.Button("Clear")

    def user(user_message, history):
        return "", history + [[user_message, None]]

    def bot(history):
        bot_message = random.choice(["Yes", "No"])
        history[-1][1] = bot_message
        time.sleep(1)
        return history

    msg.submit(user, [msg, chatbot], [msg, chatbot], queue=False).then(
        bot, chatbot, chatbot
    )
    clear.click(lambda: None, None, chatbot, queue=False)

demo.launch()

We should consider having a custom component / higher-level abstraction for the Chatbot.

[pGit1]

I agree. :( @abidlabs. Your posts are useful but this is really hard unless you spend alot of time figuring it out.

For instance how do I get a "regenerate" tab to work (to get a new chat generation for the same input in case it was poor)? Secondly how do I get this chatbot to produce outputs as it is typing like chatgpt or even this demo does: https://chat.lmsys.org/? See my hacked together code:

def chatbot2(share=False, encrypt=True):
    with gr.Blocks() as demo:
        chatbot = gr.Chatbot()
        msg = gr.Textbox()
        clear = gr.Button("Clear")
        regenerate = gr.Button('Re-generate')

        def user(user_message, history):
            return "", history + [[user_message, None]]
        
        def bot(history):
            curr_input, past_convo = history[-1][0], history[:-1] 
            bot_message = conversation_bot(curr_input, past_convo)
            history[-1][1] = bot_message
            return history

        msg.submit(user, [msg, chatbot], [msg, chatbot], queue=False).then(
            bot, chatbot, chatbot
        )
        clear.click(lambda: None, None, chatbot,  queue=False)
        
    demo.launch(share=share, encrypt=encrypt)

chatbot2()

Honestly the code above doenst make a ton of sense to me but it works. Its really hard and I find myself unable to understand what is going on in Gradio code alot of times as it takes away from ml development. I'd love if I could figure out how to get it do the two things I asked about though. :(

If any thing it would be nice to just drop models into a predefined template chatbot or some thing with several toggleable features.

[abidlabs]

Thanks @pGit1 for your questions and feedback. We'll think about this as we build towards 4.0.

In the meantime, you can get streaming responses by using yield instead of return, as described here; https://gradio.app/creating-a-chatbot/#add-streaming-to-your-chatbot

For regenerate, you would simply rerun the bot() method when the regenerate button is pressed.

If you are not already familiar with Blocks, I would go through the "Building with Blocks" section of the Gradio Guides: https://gradio.app/blocks-and-event-listeners/

[pGit1]

@abidlabs the streaming output makes sense I think, and I will update my code accordingly.

Thanks for the event listeners section. I briefly looked over it but dont seem to find the answer for regenerate button related to the answer you provided.

But Concerning the "regenerate" button would my code look something like:

def bot_return():
    return bot()

regenerate.click(bot_return, inputs=None, outputs=None)
# or 
regenerate.click(lambda:None, None, bot_return,  queue=False)

?

Thanks for your help! :)

[pGit1]

I cant get this regenerate button to work. 😢 But the "streaming" works now.

I have LITERALLY spent two hours on this and cant get desired dinctionality:

        def bot(history):
            curr_input, past_convo = history[-1][0], history[:-1]  # current input has a None, so we exlcude it
            bot_message = convo_bot(curr_input, past_convo)
            history[-1][1] = ""
            for word in bot_message:
                history[-1][1] += word
                time.sleep(0.02)
                yield history

        def regenerate_bot_response(history):
            new_history_generator = bot(history)
            history[-1][1] = ""
            for new_history in new_history_generator:
                history[-1][1] += new_history
                time.sleep(0.02)
                yield history

        regenerate.click(regenerate_bot_response, chatbot, chatbot, queue=False)

The only way I can get this to work is Without streaming the outputs.

[abidlabs]

Hi @pGit1 please take a look at this Space to get an idea of how to add a Regenerate button: https://huggingface.co/spaces/project-baize/Baize-7B

[pGit1]

@abidlabs This: https://huggingface.co/spaces/project-baize/Baize-7B/blob/main/app.py is literally the problem. This is unnecessarily complicated and EVERYONE does this stuff a significantly different way. Its quite annoying but I will try to parse all this code. Thx for the reference.

The fact that code I posted below does not have a clear retry button functionality is a bit annoying. For now I am getting regenerate functionality without the ability to yield tokens one by one.

[abidlabs]

I'd like to propose an abstraction, a ChatInterface class, which I think will solve the issues faced by users (namely having to reimplement a lot of boilerplate every time they want to create an Chatbot Interface).

Usage:

gr.ChatInterface(fn).launch()

Here, fn must be a function that takes in two parameters: a history (which is a list of user/bot message pairs) and a message string representing the current input. And it produces a fully-fleshed out chat interface with the functionality that is most popularly implemented in chatbots:

For example:

def echo(history, message):
  return message

gr.ChatInterface(echo).launch()

produces the demo above.

The ChatInterface also implements best practices (disables queuing for actions like clearing the input textbox. It can also disable/enable buttons as appropriate). It also implements a very simple API at the /chat endpoint which simply takes in an input message and returns the output message -- keeping state internally so API users don't have to worry about that. It also disables the unnecessary API endpoints, so you have a very clean API page:

Many of our existing chat demos would be much, much simpler with this abstraction. It also plays nicely with langchain's LLM abstractions. For example, here's a complete langchain example:

from langchain import OpenAI,ConversationChain,LLMChain, PromptTemplate
from langchain.memory import ConversationBufferWindowMemory

prompt = PromptTemplate(
    input_variables = ["history","human_input"],
    template = ...
)

chatgpt_chain = LLMChain(
    llm = OpenAI(temperature=0),
    prompt = prompt,
    verbose = True,
    memory = ConversationBufferWindowMemory(k=2),
)


def chat(history, input):
    chatgpt_chain.memory.chat_memory.messages = history   # there might be a better way to do this
    return chatgpt_chain.predict(input)

gr.ChatInterface(chat).launch()

The ChatInterface is intentionally very opinionated at the moment. I think we should release first and add options that are most requested by users. For example, we might add support for changing the button text, the buttons themselves, additional input parameters for the chat function, etc.

Here is the current WIP implementation of ChatInterface:

from gradio import Blocks
import gradio as gr

class ChatInterface(Blocks):
    def __init__(self, fn):
        
        super().__init__(mode="chat_interface")        
        self.fn = fn
        self.history = []                    
        
        with self:
            self.chatbot = gr.Chatbot(label="Input")
            self.textbox = gr.Textbox()
            self.stored_history = gr.State()
            self.stored_input = gr.State()
            
            with gr.Row():
                submit_btn = gr.Button("Submit", variant="primary")
                delete_btn = gr.Button("Delete Previous")
                retry_btn = gr.Button("Retry")
                clear_btn = gr.Button("Clear")
                
            # Invisible elements only used to set up the API
            api_btn = gr.Button(visible=False)
            api_output_textbox = gr.Textbox(visible=False, label="output")
            
            self.buttons = [submit_btn, retry_btn, clear_btn]

            self.textbox.submit(
                self.clear_and_save_textbox, [self.textbox], [self.textbox, self.stored_input], api_name=False, queue=False,
            ).then(
                self.submit_fn, [self.chatbot, self.stored_input], [self.chatbot], api_name=False
            )
            
            submit_btn.click(self.submit_fn, [self.chatbot, self.textbox], [self.chatbot, self.textbox], api_name=False)
            delete_btn.click(self.delete_prev_fn, [self.chatbot], [self.chatbot, self.stored_input], queue=False, api_name=False)
            retry_btn.click(self.delete_prev_fn, [self.chatbot], [self.chatbot, self.stored_input], queue=False, api_name=False).success(self.retry_fn, [self.chatbot, self.stored_input], [self.chatbot], api_name=False)
            api_btn.click(self.submit_fn, [self.stored_history, self.textbox], [self.stored_history, api_output_textbox], api_name="chat")
            clear_btn.click(lambda :[], None, self.chatbot, api_name="clear")          
    
    def clear_and_save_textbox(self, inp):
        return "", inp
        
    def disable_button(self):
        # Need to implement in the event handlers above
        return gr.Button.update(interactive=False)
        
    def enable_button(self):
        # Need to implement in the event handlers above
        return gr.Button.update(interactive=True)
                
    def submit_fn(self, history, inp):
        # Need to handle streaming case
        out = self.fn(history, inp)
        history.append((inp, out))
        return history
    
    def delete_prev_fn(self, history):
        try:
            inp, _ = history.pop()
        except IndexError:
            inp = None
        return history, inp

    def retry_fn(self, history, inp):
        if inp is not None:
            out = self.fn(history, inp)
            history.append((inp, out))
        return history
    
ChatInterface(lambda x,y:y).launch()

Would appreciate any thoughts here!

[aliabid94]

I would make the textbox have no container and no label, and then make the textbox and submit in the same row - submit should have scale=0. Or maybe all the buttons grouped in a Column and in the same row as the textbox - grouping in a column will make them all move to the next row if any of them wrap

[dawoodkhan82]

@abidlabs should this be done after the gr.Chatbot() refactor? I assume we would need to change the underlying data structure for how message history is passed. Especially if we want to be able to support other gradio components, multiple users, avatar images, etc.

[pngwn]

I think a UI something like this would be cool:"

I also think it should be possible to disable all buttons other than submit.

I also think we could actually get rid of the "submit" text here and use an Icon. We have discussed button icons before, maybe this would be a good opportunity to do it.

[abidlabs]

Thanks @aliabid94 @dawoodkhan82 @pngwn

Will try to make the UI more like what @pngwn showed

I also think we could actually get rid of the "submit" text here and use an Icon. We have discussed button icons before, maybe this would be a good opportunity to do it.

We can use emojis before the text, which is actually what a lot of popular gradio chatbots do right now.

@abidlabs should this be done after the gr.Chatbot() refactor? I assume we would need to change the underlying data structure for how message history is passed. Especially if we want to be able to support other gradio components, multiple users, avatar images, etc.

I don't think we need to wait because one value of having a high-level abstraction like this is that we could refactor underlying implementation details and everyone's demo would still work. But let's touch upon this point in retro in case I'm misunderstanding

[dawoodkhan82]

@abidlabs I was just thinking if we have to add more params for gr.Chatbot() (in addition to the history). We can always make the chat interface opinionated on these params in the underlying implementation. But just a thought since we haven't decided on what the chatbot refactor would look like.

[freddyaboulton]

Thanks for the proposal @abidlabs ! I agree this will make chat demos way easier to grok for new users. I like that it's implemented as a Blocks, that means it will be really easy to nest within other gr.Blocks. This is something we talked about in regards to custom components (we used the term composed components) so it's cool that this is a step in that direction.

Some comments I have:

• I like @pngwn 's design. I think the squareness looks nice. Adding emojis for the buttons would also look be great.
• Totally agree with your comment in the code that we should support streaming. I think the submit button should turn to stop like it does for gr.Interface.
• We should add examples to the init. That's something I see in the viral chatbot demos.
• We should add a theme to the init to allow custom theming.
• Hitting the submit button should also clear the textbox right?
• I think its slick to have the /chat route only return the most recent message but do we really need gr.Chatbot and self.stored_history to both track the history? I think adding a separate/history route to fetch the current history will be valuable to users and straightforward to add to this current design.
• Storing history in the class, e.g.self.history will probably cause troubles with concurrency. Seems to be unused so maybe it's a typo.

[abidlabs]

Thanks @freddyaboulton and all! Will open up a PR towards the end of this week for us to try out

[freddyaboulton]

The one problem I can see with this approach is that it will fall apart when a user has to deviate from the supported implementation.

For example, if a user wants to build a multi-modal chatbot, they will have to start from scratch to add an UploadButton. The quick fix would be to expand the init method of ChatInterface to allow an optional UploadButton but I wonder if there's a way to make it really easy to add incremental changes.

This is a general point about Blocks. I don't have a solution so this comment is mainly just food-for-thought.

[abidlabs]

The one problem I can see with this approach is that it will fall apart when a user has to deviate from the supported implementation.

Yes, that's the tradeoff, but I think it's the same tradeoff with Interface, and Interface has been quite successful imo in getting people started with Gradio

[abidlabs]

For example, if a user wants to build a multi-modal chatbot, they will have to start from scratch to add an UploadButton. The quick fix would be to expand the init method of ChatInterface to allow an optional UploadButton but I wonder if there's a way to make it really easy to add incremental changes.

For this use case specifically, I think the best thing to do would be to have a rich textbox component (#4668) and allow that to be used in place of the plain textbox via a parameter. I'll add a note in the upcoming PR

[pGit1]

Can we just implement a simple user facing interface to get this functionality? https://chat.lmsys.org/?arena Something like this would be awesome with hopefully an easy way to add more hyperparameters as needed.

…

On Wed, Jul 5, 2023 at 5:39 PM Abubakar Abid ***@***.***> wrote: For example, if a user wants to build a multi-modal chatbot, they will have to start from scratch to add an UploadButton. The quick fix would be to expand the init method of ChatInterface to allow an optional UploadButton but I wonder if there's a way to make it really easy to add incremental changes. For this use case specifically, I think the best thing to do would be to have a rich textbox component (#4668 <#4668>) and allow that to be used in place of the plain textbox — Reply to this email directly, view it on GitHub <#3510 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADKT4SULAYVICOQF2M262F3XOXNIPANCNFSM6AAAAAAV75P5VQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>