Streaming responses

[scottleibrand]

The https://chat.openai.com/chat web interface begins displaying responses as soon as the model generates them, allowing you to begin reading lengthy responses much sooner.

Do you know how we would go about implementing a similar feature on the command line? It probably doesn't need to stream every word, but line by line streaming would be cool.

Happy to help, whether that means picking up where you left off or figuring it out from scratch...

[acheong08]

Happy to help, whether that means picking up where you left off or figuring it out from scratch...

A pull request would be much appreciated.

Perhaps there could be another parameter to get_chat_response to specify the return type: text or stream, with text as default

[scottleibrand]

Ok, I'll try to take a look tomorrow. LMK if you find anything out about how it's done in the browser: I haven't done any research yet.

[A-kirami]

I implemented streaming response, but I don't know how to output it on the console.

[A-kirami]

I implemented streaming response, but I don't know how to output it on the console.

I solved the problem, but the method was ugly.

[acheong08]

@A-kirami I have written a rough implementation of text streaming. Can you check if it works as intended (It works for me but need verification)

get_chat_response(prompt, output="stream") should yield each message.

if __name__ == "__main__":
    with open("config.json", "r") as f:
        config = json.load(f)
    chatbot = Chatbot(config)
    if 'session_token' in config:
        chatbot.refresh_session()
    while True:
        prompt = input("You: ")
        for message in chatbot.get_chat_response(prompt, output="stream"):
            print(message)

This should print out the message as it streams. I'm not sure how to deal with the console so that it is overwritten as more data comes in.

[A-kirami]

@A-kirami I have written a rough implementation of text streaming. Can you check if it works as intended (It works for me but need verification)

get_chat_response(prompt, output="stream") should yield each message.

if __name__ == "__main__":
    with open("config.json", "r") as f:
        config = json.load(f)
    chatbot = Chatbot(config)
    if 'session_token' in config:
        chatbot.refresh_session()
    while True:
        prompt = input("You: ")
        for message in chatbot.get_chat_response(prompt, output="stream"):
            print(message)

This should print out the message as it streams. I'm not sure how to deal with the console so that it is overwritten as more data comes in.

It works normally. If you need to print it on the console, you can refer to the implementation here.

https://github.com/A-kirami/ChatGPT/blob/main/src/revChatGPT/__main__.py#L69-L81

[A-kirami]

@acheong08

Like this:

[acheong08]

I am terrible at this. It keeps erasing the lines in my implementation

[acheong08]

I'll copy your code over and modify it

[acheong08]

This doesn't work in all terminals:

Detected that the lines of output has been exceeded the height of terminal windows, which                     caused the former output remained and keep adding new lines.
检测到输出过程中, 输出行数曾大于命令行窗口行数, 这会导致输出清除不完整, 而使输出不停增长。请注意控制输出行数

[acheong08]

sys.stdout.write("\033[F\033[K")

This doesn't work properly in all terminals either for some reason. My hair is falling off lol

[acheong08]

The error seems to be that this deletes the number of lines it is aware of but in some terminals, line count is counted differently

[scottleibrand]

The approach I was attempting (but haven't perfected yet) is to only write a line once it is has completely loaded, rather than deleting lines after they're written. Would that work here?

[acheong08]

That is what I'm attempting to do. I write the whole message and then when a new one comes in, I try to delete the previous message. It's not clearing the right number of lines though.

[scottleibrand]

I'm suggesting you not try to delete the previous message (as I did with the please wait prompt), but rather keep track of how many lines you've written, and only write each new line.

[acheong08]

keep track of how many lines you've written

Tried that. An issue with this is that some environments treat lines differently. For example, my current omz shell treats each new line in the terminal as a new line (which changes depending on window size). However, Python can only see \n as new line

[acheong08]

I tried counting the characters and saving them, then deleting that number of characters when a new message comes in. However, \b isn't supported in some terminals either.

[acheong08]

Not even ChatGPT knows the solution to this...

[acheong08]

I think streaming is more suited for GUI applications. Terminals don't like overwrites

[scottleibrand]

Do you have a pointer to a commit that has streaming working but doesn't have the terminal stuff solved yet? I can restart my efforts from there.

[acheong08]

Streaming working on latest commit

[scottleibrand]

Of main or async-dev?

[acheong08]

main

[scottleibrand]

On that commit I'm getting

  File "/Users/sleibrand/src/ChatGPT/src/revChatGPT/__main__.py", line 70, in <module>
    print("Chatbot:", response['message'])
TypeError: 'generator' object is not subscriptable

[acheong08]

It returns a string. I didn't update the docs yet. You need to call with output="stream"

[scottleibrand]

I've been running it with python3 -m revChatGPT - is there another command-line you've been using?

[acheong08]

I've been running it with python3 -m revChatGPT - is there another command-line you've been using?

It can only be implemented custom right now.

[acheong08]

Example code: #5 (comment)

[scottleibrand]

https://github.com/acheong08/ChatGPT/compare/main...scottleibrand:ChatGPT:async?expand=1

~/src/ChatGPT $ python3 -m revChatGPT

    ChatGPT - A command-line interface to OpenAI's ChatGPT (https://chat.openai.com/chat)
    Repo: github.com/acheong08/ChatGPT

Type '!help' to show commands
Press enter twice to submit your question.

You: test

...
I'm sorry, but I'm not sure what you're looking for. Could you please provide
more information or context? I'm a large language model trained by OpenAI, and
I'm here to help with any questions you may have.




You:

Cleaning up, then will PR.

[scottleibrand]

Opened #23

[acheong08]

I ought to have some better way to detect auth errors

[scottleibrand]

Yeah, that has tripped me up a few times too. Can we store more persistent credentials to allow it to log in again?

[acheong08]

The thing with the next auth tokens is that it needs to be refreshed every hour via !refresh. It stays valid that way. However, if you don't refresh it, it expires.

[acheong08]

I haven't figured out their hashing system on the login page. It hashes the password client side

[acheong08]

I'll open an issue for this