token streaming to stdout

[veonua]

Describe the bug

versions after 0.16 verbose streaming

hi

stream result: {
  "id": "chatcmpl-897VdZUcQblQ4J5aHRIPfxQLk7Q0K",
  "object": "chat.completion.chunk",
  "created": 1697184509,
  "model": "gpt-4-0613",
  "choices": [
    {
      "index": 0,
      "delta": {
        "role": "assistant",
        "content": ""
      },
      "finish_reason": null
    }
  ]
}
stream result: {
  "id": "chatcmpl-897VdZUcQblQ4J5aHRIPfxQLk7Q0K",
  "object": "chat.completion.chunk",
  "created": 1697184509,
  "model": "gpt-4-0613",
  "choices": [
    {
      "index": 0,
      "delta": {
        "content": "Hello"
      },
      "finish_reason": null
    }
  ]
}
stream result: {
  "id": "chatcmpl-897VdZUcQblQ4J5aHRIPfxQLk7Q0K",
  "object": "chat.completion.chunk",
  "created": 1697184509,
  "model": "gpt-4-0613",
  "choices": [
    {
      "index": 0,
      "delta": {
        "content": "!"
      },
      "finish_reason": null
    }
  ]
}
stream result: {
  "id": "chatcmpl-897VdZUcQblQ4J5aHRIPfxQLk7Q0K",
  "object": "chat.completion.chunk",
  "created": 1697184509,
  "model": "gpt-4-0613",
  "choices": [
    {
      "index": 0,
      "delta": {
        "content": " How"
      },
      "finish_reason": null
    }
  ]
}
stream result: {
  "id": "chatcmpl-897VdZUcQblQ4J5aHRIPfxQLk7Q0K",
  "object": "chat.completion.chunk",
  "created": 1697184509,
  "model": "gpt-4-0613",
  "choices": [
    {
      "index": 0,
      "delta": {
        "content": " can"
      },
      "finish_reason": null
    }
  ]
}
stream result: {
  "id": "chatcmpl-897VdZUcQblQ4J5aHRIPfxQLk7Q0K",
  "object": "chat.completion.chunk",
  "created": 1697184509,
  "model": "gpt-4-0613",
  "choices": [
    {
      "index": 0,
      "delta": {
        "content": " I"
      },
      "finish_reason": null
    }
  ]
}
stream result: {
  "id": "chatcmpl-897VdZUcQblQ4J5aHRIPfxQLk7Q0K",
  "object": "chat.completion.chunk",
  "created": 1697184509,
  "model": "gpt-4-0613",
  "choices": [
    {
      "index": 0,
      "delta": {
        "content": " assist"
      },
      "finish_reason": null
    }
  ]
}
stream result: {
  "id": "chatcmpl-897VdZUcQblQ4J5aHRIPfxQLk7Q0K",
  "object": "chat.completion.chunk",
  "created": 1697184509,
  "model": "gpt-4-0613",
  "choices": [
    {
      "index": 0,
      "delta": {
        "content": " you"
      },
      "finish_reason": null
    }
  ]
}
stream result: {
  "id": "chatcmpl-897VdZUcQblQ4J5aHRIPfxQLk7Q0K",
  "object": "chat.completion.chunk",
  "created": 1697184509,
  "model": "gpt-4-0613",
  "choices": [
    {
      "index": 0,
      "delta": {
        "content": " today"
      },
      "finish_reason": null
    }
  ]
}
stream result: {
  "id": "chatcmpl-897VdZUcQblQ4J5aHRIPfxQLk7Q0K",
  "object": "chat.completion.chunk",
  "created": 1697184509,
  "model": "gpt-4-0613",
  "choices": [
    {
      "index": 0,
      "delta": {
        "content": "?"
      },
      "finish_reason": null
    }
  ]
}
stream result: {
  "id": "chatcmpl-897VdZUcQblQ4J5aHRIPfxQLk7Q0K",
  "object": "chat.completion.chunk",
  "created": 1697184509,
  "model": "gpt-4-0613",
  "choices": [
    {
      "index": 0,
      "delta": {},
      "finish_reason": "stop"
    }
  ]
}

Hello! How can I assist you today?

Reproduce

install latest version

Expected behavior

no streaming jsons

Screenshots

No response

Open Interpreter version

=0.1.7

Python version

3.11.6

Operating System name and version

Windows 11

Additional context

No response

[owgit]

same here , using Mac M1 Max .
even if "%debug false"

[couturelp]

same here, wou ld love a fix. happening on ubuntu console. Turned off debug, not sure how to prevent this from happening.

[grexzen]

Same.

[brisklad]

same

[xmi1an]

Same...

[wxtt522]

same

[QIanGua]

same

[jakenuts]

It's LiteLLM printing that out. If you go to whereever that is installed on your system ("C:\Users\jim\miniconda3\Lib\site-packages\litellm\utils.py" on mine using miniconda) you can comment out the line ~427 print(f"stream result: {result}") to stop that. You will need to start a new session before it takes effect.

[AdelElo13]

It's LiteLLM printing that out. If you go to whereever that is installed on your system ("C:\Users\jim\miniconda3\Lib\site-packages\litellm\utils.py" on mine using miniconda) you can comment out the line ~427 print(f"stream result: {result}") to stop that. You will need to start a new session before it takes effect.

Does it answer to you with the question you asked so when you say hi the respons is hihello how can i help you?

[Panoplos]

It's LiteLLM printing that out. If you go to whereever that is installed on your system ("C:\Users\jim\miniconda3\Lib\site-packages\litellm\utils.py" on mine using miniconda) you can comment out the line ~427 print(f"stream result: {result}") to stop that. You will need to start a new session before it takes effect.

Does it answer to you with the question you asked so when you say hi the respons is hihello how can i help you?

Not in my case. The suggested hack works, but the LiteLLM peeps should indeed fix this. Amazed this was even committed.

[brisklad]

It worked. Thx a lot!

…

On Sat, Oct 14, 2023 at 1:22 AM James White ***@***.***> wrote: It's LiteLLM printing that out. If you go to whereever that is installed on your system ("C:\Users\jim\miniconda3\Lib\site-packages\litellm\utils.py" on mine using miniconda) you can comment out the line ~427 print(f"stream result: {result}") to stop that. You will need to start a new session before it takes effect. — Reply to this email directly, view it on GitHub <#637 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHDJSWUHJHMIE5T74267QKLX7IOQLAVCNFSM6AAAAAA56YUSB6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRSGYZDINRWGE> . You are receiving this because you commented.Message ID: ***@***.***>

[QIanGua]

It's LiteLLM printing that out. If you go to whereever that is installed on your system ("C:\Users\jim\miniconda3\Lib\site-packages\litellm\utils.py" on mine using miniconda) you can comment out the line ~427 print(f"stream result: {result}") to stop that. You will need to start a new session before it takes effect.

Does it answer to you with the question you asked so when you say hi the respons is hihello how can i help you?

Not in my case. The suggested hack works, but the LiteLLM peeps should indeed fix this. Amazed this was even committed.

Has anyone raised this issue there (litellm) ?

[krrishdholakia]

this is fixed - BerriAI/litellm#598

let me know if any of y'all see this on an updated version.

We're also working on adding testing to make sure errant print statements aren't outputted. Most of our print logic is behind print_verbose. Which is useful for debugging errors.

[couturelp]

As an FYI, here's how you force upgrade litellm using pip to get the error to stop

pip install --upgrade litellm

Cheers!

[ericrallen]

I'm going to close this one as I believe it was fixed in 0.1.10.

Please feel encouraged to reopen this issue if you experience the same behavior with the latest version.