Hanging forever if the context is too large

[Happ1ness-dev]

Hello, I'm having a blast experimenting with this thing, and in my opinion it has a lot of potential.

I'd like to report one annoying issue though, the program is forever hanging on the response, if the context grows too large. Are there any plans to make it summarize the context if it's getting too big?

[KillianLucas]

Hi @Happiness777, thanks for trying Open Interpreter and for the kind words!

It just did the same for me. Let's add --debug as mentioned in #4 so we can see what's going on— tokentrim (which was built for Open Interpreter) is probably the cause. It's supposed to handle this automatically by truncating old messages. but it's passing all its tests..

Adding LLM powered summarizing to tokentrim is a great idea though. Right now it removes old messages and removes the middle of the oldest message, if even that's too large. Could be given permission to summarize, something like:

import tokentrim as tt

response = openai.ChatCompletion.create(
  model=model,
  messages=tt.trim(messages, model, summarize=True) # Summarizes old messages to fit under model's max token count
)

Then we could default to this way of doing it in Open Interpreter as long as --local is set to False. Maybe could use Langchain's summarizer under the hood if they have a good solution that's complicated enough that we wouldn't want to just implement it ourselves in tokentrim (do they?)

Anyway will add --debug so we can see what's wrong and fix it 👍

[Happ1ness-dev]

It just did the same for me. Let's add --debug as mentioned in #4 so we can see what's going on— tokentrim (which was built for Open Interpreter) is probably the cause.

Yes, I've added a debug message whenever the trimming happens, and it hangs just on this moment. Not on message sending, not anything else, but on the token trimming. I've also added a debug message counting all tokens, and I didn't even get to 4k, I only had 3k (2917 to be exact) tokens history when this happened.

[Happ1ness-dev]

Btw, what happens if the file/output is too large for the context? It tries to summarize it or just truncates it to fit the context limit?

[KillianLucas]

Yeesh. Will add more accurate tests in tokentrim to figure out what's happening. Might have to do with function calls not being handled correctly. Thanks for tracking it down.

If the output of any code block is too large, code_interpreter.py's truncate_output function will truncate it to the last 2000 characters and put a message at the top saying it was truncated. This is exactly what you + the language model see.

Thinking of adding an option for users to change that number of characters, and maybe even for the LLM to change it (I could imagine some "cell magic" at the top of any code block like %truncate_output_chars=5000 if it wanted to change it on a per-cell basis. We could include "documentation" for this in the truncate message for the LLM, so it would know how to use it if it needed to see more than 2000 chars at a time).

Summarizing cell outputs is a fantastic idea. I could imagine more cell magic: %include_output_summary=True which summarized the entire cell output and added it above the last 2000 chars of truncated output... I wonder what use-cases that might enable? Let me know if you have thoughts.

[Happ1ness-dev]

Not just summarisation though. The summarisation might be with a certain message/goal which the LLM could set for the summarizer. For example, find the error, or something like that.
It could enable the interpreter to be used for more larger tasks, like looking at the codebase and analyzing it. Maybe even editing it with some help (or no help at all).

I imagine we could even go beyond context limit, because the LLM could write down the desired output in the file, enabling it to remember it and have more lengthy final output. It could work with data in chunks, step-by-step, while not forgetting about the bigger picture. This could be interesting for story writing too, I think.

Once, a few days ago, I had gpt-4 doing smart decisions with the file, like looking for a specific string and grabbing a chunk of file with this string to check what's happening there. This was really interesting to watch.

P.S.
But maybe I'm just being unrealistic. Who knows? ¯⁠\⁠_⁠(⁠ツ⁠)⁠_⁠/⁠¯

[JBR1325]

First and foremost amazing job @KillianLucas and anyone else involved with this project. For what it is worth I have zero coding knowledge whatsoever. But with AI I have learned over the last few months how to vaguely navigate things.

I have been using this a lot in the last day, and really had some fun and good success with it! I do have it hang up on me still at that context max length and I'm thinking its still this same topic you are discussing here. I was able to make a bigger project by prompting it to do something similar to what you are suggesting here. I asked it for the specific steps needed to make a project and had it break the overall project into smaller project files that would need to be written. I had it save all of that info into a text file in a project folder. I then started fresh and pointed it to that folder and that file and told it to A. check if the file names needed in the instruction file where present in the folder, if a name was not present it was to start coding it. Every time it would hang up I would close it and rerun it with the same or similar prompt and it would start making additional missing files. At the end I restarted and prompted it to go through all the files in the folder and suggest any changes (I then took those changes and put it in a file and reran this process changing the prompt slightly) and with a little tinkering I was able to get some really impressive results. Not sure if this could be useful to anyone but figured I would share.

Again, keep up the great work! I just bought Baldur's gate 3 and Starfield and paid extra for early access... yet here I am, playing with this for the last 24 hours.

[KillianLucas]

I just bought Baldur's gate 3 and Starfield and paid extra for early access... yet here I am, playing with this for the last 24 hours.

This is the greatest review of all time lmao

Thank you so much for using it! I really want this to be a 100% no-code experience soon, not even having to open terminal. Great to hear it's still fun to use, that's the goal exactly.

I'm happy to report that this issue should be patched as of v0.0.297. It was a problem with tokentrim's ability to handle messages with function calls, which has now been resolved. Let me know if you're still having any issues whatsoever.

[Happ1ness-dev]

not even having to open terminal.

Can you leave a cli option when you'll do it though?
It's just I'm using all this in Termux :p

[Happ1ness-dev]

Let me know if you're still having any issues whatsoever.

Hello. Today I somehow got this issue again. It's just hanging right now, forever

[Happ1ness-dev]

Yes, the bug is certainly still present

[Happ1ness-dev]

Interestingly enough, I managed to get past it this time via keyboard interrupt, lol.

But I got this error: ^CFailed to parse 'role'.

[pradical]

Yes, the bug is certainly still present

Hi. I am using Code interpreter on my windows machine, and I definitely and repeatedly have this problem too. It may have something to do with context getting overrun. I am investigating further.