[BUG] SillyTavern takes >15 seconds to send a request to proxy server with long conversations (>6000 context tokens)

[rsmyst]

Describe the bug
Basically, when conversations exceed about 5500 or 6000 tokesn in context, the time taken for silly to make a request to the proxy server increases to around 15-20 seconds. that is, thw browser basically shows the loading symbol for 15-20 seconds but console shows no request being made or any changes. but after ~15 seconds approx, the console finally updates and shows the change.
Also, tested it on both firefox and brave, same results.
TLDR:
Conversations take <3 seconds to send request otherwise in the beginning of conversation, increasing to around 15-20 seconds to even send request to proxy server by the end.

Also tried restarting silly tavern a few times (closing cmd and reopening using .bat script), did not help.

EDIT: I should also add that this is same across multiple proxies with same chat. for example, in my case, switching between todd proxy and gpt4 proxy had no change becuase issue seems to be caused from tavern's end and not the proxy itself.

Edit2: the tiem to send a request is about directly proportional to context length. reducing from 7K context to 4.5K context dramatically improved performance from tavern so it's definitely a tavern side thing rather than proxy side becuase time to send context to proxy is WAY faster at 4.5K (around 4 seconds) compared to 15-20 seconds with 7K context (even if it's not filled up even)

[Cohee1207]

You did not specify the branch you're using, please use the bug template next time.

I assume this relates to tiktoken tokenizer being instantiated for every tokenization request. I tested the strategy for some time in automated group chat, but this is still an experimental fix as I'm yet unsure about possible memory leaks or other implications. Please pull from the dev and test if it's any better for you. The first request in a new chat or after reloading the tab can be slow as the message token count cache builds but will be faster in the consequent requests.

[rsmyst]

@Cohee1207
I used sillytavern 1.4.9 earlier
Ok. Done as you said with the dev version of silly.
No improvement. In fact, it seems to be slower? I set context up to 7640 and no request is sent for 2+ minutes. in fact, it's still loading as I type this report out. I am timing this on the third generation as well in case you were wondering.
So the issue seems very bad on higher contexts. No request is being sent for minutes on end (no console change at all. it's been 3 minutes)

[Cohee1207]

Just a quick question, maybe perhaps you have a monitor with high refresh rate (160Hz or more?)

[rsmyst]

Just a quick question, maybe perhaps you have a monitor with high refresh rate (160Hz or more?)

sorry, I coudn't reply earlier for some reason.
but no, just a standard 60Hz laptop screen.

[hmage]

I've encountered the same issue, and as a software engineer, I've observed the following:

1. The problem occurs with extremely long chats.
2. As the context length increases, it takes longer for the JSON to appear in the server's stdout.
3. When pressing 'send', there is 100% CPU usage on a single core, a sign of JS. In top - 100% of browser process, running Tavern's JS.
4. With 32,000 tokens for GPT-4 32k, the process takes an excessive amount of time, making it more noticeable.

What I suspect, without looking into the code, that client-side JavaScript is using a brute-force linear approach, retrying in-memory several variations of number of messages until it starts/stops fitting, and only then sending the JS to API. Could be probably optimized to make it single-pass.

This issue wasn't prevalent before, possibly due to an increase in users, a lack of testing with 32k context, and the developer not having any reason to investigate until now.

I hope this information helps in reproducing and resolving the issue.

[Cohee1207]

@hmage thanks a lot for the helpful insights. I was able to add a couple of possible improvements to address this:

1. Tokenization of OAI messages doesn't block the main thread, so the page wouldn't freeze on building the prompt.
2. Disabled the tokenization of a parallel prompt builder if the OAI API is selected. On a very long chat (400 Kb) I was able to increase the speed of sending the first request 3 times.
   Another possible improvement would be to save token counts to message metadata to not reset the token counts cache on page refresh. Pushed points 1 and 2 to dev branch.

[Cohee1207]

@rsmyst please check if it's any better for you after the recent optimizations. These are in the dev branch.

[rsmyst]

@Cohee1207
I tested dev branch with the longest chat I had (about 83KB), the performance is slightly better but it's still quite slow with long chats and long contexts (about 5000 tokens context). Takes about ~7-9 seconds still

[aetherwu]

I recently tested it on my 4090 and A100, using TheBloke_airoboros-33B-gpt4-1-4-SuperHOT-8K-GPTQ.

With A100, 8K context works flowlessly in release version with no problem at all.

The performance issue mentioned above is observed on 4090. When dealing with longer contexts (over 5k or 6k), it takes a noticeable amount of time (10-30+ seconds) to send a request to llm. At times, it seems like the process gets stuck without any response before eventually moving forward.

The response for 8K + 4090 lowers to 1-2 T/S, I think is is understandable.
Just for references.

[Cohee1207]

Added caching of token counts to release branch for all APIs and tokenizers, including LLaMA. Please reopen if that persists.