Chat is very slow when using with the llama.cpp server

[lehuythangit]

Validations

• I believe this is a way to improve. I'll try to join the Continue Discord for questions
• I'm not able to find an open issue that requests the same enhancement

Problem

Chat is very slow when using with the llama.cpp server when increasing the message because missing cache_prompt = true when call the /completion API of llama.cpp, so llama.cpp process also all previous message history when prompt instead of using from cache.

Solution

Please help add more property cache_prompt = true when call the /completion API,
or add more configuration property into the config.json

Thanks

[lehuythangit]

i just workaround with modified the .vscode\extensions\continue.continue-0.8.55-win32-x64\out\extension.js

[lehuythangit]

The llama.cpp is enabled cache_prompt by default from the build https://github.com/ggerganov/llama.cpp/releases/tag/b4170
So, please close this issue.

Thanks for your time.