title | order |
---|---|
Using KoboldCpp |
5 |
You can find the full KoboldCpp documentation here.
git clone https://github.com/LostRuins/koboldcpp
cd koboldcpp
For example, we will use OpenChat 3.5 model, which is what is used on the demo instance. There are many models to choose from.
Navigate to TheBloke/openchat_3.5-GGUF and download one of the models, such as openchat_3.5.Q5_K_M.gguf
. Place this file inside the ./models
directory.
make
./koboldcpp.py ./models/openchat_3.5.Q5_K_M.gguf
First select KoboldCpp
as the backend in the client:
settings -> ChatBot -> ChatBot Backend -> KoboldCpp
Then configure KoboldCpp
:
settings -> ChatBot -> KoboldCpp
Inside of "Use KoboldCpp" ensure that "Use Extra" is enabled. This will allow you to use the extra features of KoboldCpp, such as streaming.