koboldcpp-1.117

koboldcpp-1.117

• Fixed terminal output sometimes not showing thinking traces
• Multimodal Images are now inserted inline instead of at the top of the context. This should improve vision coherence and quality on most models.
• llama.cpp UI mic input is now disabled by default (toggle in settings)
• Prevent MTP drafting for parallel requests as it's not currently supported.
• Added Ollama compatible embeddings endpoint
• Added Ollama compatible streaming and tool calling. This make KoboldCpp fully compatible with all Ollama using software, simply set the port to 11434 and it should work out of the box.
• Merged stable-diffusion.cpp image gen updates, thanks @wbruna
• Save and display Image LoRA information when generating an image with pre-loaded LoRA.
• Increase maximum defaultgenamount to 32k
• Allow viewing the last generated image if SDUI crashes or loses connection (requires the correct genkey)
• Updated Kobold Lite, multiple fixes and improvements
  • NEW: Group chat rework: Multi-character avatar and memory management has been added, thanks to @hu-yijie
  • Minor audio embed fix by @mosujiba
• Merged fixes, new model support, and improvements from upstream

Notice: This will be the last version that supports --splitmode row in CUDA as it has been removed upstream. For those use cases, you'll have to migrate to tensor split or layer split.

Download and run the koboldcpp.exe (Windows) or koboldcpp-linux-x64 (Linux), which is a one-file pyinstaller for NVIDIA GPU users.
If you have an older CPU or older NVIDIA GPU and koboldcpp does not work, try oldpc version instead (Cuda11 + AVX1).
If you don't have an NVIDIA GPU, or do not need CUDA, you can use the nocuda version which is smaller.
If you're using AMD, we recommend trying the Vulkan option in the nocuda build first, for best support. Alternatively, you can download our rolling ROCm binary here if you use Linux.
If you're on a modern MacOS (M-Series) you can use the koboldcpp-mac-arm64 MacOS binary.
Click here for .gguf conversion and quantization tools
Newer rolling experimental builds can be found here, these are auto-updated and may be unstable.

Run it from the command line with the desired launch parameters (see --help), or manually select the model in the GUI.
and then once loaded, you can connect like this (or use the full koboldai client):
http://localhost:5001

For more information, be sure to run the program from command line with the --help flag. You can also refer to the readme and the wiki.