koboldcpp-1.70.1

koboldcpp-1.70.1

mom: we have ChatGPT at home edition

• Updated Kobold Lite:
  • Introducting Corpo Mode: A new beginner friendly UI theme that aims to emulate the ChatGPT look and feel closely, providing a clean, simple and minimalistic interface. It has a limited feature set compared to other UI themes, but should feel very familiar and intuitive for new users. Now available for instruct mode!
  • Settings Menu Rework: The settings menu has also been completely overhauled into 4 distinct panels, and should feel a lot less cramped now, especially on desktop.
  • Sampler Presets and Instruct Presets have been updated and modernized.
  • Added support for importing character cards from aicharactercards.com
  • Added copy for code blocks
  • Added support for dedicated System Tag and System Prompt (you are still encouraged to use the Memory feature instead)
  • Improved accessibility, keyboard tab navigation and screen reader support
• NEW: Official releases now provide windows binaries with included AVX1 CUDA support, download koboldcpp_oldcpu.exe
• NEW: DRY dynamic N-gram anti-repetition sampler support has been added (credits @pi6am)
• Added --unpack, a new self-extraction feature that allows KoboldCpp binary releases to be unpacked into an empty directory. This allows easy modification and access to the files and contents embedded inside the PyInstaller. Can also be used in the GUI launcher.
• Fix for a Vulkan regression in Q4_K_S mistral models when offloading to GPU (thanks @0cc4m).
• Experimental support for OpenAI tools and function calling API (credits @teddybear082)
• Added a workaround for Deepseek crashing due to unicode decoding issues.
• --chatcompletionsadapter can now be selected on included pre-bundled templates by filename, e.g. Llama-3.json, pre-bundled templates have also been updated for correctness (thanks @xzuyn).
• Default --contextsize is finally increased to 4096, default Chat Completions API output length is also increased.
• Merged fixes and improvements from upstream, including multiple Gemma fixes.

1.70.1: Fixed a bug with --unpack not including the py files, with the oldcpu binary missing some options, and swapped the cu11 linux binary to not use avx2 for best compatibility. The cu12 linux binary still uses avx2 for max performance.

To use, download and run the koboldcpp.exe, which is a one-file pyinstaller.
If you don't need CUDA, you can use koboldcpp_nocuda.exe which is much smaller.
If you have a newer Nvidia GPU, you can use the CUDA 12 version koboldcpp_cu12.exe (much larger, slightly faster).
If you're using Linux, select the appropriate Linux binary file instead (not exe).
If you're using AMD, you can try koboldcpp_rocm at YellowRoseCx's fork here

Run it from the command line with the desired launch parameters (see --help), or manually select the model in the GUI.
and then once loaded, you can connect like this (or use the full koboldai client):
http://localhost:5001

For more information, be sure to run the program from command line with the --help flag.

koboldcpp-1.69.1

koboldcpp-1.69.1

• Fixed an issue when selecting ubatch, which should now correctly match the blasbatchsize
• Added separator tokens when selecting multiple images with LLaVA. Unfortunately, the model still tends to get mixed up and confused when working with multiple images in the same request.
• Added a set of premade Chat Completions adapters selectable in the GUI launcher (thanks @henk717) which provide an easy instruct templates for various models and formats, should you want to use third party OpenAI based (chat completion) frontends along with KoboldCpp. This can help you override the instruct format even if the frontend does not directly support it. For more information on --chatcompletionsadapter see the wiki.
• Allow inserting an extra added forced positive or forced negative prompt for stable diffusion (set add_sd_prompt and add_sd_negative_prompt in a loaded adapter).
• Switched over the KoboldCpp Colab to use precompiled linux binaries, it starts and run much faster now. The Huggingface Tiefighter Space example has also been updated likewise (thanks @henk717) . Lastly, added information about using KoboldCpp in RunPod at https://koboldai.org/runpodcpp/
• Fixed some utf decode errors.
• Added tensor split GUI launcher input field for Vulkan.
• Merged fixes and improvements from upstream, including the improved mmq with int8 tensor core support and gemma 2 features have been merged.
• Updated Kobold Lite chatnames stopper for instruct mode. Also, Kobold Lite can now fall back to an alternative API or endpoint URL if the connection fails, you may attempt to reconnect using the OpenAI API instead, or to use a different URL.

1.69.1 - Merged the fixes for gemma 2 and IQ mmvq

To use, download and run the koboldcpp.exe, which is a one-file pyinstaller.
If you don't need CUDA, you can use koboldcpp_nocuda.exe which is much smaller.
If you have a newer Nvidia GPU, you can use the CUDA 12 version koboldcpp_cu12.exe (much larger, slightly faster).
If you're using Linux, select the appropriate Linux binary file instead (not exe).
If you're using AMD, you can try koboldcpp_rocm at YellowRoseCx's fork here

Run it from the command line with the desired launch parameters (see --help), or manually select the model in the GUI.
and then once loaded, you can connect like this (or use the full koboldai client):
http://localhost:5001

For more information, be sure to run the program from command line with the --help flag.

koboldcpp-1.68

koboldcpp-1.68

• Added GradientAI Automatic RoPE calculation, thanks to @askmyteapot , this should provide a better automatic RoPE scaling values for large context sizes.
• CLBlast support has been preserved, although it is now removed upstream. For now, I still intend to retain it as long as feasible.
• Multi GPU is now made easy in Vulkan, with an All GPU option in the GUI launcher added similar to cuda. Also, vulkan now defaults to the first dedicated GPU if --usevulkan is run without any other parameters, instead of just the first GPU on the list (thanks @0cc4m )
• The tokenize endpoint at /api/extra/tokencount now has an option to skip BOS tokens, by setting special to false.
• Running a KCPP horde worker now automatically sets whisper and SD to quiet mode.
• Allow the SD StableUI to be run even when no SD model is loaded.
• Allow --sdclamped to provide a custom clamp size
• Additional benchmark flags are saved (thanks @Nexesenex)
• Merged fixes and improvements from upstream
• Updated Kobold Lite:
  • Fixed Whisper not working in some versions of Firefox
  • Allow PTT to trigger a 'Generate More' if tapped, and still function as PTT if held.
  • Fixed PWA functionality, now KoboldAI Lite can be installed as a web app even when running from KoboldCpp.
  • Added a plaintext export option
  • Increase retry history stack to 3.
  • Increased default non-highres image size slightly.

Q: Why does Koboldcpp seem to constantly increase in filesize every single version?
A: Basically the upstream llama.cpp cuda maintainers believe that performance should always be prioritized over code size. Indeed, even the official llama.cpp libraries are now well over 130mb compressed without cublas runtimes, and continuing to grow in size at a geometric rate. Unfortunately, there is very little I can personally do about this.

To use, download and run the koboldcpp.exe, which is a one-file pyinstaller.
If you don't need CUDA, you can use koboldcpp_nocuda.exe which is much smaller.
If you have a newer Nvidia GPU, you can use the CUDA 12 version koboldcpp_cu12.exe (much larger, slightly faster).
If you're using Linux, select the appropriate Linux binary file instead (not exe).
If you're using AMD, you can try koboldcpp_rocm at YellowRoseCx's fork here

Run it from the command line with the desired launch parameters (see --help), or manually select the model in the GUI.
and then once loaded, you can connect like this (or use the full koboldai client):
http://localhost:5001

For more information, be sure to run the program from command line with the --help flag.

koboldcpp-1.67

koboldcpp-1.67

Hands free edition

KoboldLostAgain.mp4

• NEW: Integrated Whisper.cpp into KoboldCpp. This can be used from Kobold Lite for speech-to-text (see below). You can obtain a whisper model from the whisper.cpp repo links or download one mirrored here
  • Two new endpoints are added, /api/extra/transcribe used by KoboldCpp and the OpenAI compatible drop-in /v1/audio/transcriptions. Both endpoints accept payloads as .wav files (max 32MB), or base64 encoded wave data, please check KoboldCpp API docs for more info.
  • Can be used in Kobold Lite. Uses microphone when enabled in settings panel. You can use Push-To-Talk (PTT) or automatic Voice Activity Detection (VAD) aka Hands Free Mode, everything runs locally within your browser including resampling and wav format conversion, and interfaces directly with the KoboldCpp transcription endpoint.
  • Special thanks to @ggerganov and all the developers of whisper.cpp, without which none of this would have been possible.
• NEW: You can now utilize the Quantized KV Cache feature in KoboldCpp with --quantkv [level], where level 0=f16, 1=q8, 2=q4. Note that quantized KV cache is only available if --flashattention is used, and is NOT compatible with Context Shifting, which will be disabled if --quantkv is used.
• Merged improvements and fixes from upstream, including new MOE support for Vulkan by @0cc4m
• Fixed a bug with stable diffusion generating blank images in CPU mode.
• Updated Kobold Lite:
  • Speech-To-Text features have been added, see above.
  • Tavern Cards can now be imported in Instruct mode. Enable "Show Advanced Load" for this option.
  • Logit Bias editor now has a built-in tokenizer for strings when using with koboldcpp.
  • Fixed world info trigger probability, added escape button to close popups, fixed Cohere preamble dialog, fixed password input field sizes, various other bugfixes.

To use, download and run the koboldcpp.exe, which is a one-file pyinstaller.
If you don't need CUDA, you can use koboldcpp_nocuda.exe which is much smaller.
If you have a newer Nvidia GPU, you can use the CUDA 12 version koboldcpp_cu12.exe (much larger, slightly faster).
If you're using Linux, select the appropriate Linux binary file instead (not exe).
If you're using AMD, you can try koboldcpp_rocm at YellowRoseCx's fork here

Run it from the command line with the desired launch parameters (see --help), or manually select the model in the GUI.
and then once loaded, you can connect like this (or use the full koboldai client):
http://localhost:5001

For more information, be sure to run the program from command line with the --help flag.

koboldcpp-1.66.1

koboldcpp-1.66.1

Phi guess that's the way the cookie crumbles edition

• NEW: Added custom SD LoRA support! Specify it with --sdlora and set the LoRA multiplier with --sdloramult. Note that SD LoRAs can only be used when loading in 16bit (e.g. with the .safetensors model) and will not work on quantized models (so incompatible with --sdquant)
• NEW: Added custom SD VAE support, which can be specified in the Image Gen tab of the GUI launcher, or using --sdvae [vae_file.safetensors]
• NEW: Added in-built support for TAE SD for SD1.5 and SDXL. This is a very small VAE replacement that can be used if a model has a broken VAE, it also works faster than regular VAE. To use it, select "Fix Bad VAE" checkbox or use the flag --sdvaeauto
  • Note: Do not use the above new flags with --sdconfig, which is a deprecated flag and not to be used.
• NEW: Added experimental support for Rep Pen Slope. This is not a true slope, but the end result is it applies a slightly reduced rep pen for older tokens within the rep pen range, scaled by the slope value. Setting rep pen slope to 1 negates this effect. For compatibility reasons, rep pen slope defaults to 1 if unspecified (same behavior as before).
• NEW: You can now specify a http/https URL to a GGUF file when passing the --model parameter, or in the model selector UI. KoboldCpp will attempt to download the model file into your current working directory, and automatically load it when the download is done.
• Disable UI launcher scaling on MacOS due to display issues. Please report any further scaling issues.
• Improved EOT token handling, fixed a bug in token speed calculations.
• Default thread count will not exceed 8 unless overridden, this helps mitigate e-core issues.
• Merged improvements and fixes from upstream, including new Phi support and Vulkan fixes from @0cc4m
• Updated Kobold Lite:
  • Now attempts to function correctly if hosted on a subdirectory URL path (e.g. using a reverse proxy), if that fails it defaults back to the root URL.
  • Changed default chatmode player name from "You" to "User", which solves some wonky phrasing issues.
  • Added viewport width controls in settings, including horizontal fullscreen.
  • Minor bugfixes for markdown

Fix for 1.66.1 - Fixed quant tools makefile, fixed sd seed parsing, updated lite

To use, download and run the koboldcpp.exe, which is a one-file pyinstaller.
If you don't need CUDA, you can use koboldcpp_nocuda.exe which is much smaller.
If you have a newer Nvidia GPU, you can use the CUDA 12 version koboldcpp_cu12.exe (much larger, slightly faster).
If you're using Linux, select the appropriate Linux binary file instead (not exe).
If you're using AMD, you can try koboldcpp_rocm at YellowRoseCx's fork here

Run it from the command line with the desired launch parameters (see --help), or manually select the model in the GUI.
and then once loaded, you can connect like this (or use the full koboldai client):
http://localhost:5001

For more information, be sure to run the program from command line with the --help flag.

koboldcpp-1.65

koboldcpp-1.65

at least we have a shovel edition

• NEW: Added a new standalone UI for Image Generation, thanks to @ayunami2000 for porting StableUI (original by @aqualxx) to KoboldCpp! Now you have a powerful dedicated A1111 compatible GUI for generating images locally, with a similar look and feel to Automatic1111. And it runs in your browser, launching straight from KoboldCpp, simply load a Stable Diffusion model and visit http://localhost:5001/sdui/
• NEW: Added official CUDA 12 binaries. If you have a newer NVIDIA GPU and don't mind larger files, you may get increased speeds by using the CUDA 12 build koboldcpp_cuda12.exe
• Added a new API field bypass_eos to skip EOS tokens while still allowing them to be generated.
• Hopefully fixed tk window resizing issues
• Increased interrogate mode token amount by 30%, and increased default chat completions token amount by 250%
• Merged improvements and fixes from upstream
• Updated Kobold Lite:
  • Added option to insert Instruct System Prompt
  • Added option to bypass (skip) EOS
  • Added toggle to return special tokens
  • Added Chat Names insertion for instruct mode
  • Added button to launch StableUI
  • Various minor fixes, support importing cards from CharacterHub urls.

Important Deprecation Notice: The flags --smartcontext, --hordeconfig and --sdconfig are being deprecated.

--smartcontext is no longer as useful nowadays with context shifting, and just adds clutter and confusion. With it's removal, if contextshift is enabled, smartcontext will be used as a fallback if contextshift is unavailable, such as with old models. --noshift can still be used to turn both behaviors off.

--hordeconfig and --sdconfig are being replaced, as the number of configurations for these arguments grow, the order of these positional arguments confuses people, and makes it very difficult to add new flags and toggles as well, since a misplaced new parameter breaks existing parameters. Additionally, it also prevented me from properly validating each input for data type and range.

As this is a large change, these deprecated flags will remain functional for now. However, you are strongly advised to switch over to the new replacement flags below:

Replacement Flags:

--hordemodelname  Sets your AI Horde display model name.
--hordeworkername Sets your AI Horde worker name.
--hordekey        Sets your AI Horde API key.
--hordemaxctx     Sets the maximum context length your worker will accept.
--hordegenlen     Sets the maximum number of tokens your worker will generate.

--sdmodel     Specify a stable diffusion model to enable image generation.
--sdthreads   Use a different number of threads for image generation if specified. 
--sdquant     If specified, loads the model quantized to save memory.
--sdclamped   If specified, limit generation steps and resolution settings for shared use.

To use, download and run the koboldcpp.exe, which is a one-file pyinstaller.
If you don't need CUDA, you can use koboldcpp_nocuda.exe which is much smaller.
If you have a newer Nvidia GPU, you can use the CUDA 12 version koboldcpp_cu12.exe (much larger, slightly faster).
If you're using Linux, select the appropriate Linux binary file instead (not exe).
If you're using AMD, you can try koboldcpp_rocm at YellowRoseCx's fork here

Run it from the command line with the desired launch parameters (see --help), or manually select the model in the GUI.
and then once loaded, you can connect like this (or use the full koboldai client):
http://localhost:5001

For more information, be sure to run the program from command line with the --help flag.

koboldcpp-1.64.1

koboldcpp-1.64.1

• Added fixes for Llama 3 tokenization: Support updated Llama 3 GGUFs with pre-tokenizations.
  • Note: In order to benefit from the tokenizer fix, the GGUF models need to be reconverted after this commit. A warning will be displayed if the model was created before this fix.
• Automatically support and apply both EOS and EOT tokens. EOT tokens are also correctly biased when EOS is banned.
• finish_reason is now correctly communicated in both sync and SSE streamed modes responses when token generation is stopped by EOS/EOT. Also, Kobold Lite no longer trims sentences if a EOS/EOT is detected as the stop reason in instruct mode.
• Added proper support for trim_stop in SSE streaming modes. Stop sequences will no longer be exposed even during streaming when trim_stop is enabled. Additionally, using the Chat Completions endpoint automatically applies trim stop to the instruct tag format used. This allows better out-of-box compatibility with third party clients like LibreChat.
• --bantokens flag has been removed. Instead, you can now submit banned_tokens dynamically via the generate API, for each specific generation, and all matching tokens will be banned for that generation.
• Added render_special to the generate API, which allows you to enable rendering of special tokens like <|start_header_id|> or <|eot_id|> if enabled.
• Added new experimental flag --flashattention to enable Flash Attention for compatible models.
• Added support for resizing the GUI launcher, all GUI elements will auto-scale to fit. This can be useful for high DPI screens.
• Improved speed of rep pen sampler.
• Added additional debug information in --debugmode.
• Added a button for starting the benchmark feature in GUI launcher mode.
• Fixed slow clip processing speed issue on Colab
• Fixed quantization tool compilation again
• Updated Kobold Lite:
  • Improved stop sequence and EOS handling
  • Fixed instruct tag dropdown
  • Added token filter feature
  • Added enhanced regex replacement (now also allowed for submitted text)
  • Support custom {{placeholder}} tags.
  • Better max context handling when used in Kcpp
  • Support for Inverted world info secondary keys (triggers when NOT present)
  • Language customization for XTTS

Hotfix 1.64.1: Fixed LLAVA being incoherent on the second generation onwards. Also, the gui launcher has been tidied up, lowvram is now removed from quick launch tab and only in hardware tab. --benchmark includes version and gives clearer exit instructions in console output now. Fixed some tkinter error outputs on quit.

To use, download and run the koboldcpp.exe, which is a one-file pyinstaller.
If you don't need CUDA, you can use koboldcpp_nocuda.exe which is much smaller.
If you're using Linux, select the appropriate Linux binary file instead (not exe).
If you're using AMD, you can try koboldcpp_rocm at YellowRoseCx's fork here

Run it from the command line with the desired launch parameters (see --help), or manually select the model in the GUI.
and then once loaded, you can connect like this (or use the full koboldai client):
http://localhost:5001

For more information, be sure to run the program from command line with the --help flag.

koboldcpp-1.63

koboldcpp-1.63

Enable Sound, Press Play

kobo_gif.mp4

• Added support for special tokens in stop_sequences. Thus, if you set <|eot_id|> as a stop sequence and it can be tokenized into a single token, it will just work and function like the EOS token, allowing multiple EOS-like tokens.
• Reworked the Automatic RoPE scaling calculations to support Llama3 (just specify the desired --contextsize and it will trigger automatically).
• Added a console warning if another program is already using the desired port.
• Improved server handling for bad or empty requests, which fixes a potential flooding vulnerability.
• Fixed a scenario where the BOS token could get lost, potentially resulting in lower quality especially during context-shifting.
• Pulled and merged new model support, improvements and fixes from upstream.
• Updated Kobold Lite: Fixed markdown, reworked memory layout, added a regex replacer feature, added aesthetic background color settings, added more save slots, added usermod saving, added Llama3 prompt template

Edit: Something seems to be flagging the CI built binary on windows defender. Replaced it with a locally built one until I can figure it out.

To use, download and run the koboldcpp.exe, which is a one-file pyinstaller.
If you don't need CUDA, you can use koboldcpp_nocuda.exe which is much smaller.
If you're using AMD, you can try koboldcpp_rocm at YellowRoseCx's fork here

Run it from the command line with the desired launch parameters (see --help), or manually select the model in the GUI.
and then once loaded, you can connect like this (or use the full koboldai client):
http://localhost:5001

For more information, be sure to run the program from command line with the --help flag.

koboldcpp-1.62.2

koboldcpp-1.62.2

There and back again edition

• NEW: Img2Img is now supported when generating images using KoboldCpp. An A1111 compatible endpoint /sdapi/v1/img2img is now emulated. When using Kobold Lite, you can now click an existing image, and generate a new image based off it with Img2Img.
• NEW: OpenAI Chat Completions adapters can now be specified on load with --chatcompletionsadapter. This allows you to use any instruct tag format you want via the Chat Completions API. Please refer to the wiki for documentation. The instruct tags should now also handle all stop sequences correctly and not overflow past them when using OpenAI Chat Completions API.
• Added automatic cleanup of old orphaned koboldcpp pyinstaller temp directories.
• Added more usage statistics available in /api/extra/perf/
• Do not display localhost url if using remote tunnel
• Added /docs endpoint which is an alias for /api, containing API documentation
• Embedded Horde Worker job polling URL changed to aihorde.net
• Embedded Horde Workers will now give priority to the local user, and pause/unpause themselves briefly whenever generating on a local active client, and then returning to full speed when idle. This should allow you to comfortably run a busy horde worker, even when you want to use KoboldCpp locally at the same time.
• Try to fix SSL cert directory not found by specifying a default path.
• Fixed old quant tools not compiling
• Pulled and merged new model support, improvements and fixes from upstream.
• Updated Kobold Lite with some layout fixes, support for Cohere API, Claude Haiku and Gemini 1.5 API, and Img2Img features for local and horde.

Hotfix 1.62.1 - Merged command R plus from upstream. I cannot confirm if it works correctly as CR+ is too big for me to run locally.
Hotfix 1.62.2 - CommandR lite template and fix for appending stop sequences in chat completions

To use, download and run the koboldcpp.exe, which is a one-file pyinstaller.
If you don't need CUDA, you can use koboldcpp_nocuda.exe which is much smaller.
If you're using AMD, you can try koboldcpp_rocm at YellowRoseCx's fork here

Run it from the command line with the desired launch parameters (see --help), or manually select the model in the GUI.
and then once loaded, you can connect like this (or use the full koboldai client):
http://localhost:5001

For more information, be sure to run the program from command line with the --help flag.

CUDA 11 Cublas Libraries

This release is NOT a proper koboldcpp build!
It only contains the CUDA11 CuBLAS libraries to be packaged with KoboldCpp pyinstallers, intended for CI usage.

If you're looking for KoboldCpp, please get it from here: https://github.com/LostRuins/koboldcpp/releases/latest