Frankenstein 1.68b "Fork" of KoboldCPP Experimental up to the 11/06/2024, 16h GMT+2.
Based on Llama.CPP b3132.
DISCLAIMER:
The KoboldCPP-Frankenstein builds are not supported by the KoboldCPP team, Github, or discord channel. They are for greedy-test and amusement only.
My KCPP-Frankenstein version number bumps as soon as the version number in the official experimental branch bumps. They are not "upgrades" over the official version. And they might be bugged at time: only the official KCPP releases are to be considered correctly numbered, reliable and "fixed".
The LllamaCPP version + the additional PRs integrated follow my KCPP-Frankenstein versioning in the title, so everybody knows what version they deal with.
For KCPP official version, it's here : https://github.com/LostRuins/koboldcpp/releases
FRANKENSTEIN FEATURES:
Enhanced benchmark (reflecting a maximum of indicators, including the KV cache option)
18 KV cache options (all should be considered experimental except F16 and KV Q8_0) 👍
F16 -> Fullproof (the usual KV quant since the beginning of LCPP/KCPP)
K F16 with : V Q8_0, Q5_1, Q5_0, Q4_1, Q4_0
K Q8_0 with : V Q8_0 (stable, my current main, part of the LCPP/KCPP main triplet), Q5_1 (maybe unstable), Q5_0 (maybe unstable), Q4_1 (maybe stable), the rest is untested beyond benches), Q4_0
K Q5_1 with : V Q5_1, Q5_0, Q4_1, Q4_0
K Q5_0 with : V Q4_0
K Q4_1 with : V Q4_0
KV Q4_0 (quite stable, if we consider that it's part of the LCPP/KCPP main triplet)
Works in command line, normally also via the GUI, and normally saves on .KCPPS config files.
A better Autorope thanks to askmyteapot's PR on KoboldCPP official github.
ARGUMENTS 👍
Note : I had to use a simple 0-17 numbering scheme to allow the GUI and the kcpps preset saving to work properly. The problems with the previous 4 numbers quant scheme are fixed.
--quantkv",
help="Sets the KV cache data type quantization.
0 = 1616/F16 (16 BPW),
1 = 1680/Kf16-Vq8_0 (12.25BPW),
2 = 1651/Kf16-Vq5_1 (11BPW),
3 = 1650/Kf16-Vq5_0 (10.75BPW),
4 = 1641/Kf16-Vq4_1 (10.5BPW),
5 = 1640/Kf16-Vq4_0 (10.25BPW),
6 = 8080/KVq8_0 (8.5 BPW),
7 = 8051/Kq8_0-Vq5_1 (7.25BPW),
8 = 8050/Kq8_0-Vq5_0 (7BPW),
9 = 8041/Kq8_0-Vq4_1 (6.75BPW),
10 = 8040/Kq8_0-Vq4_0 (6.5BPW),
11 = 5151/KVq5_1 (6BPW),
12 = 5150/Kq5_1-Vq5_0 (5.75BPW),
13 = 5141/Kq5_1-Vq4_1 (5.5BPW),
14 = 5140/Kq5_1-Vq4_0 (5.25BPW),
15 = 5040/Kq5_0-Vq4_0 (5BPW),
16 = 4140/Kq4_1-Vq4_0 (4.75BPW),
17 = 4040/KVq4_0 (4.5BPW)
choices=[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17], default=0)
Lowvram option's speed is (logically) boosted due to the smaller KV context in RAM. From 25%+ in KV Q8_0 to 50%+ in KV Q4_0.
REMARKS :
You MUST use Flash attention
(tag : --flashattention in CL, or in the GUI)
Contextshift doesn't work with anything else than KV F16, but Smartcontext does.
CREDITS :
Of course, all credits go to Concedo/LostRuins and the other contributors to KoboldCPP, and to GGermanov and all the other contributors to LlamaCPP. Special big-up to Johannes Gaessler for the quantized KV cache!
I'm just poking, merging, and building around their work.
BUILDS :
All builds, aka. Cublas 12.1/12.2 (recommended for Ada Lovelace and Ampere for additional PP speed), Cublas 11.4.4/11.7 (more pertinent than the 12.2 for Pascal, Maxwell.. And 11.4.4 for Kepler), and the standard one are including OpenBLAS, CLBLAST, and Vulkan support provided by the devs.
What's Changed
-
b3132 by @Nexesenex in #156
-
Direct I/O and Transparent HugePages by Pavel Fatin in ggerganov#7420
-
GradientAI Auto ROPE Base calculation by @askmyteapot in LostRuins#910
Full Changelog: v1.68a_b3124+1...v1.68b_b3132+1
