Testing and results for measuring performance loss of LLM models with the effects of quantization and context size.
I didn't find many results that filled up a models context in a reproducable way to test their abilities.
I'm working on 3 private benchmarks to test the abilities of a model to accurately perform tasks that is expected of them. Since we are running local models, it's important to see the effects of quantization on longer performance, especially on longer contexts.
Consists of Question and example Answer paring, as well as the marking schema to see if the answer from the LLM has mentioned the factual events, remembering the key facts. I would expect smaller models performance to drop off at a small but noticable rate when quantized and filling up longer context.
Model is asked to answer complex math questions by outputting a snippet of python code that should return the correct value.
MOdel is provided an simulated user message that should prompt the correct toolcall.
My abilities to generate a private benchmark is limited to a mix of sythentic and random combinations but with reference to existing factual sources. My manual review is limited and benchmarks wont (absolutly impossible) achieve 100%.
My budget and local hosting abilities is limited. I will test smaller models or limit my testing range.
Context sizes: 4096, 8192, 16384, 32768, 65536, 131072, 262144
F16
UD-Q8_K_XL
UD-Q6_K_XL
UD-Q5_K_XL
Q5_K_S
UD-Q4_K_XL
Q4_K_S
IQ4_NL
UD-Q3_K_XL
UD-IQ3_XXS
UD-IQ2_M
Preliminary HistoryQA results will be ready soon.