v0.7.1

See CHANGELOG.md

0.6.1 - translation eval (FLORES, 101 languages) + translation-aware routing

A translation benchmark, and routing that knows when a request is a translation.

New

• localllm eval --suites translate runs FLORES-101 in 101 languages. It translates the first 100 sentences each way (English to the language and back) and scores them with chrF++. The score matches sacreBLEU to the second decimal and needs no extra dependency.
• Routing detects translation requests ("translate", แปล, 翻译, 翻訳, 번역, ترجم, अनुवाद, перевед, ...) and picks the model by its measured translation score.
• doctor now shows your language's translation score.

Measured on RX 9070 XT (chrF++, en<->xx, 100 sentences each way)

	Thai	Chinese	Japanese	Hindi	Arabic
gemma-4-26B-A4B QAT	55.6	47.7	48.0	62.8	61.9
Qwen3.8-27B Q3	54.1	49.1	46.4	58.1	59.0

Effect on routing (verified on the GPU from the installed wheel):

• A Chinese translation request now stays on gemma (Qwen leads Chinese translation by only 1.4) instead of costing an 8-9 s swap to Qwen.
• Chinese knowledge questions still go to Qwen (+5.5).

Release gate: 44 tests; CI on Linux/Windows/macOS; fresh-venv wheel install with --models auto on the GPU; pip install make-localllm-easier==0.6.1 from PyPI.

0.6.0 - smart routing between local models (opt-in)

Smart routing between local models, opt-in, with honest numbers.

New

• localllm serve --models auto (or --models A,B). Each message goes to the model with the best measured score for its language and task (math vs general). Every answer reports which model wrote it and why: the X-Localllm-Model header, also shown under each answer in localllm chat.
  • Resident mode: if all models fit in VRAM together, they all stay loaded and routing is instant.
  • Lazy-load mode: otherwise one model is loaded at a time, and it swaps only when another model is at least 3 points better.
• Math eval: localllm eval --suites math runs MGSM, the same 250 word problems in 11 languages.
• Faster start: the VRAM spill check now runs in the background (saves ~1.5 s per load), and health checks poll every 100 ms.
• Fixes: doctor labels math scores correctly, and the cloud language-floor rule uses knowledge scores only.

Measured on RX 9070 XT 16 GB / 32 GB RAM, from the installed wheel

Message	Routed to	Why
Thai / English	gemma-4-26B-A4B QAT	within 2 pts of Qwen, 1.8x faster
Chinese knowledge	Qwen3.8-27B Q3	+5.5 pts (swap 8.3 s)
Chinese math	gemma-4-26B-A4B QAT	+4.4 pts on MGSM (swap 9.2 s)

MGSM math: gemma 96.8 / 89.6 / 88.8 vs Qwen 94.4 / 87.2 / 84.4 (en / th / zh).

Why routing is not the default: the roadmap gate says to ship routing by default only if quality AND latency beat a single model. On a 16 GB card:

• Only Chinese and Spanish gain 3+ points.
• A swap costs 4-9 s. It is 8.6 s when alternating two models, because the page cache can't hold both on 32 GB RAM, and 4.1 s warm with 16 MiB upload buffers.
• Keeping both models on the card through Windows VRAM paging made both 4-5x slower, so that approach is rejected.

On 24 GB+ cards resident mode avoids swaps entirely. It is tested with stand-in servers only so far, so results from bigger GPUs are very welcome. Faster switching is tracked in #32.

Release gate: 41 tests; CI on Linux/Windows/macOS; a fresh-venv wheel install serving on the GPU with default and --models auto (Thai, English, Chinese and Chinese-math answers, eval sanity th 75% / en 90% on 20 questions); pip install make-localllm-easier==0.6.0 from PyPI.

0.5.0 - per-language compression research

Compression research, measured per language - not English perplexity.

New

• Quality floor: localllm doctor warns when the best model that fits is below ~2.7 bits per weight (post-training quants; vendor QAT checkpoints are exempt).
• QAT tie-break: when two models are within the benchmark margin, a vendor QAT checkpoint wins over a post-training quant.
• tools/kld_per_language.py: KL divergence and top-1 agreement of any GGUF vs a reference model, one score per language.

Measured on RX 9070 XT (Qwen3.8-27B vs its Q8_0, held-out Wikipedia text)

Quant	Size	Thai	Hindi	Arabic	English
UD-Q3_K_XL	12.2 GB	0.034 / 91.7%	0.037 / 90.2%	0.078 / 92.5%	0.021 / 93.2%
UD-IQ2_S	7.8 GB	0.188 / 81.5%	0.224 / 77.5%	0.291 / 84.4%	0.111 / 86.0%
IQ2_S, our mixed imatrix, from BF16	7.8 GB	0.210 / 81.8%	0.215 / 77.1%	0.365 / 82.1%	0.137 / 83.6%

(mean KLD / same top token; lower KLD is better)

Findings

• Dropping from 3 to 2 bits multiplies KL divergence 4-6x in every language, and top-token agreement ends 2-9 points lower in Arabic, Thai and Hindi than in English.
• Build 2-bit files from BF16: re-quantizing from Q8 cost Thai ~9 points. A Thai or mixed-language importance matrix recovered most of that, but from BF16 our mixed matrix lost to Unsloth's own UD-IQ2_S in 3 of 4 languages, so we did not publish it.

Release gate: 35 tests, CI on Linux/Windows/macOS, a fresh-venv install of the wheel serving gemma-4-26B-A4B on the GPU (Thai + English at 127 tok/s, eval sanity th 75% / en 90% on 20 questions), and pip install make-localllm-easier==0.5.0 from PyPI.

Still open research (0.5 issues): better 2-bit formats, LoRA self-distillation, vocabulary trimming, and sensitivity-aware recipes.

v0.4.0 - 2.5x faster on the same GPU with localllm tune

2.5x faster on the same GPU - measured, not guessed

New: localllm tune runs your model a few times with candidate llama.cpp settings and keeps a setting only if it is at least 1.1x faster on your PC. Every later launch uses the result.

On an RX 9070 XT (Windows, Vulkan, Resizable BAR off):

model	before	after localllm tune
Qwen3.8-27B Q3	21.7 tok/s	55 tok/s (2.5x) - small-BAR fix 1.65x, then MTP drafting 2 tokens
gemma-4-26B-A4B QAT	117 tok/s with the fix	128 tok/s - tune found the fix hurts this model and left it off

What changed

• The Resizable-BAR fix is now per model. It makes hybrid DeltaNet models (Qwen3.5/3.8) 1.64x faster but gemma-4 7-9% slower, so localllm applies it only where it was measured to help; tune decides on your PC. It also turned out to fix the DeltaNet state-copy overhead (8.2 -> 0.9 ms per token).
• llama.cpp treats GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM=0 as on; "off" now means unset.
• MTP draft length is measured (2 beats 3 on RDNA4) instead of assumed.
• doctor shows the speed tune measured on this PC.
• NVIDIA: tune also tries GGML_CUDA_GRAPH_OPT=1 - not measured by us yet, results from NVIDIA owners welcome (#22).

pip install -U make-localllm-easier then localllm tune

v0.3.0 - one endpoint for OpenAI, Anthropic, Ollama and Gemini apps

One local endpoint for the APIs your apps already speak

localllm / localllm serve now puts a gateway at http://127.0.0.1:8080 in front of the local model. Change only the base URL:

your app speaks	base URL
OpenAI	http://127.0.0.1:8080/v1
Anthropic Messages	http://127.0.0.1:8080
Ollama	http://127.0.0.1:8080
Gemini	http://127.0.0.1:8080/v1beta

Verified end to end with the official OpenAI, Anthropic and Ollama SDKs and Gemini REST, streaming included, all answered by the local model (gemma-4-26B-A4B, ~110 tok/s on RX 9070 XT). The built-in chat page and localllm chat work through it too.

Optional: mix in your own cloud key (off by default)

With ~/.localllm/route.json enabled and a key in an environment variable (or the OS keychain), a request goes to your cloud provider only when a rule says so: the app asked for a cloud model by name, the prompt is too long for local, or the local model's measured score in the message's language is below your floor. Every response says where it went in X-Localllm-Route. localllm route --test "prompt" compares local vs cloud: answer, time, tokens, cost. See docs/apis.md.

Cloud routing is covered by tests against stand-in servers; a live run with a real key is for you to try.

pip install -U make-localllm-easier

v0.2.0 - 4x less RAM, terminal chat

4x less RAM, same speed

llama-server's defaults keep up to an 8 GiB prompt cache plus 32 conversation checkpoints in system RAM, so RAM keeps growing while you chat. localllm now sizes both to your PC. Same 30-turn chat, RX 9070 XT, 32 GB RAM:

model	llama.cpp defaults	localllm 0.2	speed
Qwen3.8-27B Q3	9.30 GB	2.37 GB	35.2 tok/s both
gemma-4-26B-A4B QAT	9.24 GB	2.21 GB	~85 tok/s both

~15% less CPU per answer, ~0% CPU while idle.

New

• localllm chat - chat in the terminal: streaming, history, /save, /think, Ctrl+C stops an answer.
• Smaller GPUs can run gemma-4: when it doesn't fit whole, some layers' experts stay in RAM (measured 45 / 36 / 31 tok/s on 12 / 10 / 8 GB-class cards) - more accurate than a 2-bit model that fits.
• doctor shows system RAM the model will use and what stays free (thanks @azmi2409, #12 - our first outside contribution!).
• Warning when VRAM spills into system RAM on Windows.
• Weights are always read straight into place (mmap kept the whole file resident: 13.5 vs 6.1 GB).

Research

• 2-bit + calibrating the quantization on your languages helps: +6 Thai, best average with a mixed multilingual set (#24).

pip install -U make-localllm-easier

v0.1.0 - one command, best measured local LLM

First release: one command runs the best local LLM your PC can handle.

pip install make-localllm-easier
localllm

• localllm - detect GPU/RAM, pick the most accurate measured model for your language, download llama.cpp + model, tuned server, chat page
• localllm doctor - which model sizes your GPU suits, speed estimates, how much text the model holds
• localllm eval - Global-MMLU-Lite (23 languages) + INCLUDE (44 countries) + ThaiExam
• Tuned launch: Resizable-BAR-off fix (1.7x decode on RX 9070 XT), MTP drafting (+40%), single-slot unified KV

Measured on RX 9070 XT 16 GB: Qwen3.8-27B Q3 81.8% English @ ~50 tok/s, gemma-4-26B-A4B QAT 82.2% @ ~69 tok/s; 2-bit costs 8-13 points (Hindi/Arabic/Thai most). Roadmap: less RAM (0.2), cloud API interop (0.3), GPU squeeze (0.4), per-language compression research (0.5).