Repository navigation
v1.2.0
HomeBred-LLM v1.2.0 — Run models bigger than your GPU
This release lets HomeBred-LLM run GGUF models that don't fit in VRAM by
splitting them across VRAM → RAM → disk. A new Capacity Calculator
tells you what your machine can run before you download anything.
✨ Highlights
🧠 Tiered memory for GGUF models
- Per-model memory budgets (Config → Memory tiers): set how much VRAM and RAM
a model may use, or leave both on Auto, and choose whether weights may stay on disk. - Smart placement: weights read on every token go to VRAM first. Experts and
layers that don't fit are computed on the CPU from RAM, and anything beyond
the RAM budget is read from disk on demand. - MoE-aware RAM tier: for Mixture-of-Experts models, the app keeps the
individual experts the model actually uses in RAM. You can optionally lock
them so the OS can't page them out. - Learns expert usage: while the model is idle, the app records which
experts each layer routes to. It then re-targets the RAM tier live, with no
reload, and saves the profile for the next start. - Plan preview before loading: see the VRAM/RAM/disk split, an estimated
tok/s, which part (GPU, CPU or disk) slows generation most, plus warnings
and suggestions. - KV cache type (F16 / Q8_0 / Q4_0), KV placement and flash attention are now
configurable per model.
🧮 Capacity Calculator (new page)
- "How big a model can I run?" for three strategies: GPU only,
GPU + RAM, and GPU + RAM + disk. Works for dense and MoE models, with
an optional minimum tok/s. - Estimated-speed-vs-model-size chart.
- "Can I run this?" for any local model, with suggestions: the largest
context that keeps everything on the GPU, a quant that would fit, a Q8_0 KV
cache, or how much more RAM to allot. - Hardware benchmark: measures RAM bandwidth and the read speed of your
model drive. Estimates also self-calibrate from your real runs.
🏷️ Fit badges in the Model Library
Every local model and Hugging Face file shows whether it fits on your GPU,
needs RAM offload, needs the disk tier, or is too large, with an estimated
speed.
📊 Analytics
- Live tier view for the running model.
- Prefill and decode speed reported separately, next to the estimate.
- Disk reads, PCIe traffic and page faults.
- Modeled time per token, split by where it's spent.
- MoE expert-routing heatmap.
🧪 Experimental: per-expert VRAM cache (Phase 2)
- Keeps the hottest individual experts of each MoE layer in VRAM slots; the
rest run on the CPU. Output is bit-identical to normal inference. - Requires a patched llama.cpp built from
native/llama.cpp-hbec
(build.ps1 -Backend cuda/build.sh cuda). The app loads it
automatically fromHomeBred-LLM/native/when present. - Opt-out per model: Config → Memory tiers → Per-expert VRAM cache.
⚠️ The CUDA build has not been validated on GPU hardware yet; run the
bundledhbec_testafter building.
🔧 Changes & fixes
- New GGUF models default to Tiered placement with Auto budgets.
Simple mode (GPU Layer Count only) is still available per model. - Inference stats are now recorded once per chat turn instead of being
repeated in every 5-second sample, so historical averages are more accurate. - Fixed analytics/calculator charts crashing when an axis had no data, and
made chart legends readable in the dark theme. - Config page: Save/Reset moved below all settings.
⬆️ Upgrading
- The database is upgraded automatically on first launch (new tables and
analytics columns). No action needed. - GPU note: official builds ship with the CPU llama.cpp backend
(LLamaSharp.Backend.Cpu), so the VRAM tier is disabled and the app says
so. For GPU offload, referenceLLamaSharp.Backend.Cuda12or use the
patched native build above.
🛠️ For developers
- New
TieringVerificationTestsconsole runner covering estimators, the
planner, the calculator, the warm tier, schema upgrade, and end-to-end
native tests on a tiny MoE test model. HomeBred-LLM/TestData/test_tiny_moe.ggufis git-ignored by*.gguf;
add it withgit add -f.- Docs:
native/llama.cpp-hbec/README.md,docs/hb_expertcache.h.
What's Changed
- ci: added build pipeline by @Eisenmann in #4
- Run big LLMs on the local GPU — tiered memory + per-expert VRAM cache by @Eisenmann in #5
Full Changelog: 1.1.0...v1.2.0