DynaLM v0.2.0

DynaLM 0.2.0. The same code as 0.2.0-beta.1, now the stable release. It is verified on macOS on Apple Silicon. The Linux and Windows builds are CI-tested: built and tested in CI without real models.

Overview

DynaLM runs large language models locally, on the CPU or the GPU, as one program with no Python. It provides dynalm run, chat and serve, and an OpenAI-compatible HTTP API with streaming. Feedback from Linux and Windows users is especially welcome.

Main improvements

• Fast CPU path, accuracy against llama.cpp:
  • Granite MoE expert weights are repacked as llama.cpp does: token agreement 93.8–94.4% → 99.96–100%; prefill +26%, decode +5% (Granite-3.1-1B-A400M).
  • Gemma's query is scaled before attention: 96.9–97.1% → 100%.
• CPU scheduling: the fast CPU path no longer uses the GPU's mixed-step settings. With 4 users, tokens/s +9.5–15% and 99th-percentile token gaps −53–58%.
• Metal:
  • GPU graphs are reordered for concurrency.
  • A one-line ggml patch removes false barriers.
  • A long-running server no longer slows down as its prefix cache fills the KV pool. The sparse attention used for this is automatic for steps whose span is at least 32× the context, up to 4,096 tokens.
  • Measured on Qwen2.5-1.5B with 16 users on a cache-filled 65,536-token pool: typical token gap 135.7 → 23.3 ms, tokens/s 98.6 → 152.3.
• Benchmarks you can trust:
  • Prompts are unique across benchmark points; earlier multi-user results reused prompts and are withdrawn.
  • Each engine's backend is verified from its own log.
  • The engine order is balanced.
  • Teacher-forced parity tooling compares against llama.cpp.
• Tests: sampling regression tests (temperature, top-k, top-p, min-p, penalties) and a Metal dirty-pool attention test.

Supported environment

Artifact	Platform	Validation for this release
dynalm-macos-arm64.tar.gz	macOS on Apple Silicon, CPU and Metal	Verified on one Apple M4 Pro: full test suite, parity with llama.cpp on eight models, DD-053 GPU accuracy, multi-user benchmarks
dynalm-macos-x86_64.tar.gz	macOS on Intel, CPU	CI-tested: built and tested in CI under Rosetta (model-free tests)
dynalm-linux-x86_64.tar.gz, dynalm-linux-arm64.tar.gz	Linux, CPU	CI-tested: built, tested (model-free) and fresh-install checked in CI; not checked with real models
dynalm-windows-x86_64.zip, dynalm-windows-arm64.zip	Windows, CPU	CI-tested: as Linux

• Vulkan and CUDA are experimental. They are built in CI, but Vulkan is tested only on a software GPU (lavapipe) and CUDA only built. The Vulkan archives (dynalm-linux-x86_64-vulkan.tar.gz, dynalm-windows-x86_64-vulkan.zip) are attached as experimental builds.

• CI and Metal: CI's macOS runner has no Metal device, so Metal GPU behaviour was verified on the one physical Mac above.

• Verified models (GGUF):

  • Llama-3.1-8B Instruct Q8_0;
  • Qwen2.5-0.5B and 1.5B Instruct Q4_K_M;
  • Qwen3-0.6B Q8_0, Qwen3-4B and 8B Q4_K_M;
  • Gemma-3-1B Q4_K_M;
  • Granite-3.1-1B-A400M Q4_K_M.

  Other models and quantizations are preview.

Performance and accuracy

All results are from one Apple M4 Pro (24 GB, macOS 26.7, AC power).

Accuracy against llama.cpp (same GGUF files, teacher-forced, 10 × 512 tokens of wikitext-2):

Check	Result
CPU next-token agreement	99.96–100%, identical perplexity
Metal next-token agreement	99.80–100%
Metal greedy text	identical on 4 of 4 prompts per model
DD-053 GPU accuracy	passes on all eight models

This measures agreement with llama.cpp, not answer quality.

Speed against llama.cpp's server (Qwen2.5-1.5B, Qwen3-4B and Qwen3-8B Q4_K_M, Llama-3.1-8B Q8_0; 512-token prompts, 128 output tokens, 2,048 tokens of context per user; medians over 3 rounds):

A difference is "confirmed" only when all 3 rounds agree beyond ±10% and both engines varied by 10% or less. llama.cpp's multi-user results varied by up to about 30% between rounds, so several differences are not confirmed.

Device and load	Result	Confirmed?
Metal, one user	about the same speed	parity on the 8B models; the others inconclusive
Metal, 16 users: total tokens/s	0.71–0.78× (llama.cpp ahead)	yes, 3 of 4 models
Metal, 16 users: 99th-percentile token gaps	7–10× shorter with DynaLM	yes for the 8B models
Metal, 16 users: first token	2.1–3.3× sooner (median)	no, on all four models
Metal, 4 users: 99th-percentile token gaps	longer with DynaLM (0.19–0.30×)	no (consistent direction)
CPU, one user: time per token	0.77–0.97× llama.cpp's (the CPU builds differ: llama.cpp native with Accelerate, DynaLM portable)	yes, 3 of 4 models

DynaLM is not faster than llama.cpp overall.

Known limitations

• Verified on one physical Mac and eight models. Linux and Windows inference with real models has not been validated for this release.
• 16 users on Metal: total throughput is 0.71–0.78× llama.cpp's.
• 4 users: 99th-percentile token gaps are longer than llama.cpp's.
• CPU decode: 0.77–0.97× llama.cpp's speed.
• Memory limits are enforced at load. Llama-3.1-8B Q8_0 with 16 users × 4,096 tokens on a 24 GB Mac is refused (8.0 GiB of KV against 6.9 GiB available); use --ctx 57024 or less, or --kv q8_0.
• Granite MoE on the CPU uses about 0.7 GB more memory for its repacked expert weights.
• Sparse attention keeps a cache-filled Metal server fast for contexts up to 4,096 tokens only; longer contexts still slow down. Vulkan, CUDA and the CPU path don't use it.
• Vulkan and CUDA are experimental. Some models (YaRN RoPE scaling; MoE on CUDA) run on the CPU.

Installation

Linux and macOS:

curl -fsSL https://raw.githubusercontent.com/dynalm-project/dynalm/main/scripts/install.sh | sh
dynalm --version   # DynaLM   0.2.0

Windows (PowerShell): irm https://raw.githubusercontent.com/dynalm-project/dynalm/main/scripts/install.ps1 | iex. Each archive is listed in SHA256SUMS and carries signed build provenance: gh attestation verify <file> --repo dynalm-project/dynalm. To build from source, see docs/build.md.

Changes and compatibility

• --backend native-cpu is no longer available. It stops with an error naming the replacement:

  • --backend cpu for the fast CPU kernels;
  • --mode exact for DynaLM's own DD-053 CPU kernels.

  Configuration files with runtime.device: native-cpu need runtime.device: cpu and runtime.mode: exact.

• DYNACORE_GPU_SPARSE: 0 restores the previous dense-only Metal attention, 1 forces sparse attention.

• dynalm benchmark --prompt-base N: new option.

• Docker: ghcr.io/dynalm-project/dynalm:cpu is updated to 0.2.0.

What's Changed

• AutoTune, ARM int8 kernels, model compatibility and accuracy fixes by @sajosam in #5
• Post-E2E fixes: Metal KV budget, tokenizer-aware support, decidable A/B by @sajosam in #6
• Prepare DynaLM beta release by @sajosam in #7
• Release 0.2.0-beta.1: version source and beta-safe release workflow by @sajosam in #8
• docs: label Linux and Windows builds CI-tested by @sajosam in #9
• Release 0.2.0 by @sajosam in #10

Full Changelog: v0.1.7...v0.2.0