DynaLM v0.2.0

DynaLM 0.2.0. The same code as 0.2.0-beta.1, now the stable release. It is verified on macOS on Apple Silicon. The Linux and Windows builds are CI-tested: built and tested in CI without real models.

Overview

DynaLM runs large language models locally, on the CPU or the GPU, as one program with no Python. It provides dynalm run, chat and serve, and an OpenAI-compatible HTTP API with streaming. Feedback from Linux and Windows users is especially welcome.

Main improvements

• Fast CPU path, accuracy against llama.cpp:
  • Granite MoE expert weights are repacked as llama.cpp does: token agreement 93.8–94.4% → 99.96–100%; prefill +26%, decode +5% (Granite-3.1-1B-A400M).
  • Gemma's query is scaled before attention: 96.9–97.1% → 100%.
• CPU scheduling: the fast CPU path no longer uses the GPU's mixed-step settings. With 4 users, tokens/s +9.5–15% and 99th-percentile token gaps −53–58%.
• Metal:
  • GPU graphs are reordered for concurrency.
  • A one-line ggml patch removes false barriers.
  • A long-running server no longer slows down as its prefix cache fills the KV pool. The sparse attention used for this is automatic for steps whose span is at least 32× the context, up to 4,096 tokens.
  • Measured on Qwen2.5-1.5B with 16 users on a cache-filled 65,536-token pool: typical token gap 135.7 → 23.3 ms, tokens/s 98.6 → 152.3.
• Benchmarks you can trust:
  • Prompts are unique across benchmark points; earlier multi-user results reused prompts and are withdrawn.
  • Each engine's backend is verified from its own log.
  • The engine order is balanced.
  • Teacher-forced parity tooling compares against llama.cpp.
• Tests: sampling regression tests (temperature, top-k, top-p, min-p, penalties) and a Metal dirty-pool attention test.

Supported environment

Artifact	Platform	Validation for this release
dynalm-macos-arm64.tar.gz	macOS on Apple Silicon, CPU and Metal	Verified on one Apple M4 Pro: full test suite, parity with llama.cpp on eight models, DD-053 GPU accuracy, multi-user benchmarks
dynalm-macos-x86_64.tar.gz	macOS on Intel, CPU	CI-tested: built and tested in CI under Rosetta (model-free tests)
dynalm-linux-x86_64.tar.gz, dynalm-linux-arm64.tar.gz	Linux, CPU	CI-tested: built, tested (model-free) and fresh-install checked in CI; not checked with real models
dynalm-windows-x86_64.zip, dynalm-windows-arm64.zip	Windows, CPU	CI-tested: as Linux

• Vulkan and CUDA are experimental. They are built in CI, but Vulkan is tested only on a software GPU (lavapipe) and CUDA only built. The Vulkan archives (dynalm-linux-x86_64-vulkan.tar.gz, dynalm-windows-x86_64-vulkan.zip) are attached as experimental builds.

• CI and Metal: CI's macOS runner has no Metal device, so Metal GPU behaviour was verified on the one physical Mac above.

• Verified models (GGUF):

  • Llama-3.1-8B Instruct Q8_0;
  • Qwen2.5-0.5B and 1.5B Instruct Q4_K_M;
  • Qwen3-0.6B Q8_0, Qwen3-4B and 8B Q4_K_M;
  • Gemma-3-1B Q4_K_M;
  • Granite-3.1-1B-A400M Q4_K_M.

  Other models and quantizations are preview.

Performance and accuracy

All results are from one Apple M4 Pro (24 GB, macOS 26.7, AC power).

Accuracy against llama.cpp (same GGUF files, teacher-forced, 10 × 512 tokens of wikitext-2):

Check	Result
CPU next-token agreement	99.96–100%, identical perplexity
Metal next-token agreement	99.80–100%
Metal greedy text	identical on 4 of 4 prompts per model
DD-053 GPU accuracy	passes on all eight models

This measures agreement with llama.cpp, not answer quality.

Speed against llama.cpp's server (Qwen2.5-1.5B, Qwen3-4B and Qwen3-8B Q4_K_M, Llama-3.1-8B Q8_0; 512-token prompts, 128 output tokens, 2,048 tokens of context per user; medians over 3 rounds):

A difference is "confirmed" only when all 3 rounds agree beyond ±10% and both engines varied by 10% or less. llama.cpp's multi-user results varied by up to about 30% between rounds, so several differences are not confirmed.

Device and load	Result	Confirmed?
Metal, one user	about the same speed	parity on the 8B models; the others inconclusive
Metal, 16 users: total tokens/s	0.71–0.78× (llama.cpp ahead)	yes, 3 of 4 models
Metal, 16 users: 99th-percentile token gaps	7–10× shorter with DynaLM	yes for the 8B models
Metal, 16 users: first token	2.1–3.3× sooner (median)	no, on all four models
Metal, 4 users: 99th-percentile token gaps	longer with DynaLM (0.19–0.30×)	no (consistent direction)
CPU, one user: time per token	0.77–0.97× llama.cpp's (the CPU builds differ: llama.cpp native with Accelerate, DynaLM portable)	yes, 3 of 4 models

DynaLM is not faster than llama.cpp overall.

Known limitations

• Verified on one physical Mac and eight models. Linux and Windows inference with real models has not been validated for this release.
• 16 users on Metal: total throughput is 0.71–0.78× llama.cpp's.
• 4 users: 99th-percentile token gaps are longer than llama.cpp's.
• CPU decode: 0.77–0.97× llama.cpp's speed.
• Memory limits are enforced at load. Llama-3.1-8B Q8_0 with 16 users × 4,096 tokens on a 24 GB Mac is refused (8.0 GiB of KV against 6.9 GiB available); use --ctx 57024 or less, or --kv q8_0.
• Granite MoE on the CPU uses about 0.7 GB more memory for its repacked expert weights.
• Sparse attention keeps a cache-filled Metal server fast for contexts up to 4,096 tokens only; longer contexts still slow down. Vulkan, CUDA and the CPU path don't use it.
• Vulkan and CUDA are experimental. Some models (YaRN RoPE scaling; MoE on CUDA) run on the CPU.

Installation

Linux and macOS:

curl -fsSL https://raw.githubusercontent.com/dynalm-project/dynalm/main/scripts/install.sh | sh
dynalm --version   # DynaLM   0.2.0

Windows (PowerShell): irm https://raw.githubusercontent.com/dynalm-project/dynalm/main/scripts/install.ps1 | iex. Each archive is listed in SHA256SUMS and carries signed build provenance: gh attestation verify <file> --repo dynalm-project/dynalm. To build from source, see docs/build.md.

Changes and compatibility

• --backend native-cpu is no longer available. It stops with an error naming the replacement:

  • --backend cpu for the fast CPU kernels;
  • --mode exact for DynaLM's own DD-053 CPU kernels.

  Configuration files with runtime.device: native-cpu need runtime.device: cpu and runtime.mode: exact.

• DYNACORE_GPU_SPARSE: 0 restores the previous dense-only Metal attention, 1 forces sparse attention.

• dynalm benchmark --prompt-base N: new option.

• Docker: ghcr.io/dynalm-project/dynalm:cpu is updated to 0.2.0.

What's Changed

• AutoTune, ARM int8 kernels, model compatibility and accuracy fixes by @sajosam in #5
• Post-E2E fixes: Metal KV budget, tokenizer-aware support, decidable A/B by @sajosam in #6
• Prepare DynaLM beta release by @sajosam in #7
• Release 0.2.0-beta.1: version source and beta-safe release workflow by @sajosam in #8
• docs: label Linux and Windows builds CI-tested by @sajosam in #9
• Release 0.2.0 by @sajosam in #10

Full Changelog: v0.1.7...v0.2.0

DynaLM v0.2.0-beta.1

DynaLM 0.2.0-beta.1: a beta pre-release. It is verified on macOS on Apple Silicon. The Linux and Windows builds are CI-tested: built and tested in CI without real models. It is not a stable release, and it does not replace 0.1.7 as the latest release.

Overview

DynaLM runs large language models locally, on the CPU or the GPU, as one program with no Python. It provides dynalm run, chat and serve, and an OpenAI-compatible HTTP API with streaming. This beta is for developers who want to try it on an Apple Silicon Mac and report what they find.

Main improvements

• Fast CPU path, accuracy against llama.cpp:
  • Granite MoE expert weights are repacked as llama.cpp does: token agreement 93.8–94.4% → 99.96–100%; prefill +26%, decode +5% (Granite-3.1-1B-A400M).
  • Gemma's query is scaled before attention: 96.9–97.1% → 100%.
• CPU scheduling: the fast CPU path no longer uses the GPU's mixed-step settings. With 4 users, tokens/s +9.5–15% and 99th-percentile token gaps −53–58%.
• Metal:
  • GPU graphs are reordered for concurrency.
  • A one-line ggml patch removes false barriers.
  • A long-running server no longer slows down as its prefix cache fills the KV pool. The sparse attention used for this is automatic for steps whose span is at least 32× the context, up to 4,096 tokens.
  • Measured on Qwen2.5-1.5B with 16 users on a cache-filled 65,536-token pool: typical token gap 135.7 → 23.3 ms, tokens/s 98.6 → 152.3.
• Benchmarks you can trust:
  • Prompts are unique across benchmark points; earlier multi-user results reused prompts and are withdrawn.
  • Each engine's backend is verified from its own log.
  • The engine order is balanced.
  • Teacher-forced parity tooling compares against llama.cpp.
• Tests: sampling regression tests (temperature, top-k, top-p, min-p, penalties) and a Metal dirty-pool attention test.

Supported beta environment

Artifact	Platform	Validation for this release
dynalm-macos-arm64.tar.gz	macOS on Apple Silicon, CPU and Metal	Beta-verified on one Apple M4 Pro: full test suite, parity with llama.cpp on eight models, DD-053 GPU accuracy, multi-user benchmarks
dynalm-macos-x86_64.tar.gz	macOS on Intel, CPU	CI-tested: built and tested in CI under Rosetta (model-free tests)
dynalm-linux-x86_64.tar.gz, dynalm-linux-arm64.tar.gz	Linux, CPU	CI-tested: built, tested (model-free) and fresh-install checked in CI; not checked with real models
dynalm-windows-x86_64.zip, dynalm-windows-arm64.zip	Windows, CPU	CI-tested: as Linux

• Vulkan and CUDA are experimental. They are built in CI, but Vulkan is tested only on a software GPU (lavapipe) and CUDA only built. The Vulkan archives are not attached to this pre-release.

• CI and Metal: CI's macOS runner has no Metal device, so Metal GPU behaviour was verified on the one physical Mac above.

• Verified models (GGUF):

  • Llama-3.1-8B Instruct Q8_0;
  • Qwen2.5-0.5B and 1.5B Instruct Q4_K_M;
  • Qwen3-0.6B Q8_0, Qwen3-4B and 8B Q4_K_M;
  • Gemma-3-1B Q4_K_M;
  • Granite-3.1-1B-A400M Q4_K_M.

  Other models and quantizations are preview.

Performance and accuracy

All results are from one Apple M4 Pro (24 GB, macOS 26.7, AC power).

Accuracy against llama.cpp (same GGUF files, teacher-forced, 10 × 512 tokens of wikitext-2):

Check	Result
CPU next-token agreement	99.96–100%, identical perplexity
Metal next-token agreement	99.80–100%
Metal greedy text	identical on 4 of 4 prompts per model
DD-053 GPU accuracy	passes on all eight models

This measures agreement with llama.cpp, not answer quality.

Speed against llama.cpp's server (Qwen2.5-1.5B, Qwen3-4B and Qwen3-8B Q4_K_M, Llama-3.1-8B Q8_0; 512-token prompts, 128 output tokens, 2,048 tokens of context per user; medians over 3 rounds):

A difference is "confirmed" only when all 3 rounds agree beyond ±10% and both engines varied by 10% or less. llama.cpp's multi-user results varied by up to about 30% between rounds, so several differences are not confirmed.

Device and load	Result	Confirmed?
Metal, one user	about the same speed	parity on the 8B models; the others inconclusive
Metal, 16 users: total tokens/s	0.71–0.78× (llama.cpp ahead)	yes, 3 of 4 models
Metal, 16 users: 99th-percentile token gaps	7–10× shorter with DynaLM	yes for the 8B models
Metal, 16 users: first token	2.1–3.3× sooner (median)	no, on all four models
Metal, 4 users: 99th-percentile token gaps	longer with DynaLM (0.19–0.30×)	no (consistent direction)
CPU, one user: time per token	0.77–0.97× llama.cpp's (the CPU builds differ: llama.cpp native with Accelerate, DynaLM portable)	yes, 3 of 4 models

DynaLM is not faster than llama.cpp overall.

Known limitations

• Verified on one physical Mac and eight models. Linux and Windows inference with real models has not been validated for this release.
• 16 users on Metal: total throughput is 0.71–0.78× llama.cpp's.
• 4 users: 99th-percentile token gaps are longer than llama.cpp's.
• CPU decode: 0.77–0.97× llama.cpp's speed.
• Memory limits are enforced at load. Llama-3.1-8B Q8_0 with 16 users × 4,096 tokens on a 24 GB Mac is refused (8.0 GiB of KV against 6.9 GiB available); use --ctx 57024 or less, or --kv q8_0.
• Granite MoE on the CPU uses about 0.7 GB more memory for its repacked expert weights.
• Sparse attention keeps a cache-filled Metal server fast for contexts up to 4,096 tokens only; longer contexts still slow down. Vulkan, CUDA and the CPU path don't use it.
• Vulkan and CUDA are experimental. Some models (YaRN RoPE scaling; MoE on CUDA) run on the CPU.

Installation

From this release (macOS on Apple Silicon):

curl -fsSL https://raw.githubusercontent.com/dynalm-project/dynalm/main/scripts/install.sh | sh -s -- --version v0.2.0-beta.1
dynalm --version   # DynaLM   0.2.0-beta.1

The installer without --version installs the latest stable release (0.1.7). Each archive is listed in SHA256SUMS and carries signed build provenance: gh attestation verify <file> --repo dynalm-project/dynalm. To build from source, see docs/build.md.

Changes and compatibility

• --backend native-cpu is no longer available. It stops with an error naming the replacement:

  • --backend cpu for the fast CPU kernels;
  • --mode exact for DynaLM's own DD-053 CPU kernels.

  Configuration files with runtime.device: native-cpu need runtime.device: cpu and runtime.mode: exact.

• DYNACORE_GPU_SPARSE: 0 restores the previous dense-only Metal attention, 1 forces sparse attention.

• dynalm benchmark --prompt-base N: new option.

• No Docker image: this pre-release does not update ghcr.io/dynalm-project/dynalm:cpu.

DynaLM v0.1.7

What's Changed

• v0.1.7: resumable pull on stalled connections, accurate docs by @sajosam in #4

Full Changelog: v0.1.6...v0.1.7

DynaLM v0.1.6

DynaLM 0.1.6: GPU by default

The normal install now includes GPU support, and DynaLM uses the GPU first.

Install (one command)

• Mac / Linux: curl -fsSL https://raw.githubusercontent.com/dynalm-project/dynalm/main/scripts/install.sh | sh
• Windows: irm https://raw.githubusercontent.com/dynalm-project/dynalm/main/scripts/install.ps1 | iex

Your computer	You get
Mac with Apple Silicon	GPU version (Metal)
Linux x86-64 / Windows x64 with graphics drivers	GPU version (Vulkan: AMD, Intel, NVIDIA)
Anything else	CPU version

To choose yourself when installing, add --cpu or --gpu (Windows: -Cpu / -Gpu).

Switch any time

dynalm run qwen2.5:1.5b          # GPU first (default)
dynalm run qwen2.5:1.5b --cpu    # use the CPU
dynalm run qwen2.5:1.5b --gpu    # back to the GPU

Also new

• dynalm --version shows the devices built in, for example devices: cpu metal.
• Quieter start-up output.
• install.sh --from-source turns on every GPU toolkit it finds (Metal, Vulkan, CUDA).
• Release builds are more robust: the toolchain install retries, and publishing no longer fails on build records.

Full list of changes: CHANGELOG.md.

DynaLM v0.1.5

DynaLM 0.1.5: GPU support, automatic CPU/GPU selection, security hardening

Run AI models on your own computer on the CPU or the GPU, fully offline.

Highlights

• GPU backends (preview):

  • Metal for Apple Silicon;
  • Vulkan for AMD, Intel, NVIDIA and Qualcomm GPUs;
  • CUDA for NVIDIA.

  Build with -DENABLE_METAL=ON, -DENABLE_VULKAN=ON or -DENABLE_CUDA=ON. See the
  GPU guide.

• Automatic device choice: --backend auto (the default) uses the GPU when it can run the
  model, and the CPU otherwise. --backend cpu|metal|vulkan|cuda forces a device.

• Less memory on the GPU: in our tests on Apple Silicon, about 3× less memory than
  llama.cpp's server for the same model, and a typical first word 2× faster with 4–8 users.

• Windows on ARM: a new dynalm-windows-arm64.zip. The installer picks it automatically.

• Security:

  • API keys (--api-key-file); a key is required to listen on a network address.
  • A Host check and JSON-only requests block browser-based attacks.
  • Error messages are safer, and model metadata has limits.
  • cpp-httplib is updated to v0.61.0.
  • Fixes found by fuzzing are included.

Upgrade notes

• API key for network binds: dynalm serve --host 0.0.0.0 now needs an API key
  (--api-key-file). Use --allow-unauthenticated only behind a trusted gateway.
• GPU builds choose the GPU: in a GPU build, --backend auto uses the GPU. CPU-only
  builds behave as before.
• 0.1.4 was never published: its changes are part of this release.

Install

• Linux / macOS: curl -fsSL https://raw.githubusercontent.com/dynalm-project/dynalm/main/scripts/install.sh | sh
• Windows (x64 and ARM64): irm https://raw.githubusercontent.com/dynalm-project/dynalm/main/scripts/install.ps1 | iex

Release archives are CPU builds. GPU support is enabled when you build from source.

Full list of changes: CHANGELOG.md.

DynaLM v0.1.3

Full Changelog: v0.1.2...v0.1.3

DynaLM v0.1.2

Full Changelog: v0.1.1...v0.1.2

DynaLM v0.1.1

Full Changelog: v0.1.0...v0.1.1

DynaLM v0.1.0

First cross-platform CPU release of DynaLM, an LLM inference platform built on the DynaCore runtime.

Install

• Linux / macOS: curl -fsSL https://raw.githubusercontent.com/dynalm-project/dynalm/main/scripts/install.sh | sh
• Windows (PowerShell): irm https://raw.githubusercontent.com/dynalm-project/dynalm/main/scripts/install.ps1 | iex
• Then: dynalm doctor, dynalm pull qwen3:4b, dynalm run qwen3:4b

Assets: Linux x86-64 (glibc >= 2.35) and ARM64, macOS Apple Silicon and Intel, Windows x86-64, and SHA256SUMS. Each archive was built, unit- and integration-tested, installed through the installer and smoke-tested (CLI, inference, OpenAI server) on its platform before upload.

Highlights: DynaCore/DynaLM split; model names (qwen3:4b); doctor with a kernel self-test; YAML config; OpenAI-compatible server; optional --execution compiled (DynaCore IR + fusion; +7.9% single-stream decode on Qwen2.5-1.5B); --kv q8_0; the .dyna language and dynacorec. CPU only. Details in CHANGELOG.md.

Full Changelog: https://github.com/dynalm-project/dynalm/commits/v0.1.0