Skip to content

LLMario Beta 0.3.0-beta.3

Pre-release
Pre-release

Choose a tag to compare

@mannysinghx mannysinghx released this 08 Oct 14:18

LLMario Beta 0.3.0-beta.3 makes replies faster on models built for it, shows how fast each model will run on your computer, and adds smaller model builds for 16 GB machines. It installs next to LLMario and does not touch it: it has its own app, settings, models and chat history. Updating from an earlier beta keeps your beta models and settings.

This is a pre-release. For the stable version, use LLMario 0.2.1.

Downloads

File
macOS 13+ (Apple Silicon and Intel) LLMario-Beta-0.3.0-beta.3-macos-universal.dmg Open it and drag LLMario Beta into Applications (replace the old one). Signed and notarized by Apple.
Windows 10/11 x64: app LLMario-Beta-0.3.0-beta.3-windows-x64-setup.exe Installs for your user account; no admin rights needed.
Windows 10/11 x64: command line llmario-beta-0.3.0-beta.3-windows-x64.zip llmario-beta.exe (CLI and local OpenAI-compatible API). Unzip anywhere.

Each file has a .sha256 next to it. To check a download:

  • macOS / Linux: shasum -a 256 -c <file>.sha256
  • Windows (PowerShell): Get-FileHash <file> and compare with the .sha256 file

The Windows files are not code-signed yet. If Windows shows "Windows protected your PC", click More info → Run anyway.

Before you start: install an engine

The app does not include an engine yet:

  • macOS: brew install llama.cpp, and optionally pip3 install mlx-lm on Apple Silicon
  • Windows: winget install ggml.llamacpp, then restart LLMario Beta

The app's Models → Engines tab shows what it found.

What's new in beta.3

  • Faster replies on MTP models. Some models ship extra layers that guess the next token (MTP). LLMario Beta now uses them automatically: the model checks each guess, so answers do not change.
    • Measured on Qwen3.5 9B: 16% faster overall (prose +11%, code edits +19%), with the same answers.
    • The library has 6 new MTP builds, marked MTP: Qwen3.5 0.8B, 2B, 4B and 9B, Qwen3.6 27B and Qwen3.6 35B-A3B.
    • The other ways of guessing ahead stay off by default, because in our tests they slowed some workloads or changed an answer. You can turn them on in the config file (speculative = "ngram" or "draft" under [backends.llamacpp]). In our tests the n-gram mode made gpt-oss-20b 87% faster but Gemma 4 12B 8% slower.
    • To turn the feature off: speculative = "off" under [backends.llamacpp].
  • Speed estimates: "~N tok/s on this computer". Shown in the model library, in llmario-beta model fit (with how much is read per token) and in the app.
    • Estimates come from your chip's memory bandwidth and how much of the model each token reads. On our test Mac they came within 8% of the measured speed for the reference models, and were too cautious for very small models.
    • llmario-beta bench measures the real speed and remembers it. Measured models show their measured speed, and when two installed builds of the same model have been measured, the faster one is used.
  • Smaller builds for 16 GB computers. 8 new 3-bit builds of 7 models (Qwen3.8 27B, Qwen3.6 27B, Gemma 4 26B, Ministral 3 14B, Devstral Small 2 24B, Mistral Small 3.2 24B, Phi-4), labelled with their quality cost.
    • On computers with 16 GB or less, recommendations only include models that fit comfortably and, when their speed is known, run at 10 tok/s or more.
    • The library has a Fits comfortably filter (llmario-beta model catalog --comfortable on the command line).
  • Optional: run a model slightly too big for the GPU. With offload = "auto" under [backends.llamacpp], a mixture-of-experts model that is a little over the GPU limit keeps part of itself in regular memory, at some cost in speed.
    • Example on a simulated 16 GB Mac: gpt-oss-20b planned at 13.3 GiB, peaked at 12.5 GiB and ran at 61–71 tok/s (about 105 tok/s when it fits on the GPU).
    • Off by default until it has been tested on real 16 GB hardware.
  • Models on external drives: the hidden ._ files macOS creates on exFAT drives are no longer recorded as part of a model. Run llmario-beta model refresh once to remove such records from models added earlier; it never deletes files.
  • llmario-beta doctor shows the speculative-decoding mode in effect.

From earlier betas

  • beta.2: a fix for removing downloaded models when the models folder is a link, the "fits, tight" label on 16 GB computers, and optional memory_profile = "auto" and kv_cache_type = "q8_0" settings.
  • beta.1: a memory check that counts only the memory that grows with the conversation, so more models fit (Gemma 4 12B plans at 10.2 GiB on a 16 GB Mac, refused at 14.6 GiB before). Also less memory in long chats with llama.cpp, and correct memory readings on macOS. The beta runs alongside LLMario with its own app, command (llmario-beta), data folder (~/.llmario-beta) and port (11501).

If a model fails to load because memory ran out, add kv_accounting = "conservative" under [runtime] in the config file to go back to the stricter memory check, and please report it.

Status

  • Tested on: an Apple Silicon Mac (M4 Max, 64 GB). Windows is built and tested automatically (install, launch, a real chat through llama.cpp).
  • Not yet tested on a 16 GB Mac. The 16 GB figures above come from the memory check, or from runs on the 64 GB Mac with its GPU limit set to a 16 GB Mac's.
  • Speed estimates are calibrated on that Mac. On other chips the first bench replaces the estimate with a measurement. NVIDIA GPUs have no estimate until measured.
  • MTP and the GPU/CPU split on Windows and NVIDIA are untested.
  • Gemma 4 12B on MLX doesn't run yet: MLX-LM 0.31.3 does not support it. Use the GGUF build.
  • The Intel part of the universal Mac build is untested on real Intel hardware.

Website: https://llmario.com · Source: https://github.com/mannysinghx/LLMario