Skip to content

NVMAI 4.0

Choose a tag to compare

View all tags
@Pummelchen Pummelchen released this 21 Aug 01:22
· 978 commits to main since this release
v4.0
9a74852

NVMAI 4.0 — Ornith and extended context

NVMAI 4.0 makes text-only Ornith 1.5 35B-A3B 8-bit the default while
preserving bounded RAM use and SSD-streamed routed experts on Apple Silicon.
Qwen 3.6 and Ornith 4-bit remain supported explicit choices.

Default profile

  • Ornith 1.5 35B-A3B 8-bit weights
  • Native 262,144-token context
  • Multi-prefix prompt cache on, 8-bit KV, and 8 GiB expert-cache budget
  • MTP off, Concise off, Fast alias off, and Thinking off
  • Temperature 0.6, Top-P 0.95, Top-K 20, and presence penalty 0.0

Highlights

  • Added reproducible Ornith 4-bit and 8-bit installers on the shared
    Qwen3.5-MoE inference and bounded expert-streaming path.
  • Added selectable 16-bit, 8-bit, and 4-bit KV storage; 8-bit is the default.
  • Added optional YaRN scaling for 512K and 1M contexts, with native RoPE still
    the default.
  • Added Ornith's official one-layer MTP draft as an experimental bounded
    sidecar. It remains off because current M3 measurements show no speed gain.
  • Added the models' official binary Thinking on/off control throughout the
    CLI, server, Mac app, launchers, and benchmark profiles.
  • Added coding-CLI and four-program Swift/Python/TensorFlow/PyTorch benchmark
    coverage. In the eight-cell tooling matrix, 8-bit passed all four mode cells;
    8-bit with Concise off and Thinking off was the fastest correct cell.

Validation

  • tools/lint.sh: passed
  • python3 -m unittest discover -s benchmark -p 'test_*.py': 18 passed
  • swift test --no-parallel: 755 tests in 132 suites passed
  • Ornith 8-bit greedy golden inference: byte-identical repeat check passed on
    Mac15,3 (M3, 24 GB), macOS 26.6.2, Swift 6.3.3

Binaries

The archive contains Apple Silicon/arm64 binaries for NVMAIServer, NVMAICLI,
NVMAIMac, NVMAIDecodeService, NVMAIRepack, and NVMAIBench plus their Metal
resources. It requires macOS 26 or newer. Model weights are not included.

The binaries are unsigned and not notarized. Verify the archive before
clearing its quarantine attribute.

nvmai-4.0-macos-arm64.tar.gz SHA-256: 8851981b5d54ad4862c16724626ff40436f8f5dfaeb332987717dbedd500fa81

Assets 4
Loading