Repository navigation
Edge0 145B Q3_K on Apple M5 Max (MacBook Pro, 128 GB): speed results on Metal #219
lxove
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Thanks for the macOS / Metal support in #214. I ran Edge0 145B Q3_K on an Apple M5 Max with the same
strata_bench.pysuite used there, so the numbers are comparable with the Apple M2 Ultra row in the README. This is a speed check only. No quality or correctness evaluation was run.Results
results.json
Medians, with range over runs in parentheses, tok/s. Speeds are the engine's own timings.
Time to first token: 3.26 s at 4K (4,019 prompt tokens), 12.78 s at 16K (16,058 prompt tokens), 0.40 s for the short chat questions. Agent session: 65.7 % of prompt tokens were reused from the conversation cache over the follow-up turns, and the median time to first token per turn was 2.74 s.
Runs: 3 per prompt test, 9 chat requests (3 questions × 3), 15 agent follow-up turns (5 turns × 3 sessions).
Hardware
pmset -g therm.Software
mainat7b60076(includes Feat/edge0 strata mac #214), built from source:0.1.55-metal), built as inedge0-strata/EDGE0.md→ "Build (macOS, Apple Silicon)". Pinned ggml3cf0325, metal-cpp 26 (SHA-256 checked).moe_mm_parity1280 and 1792 both PASS.cmake --preset mac-metalEDGE0.mdModel
Edge0/Edge0-145B-q3k, revision2d80116, used as downloadedmtp-q2_0.ggufgenerated by the template's prepare step on the first start (tools/mtp_gguf.py)Settings
edge0-strata/edge0/models/edge0-145b-metal.json, unchanged. No--strata-argoverrides.du=q4,ple-io=direct(the macOS defaults), context 32,768, MTP speculative decoding on as in the templateWorkload
edge0-infer serveproxy, the same way as in Feat/edge0 strata mac #214Notes and limitations
strata bench run), which is written for the Qwen3.8 models.--long), concurrency, and the 120B model.Full strata_bench.py report (README.md)
Community benchmark on Apple M5 Max
Measured on 2026-11-14 with
tools/strata_bench.py(suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run.Hardware and software
Model and configuration
Method
Run by
python tools/strata_bench.pyagainst the HTTP API of a Strata server (already running); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests.tools/ab_engine.py, up to 100 new tokens, 3 runscached_tokens) and timings (prompt_n)Results
Ranges are the lowest and highest of the runs;
Runscounts requests that returned a token.Agent session (conversation cache)
Cache hit over the follow-up turns: 65.7 % (65.5-65.7); earlier-turn tokens read again per session: 4 (3-73) (earlier turns that the cache did not hold; the 13515 (13515-13516) new tokens of the files are read once); first turn (nothing cached) first token: 0.85 s (0.80-0.85).
Per-request data:
results.json. Engine log lines from the start:engine-log-excerpt.txt(when this run started the server).Correctness and limitations
Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server,
nvidia-smi/rocm-smiand the OS; background load is not recorded. Total time of the suite: 143 s.All reactions