Repository navigation
mlxtop learns what fast means for the machine it watches. It reads what the
model reads per forward pass and what the chip can move, and estimates how
close decode runs to that limit; it judges usual speed per verify pass and at
the same context depth, keeps it across sessions, says how much more context
fits, and names what first tokens lose to cache misses, model loads and
overlapping requests. It also carries every fix from the review of 3.3.0
(3.3.1 was never released; its fixes ship here). Update with the install
script or Homebrew: 3.3.0's updater runs on the TLS crates this release
replaces; if you use it anyway, check what it installed as shown at the end
of these notes.
Is this speed normal?
- How close decode runs to the chip. Diagnostics (
d) estimates the
share of the chip's peak bandwidth decode reads, counted per verify pass,
and with a draft its payoff;--onceaddsDECODE/DRAFTlines, the
session summary gives it by context, and a healthy verdict notes when decode
is near the best this Mac has reached. Nothing is shown when an input is
unknown or it would exceed 100%. - The chip's memory bandwidth. The runtime card's bottom edge names the
peak its maker publishes beside the chip (410 GB/son a 32-core M4 Max,
not the 40-core part's 546), and--onceadds it to the METAL line. A chip
or bin mlxtop cannot tell apart shows nothing, and the figure is the first
field to go when the edge is short. - What the model costs, in Diagnostics.
dandmlxtop doctorshow the
weights one forward pass reads (decoder layers and output head, not a
vision tower, MTP head or embedding lookup), bits per weight and
quantization, KV bytes per token and its format, the state each sequence
holds, the draft, the context allowed and trained, and the runtime's
version. For a local MLX model they come from itsconfig.jsonand
safetensors headers, never its weights, read once per model; llama.cpp,
Ollama, LM Studio, vLLM and SGLang give theirs through their APIs. - Usual speed compared like with like. With a draft whose runtime counts
tokens per verify pass (Splash), speed is judged per pass, so text the draft
guesses less well, such as thinking on, is not a slowdown. With the model's
weight and KV bytes known, earlier speeds are scaled to the current depth,
so a KV-heavy model twice as deep is not called slow. - Usual speeds survive restarts and upgrades. mlxtop keeps each model's
usual generation speed by context, numbers only, in a private file, so a
new session judges its first request against earlier sessions. A model that
now runs slower says so once: "This model now generates 31% below its usual
from earlier sessions at this depth, since oMLX 0.7.3 became 0.8.0". Turn it
off in Settings;rnow asks first, andfthere forgets what was kept.
What limits it
- Heat is named on Apple Silicon. mlxtop reads macOS's thermal pressure
level (notifyutil) every 5 s;pmsetprinted no CPU limit on Apple
Silicon, so Thermal never fired there. Fromheavyup the card reads
THERMAL heavyand the verdictThermal limit; a slowdown while the level
rises (frommoderateup) is put down tothermal pressure …. A level that
cannot be read showsthermal —, never nominal; Intel Macs keeppmset's
CPU speed limit. - How much more context fits. The memory panel shows the tokens each
request in flight can still add (≈37k tokens at 2 requests), from the
runtime's own KV room or plan where it reports one, the configured window
for llama.cpp-style runtimes, or memory left for MLX. Diagnostics shows the
arithmetic. It is an estimate, never coloured, and the first row to drop. - Wired-limit advice comes from 90% of
iogpu.wired_limit_mbor of
Metal's recommended working set, where MLX warns generation can slow. - The session summary keeps each model load's peak memory, with the
context it held then, after the model is unloaded. - What first tokens lose. The Journal names a long prompt read in full
right after a cached one of similar length (CACHE MISS), a model load
(MODEL LOAD), requests recomputed for a full KV cache (KV CACHE FULL)
and, once a session, two requests at once that bought no throughput
(OVERLAP, noting oMLX'smax_concurrent_requests). The summary adds the
time first tokens lost and Diagnostics the evidence. Nothing is coloured: on
a month of one Mac Studio's logs the cache rule fired 5 times in 10,653
requests. - A full KV cache is named. A slowdown while the runtime recomputes
preempted requests readsKV cache full · 2 requests recomputed, ahead of
the prefill it causes. - NVIDIA and AMD limits. The GPU table adds power against its cap
(183/450 W), clock against its maximum, memory activity and the PCIe link
as width allows. Throttling, slow links and who holds a full card's memory
are told in Diagnostics, and a card held for heat is named when generation
slows. A field an older driver lacks no longer costs the core readings. - Strix Halo and other AMD APUs show their memory as the one pool it is,
RAM held against its GTT limit, never a near-empty VRAM meter. AMD cards add
memory activity, power, clocks and GTT, without waking a sleeping card. - Energy per token on Linux and Windows uses the GPUs' own power,
labelled as such.
Runtimes
- Splash's draft, counted per verify pass. mlxtop reads tokens per verify
pass from Splash's counters, matching Splash's own log within its rounding,
shows them beside a lone request's output, and reads the KV room in
Splash's memory plan. Splash 1.6 also says what paused decoding and held
each first token. Under oMLX, Splash's status is private, and Diagnostics
says so. - The completion log no longer passes off a whole request's rate as
generation speed. - llama.cpp no longer wakes a sleeping server. mlxtop reads
/metrics
first and/slotsonly while a request runs; a server started with
--sleep-idle-secondscan sleep and reads as idle, withAsleepin
Diagnostics. Each answering request gets its own row in prompt load, the KV
cache panel shows its hit rate, and Diagnostics adds slots and speculative
decoding. - Prompt speed and model load for finished requests. The usage recorder
reads llama-servertimingsfrom any runtime, Ollama's prompt and load
durations, LM Studio's model load time, and oMLX's prompt speed and model
load. Prompt load showsMODEL LOAD
andPREFILLbeside a reported request's speed. - SGLang and TensorFold speeds are live. Decode rates come from counters
that move every step, so slowdowns are judged while they generate. vLLM,
SGLang and TensorFold add KV room left, preemptions, draft tokens per verify
pass and per-interval queue waits to Diagnostics. Ollama (one read-only
/api/showper model) and LM Studio add quantization, parameters, experts
and context to the model facts. No adapter asks a runtime for a route that
changes state or generates a token. - No request to oMLX's port until an oMLX is found. With no oMLX process
detected and none chosen, mlxtop no longer probes127.0.0.1:8000/health;
SGLang answers/healthby generating a token, so an unseen SGLang there
was made to generate one every few seconds. An oMLX behind a tunnel needs
MLXTOP_PROVIDER=omlxor anomxport.
Output
-
JSON for scripts and agents.
mlxtop --once --jsonprints the static
report as one line of JSON, andmlxtop streamwrites one per sample,
ending quietly when its reader goes away or on a signal. Units are in every
field name, unknown readings are null, each figure is tagged host, runtime
or estimate, and no process names leave the machine. Schema 1, experimental
through 3.4. -
mlxtop serveanswers/metrics(Prometheus),/snapshotand
/healthon 127.0.0.1:10063 from the newest sample, loopback only (reach it
from another machine withssh -L). The snapshot and metrics carry the
chip's published bandwidth, the model's facts, decode efficiency, context
left, the runtime's KV room, preemptions and tokens per verify pass, each
tagged or described as an estimate where it is one. -
For scripts and Prometheus:
streamandserveadd the session's time
lost before first tokens; every snapshot adds the finding against earlier
sessions (how far below, and the update), and/metricscarries both.
Evidence
mlxtop replaytallies each reason: its slow samples, the episodes the
Journal began with it and episodes per hour of generation. It also lists
cache misses, model loads, full KV caches and overlap with the slowdowns,
tallied with their rows per hour and the time each cost.- Each sample keeps more evidence. Log lines end with
pressure=,
thermal=,power_mw=,draft=andts=(naif not read), and recorded
samples carryts; older recordings replay as before.
Fixes
- A Mac's memory is one pool wherever it is shown. The memory panel chose
its layout from the build target, so four tests failed on Linux in the
3.3.1 test builds (and would in CI there), and a Mac's recording replayed on Linux
drew the wrong panel. An Apple GPU reading now decides. - A local port nobody listens on is asked again within seconds on
Windows too. Windows retries a refused connect for about two seconds, so
the oMLX client read it as a hung server and waited up to 30 s. A connect
that times out on a loopback address is now a refusal; a hung server still
waits up to 30 s.
Fixes from the review of 3.3.0
The rest of this release's notes are the fixes made after a full review of
3.3.0, tried as 3.3.1 test builds and never released on their own.
Security and privacy
- TLS crates refreshed. rustls 0.23.46, rustls-webpki 0.103.17 and
webpki-roots 1.0.12 clear RUSTSEC-2026-0346 to -0349, -0381 and -0382;
the last two let a distrusted certificate authority vouch for a server,
which mattered tomlxtop update. CI now runs every cargo-deny check,
weekly too. - Recordings hold no command lines and are created readable by you
only. A runtime's command line can carry a key or a home path, and a
recording is meant to go with a bug report. Recordings made by 3.3.0 do
contain them: check one before sharing it. - No control characters reach the terminal from
--once,doctor,
replayor the exit summary, whatever a runtime or a recording holds. - The oMLX key goes only to a found oMLX. It is sent when an oMLX
process was detected or oMLX was chosen, never to another program
answering on port 8000. The updater accepts release assets only from
https://github.com/; the log is created owner-only.
The verdict
- A measured slowdown leads the headline. A 67% drop read
Compression activewhile the chart and Journal showed the slowdown; any host state
other than a critical one now gives way toGeneration slowedor
Generation stalled. - The verdict holds between samples at every interval. Freshness is
decided when a sample is taken, so a slowdown no longer turnsHealthy5 s
after a sample at long intervals or while paused. Slowdowns are found at
30 s and 60 s intervals, the session summary counts work above 10 s, and
counter-based runtimes keep their rates at 5 s and more. - One bell and one story per incident. A critical episode ends after
three calm samples (the bell rang 8 times in 75 s near a threshold), a
state already told within 10 samples is not told again, and a stall that
comes back within a minute is the same stall.Paging activebegins at
the documented 4 MiB/s. - A prompt being read is not a slowdown on runtimes that count tokens
but list no requests, and a slowdown whose request ends is closed with an
ENDEDrow. - Stopped sampling is shown.
NO SAMPLE 12sorSTOPPEDreplaces the
heartbeat, and the verdict readsSampling stopped; a panic message no
longer prints over the dashboard.
Readings
- A failed reading is unknown, not zero. One failed
psmade the
runtime vanish and emptied its speed chart; one failedvm_statraised
SWAP THRASHINGat 76 GiB/s. A comma-decimal locale (German, French,
...) hid every process and zeroed swap; one non-UTF-8 argument anywhere
discarded the process list. - Process names are real on macOS (
Google Chrome, notGo), and only
serving processes are runtimes:tail -f ollama-notes.txtis not Ollama.
Runtimes started throughpythonornodeare still found;
MLXTOP_PROVIDERforces a choice. - A stock oMLX shows its speeds. The key is read from
MLXTOP_PROVIDER_API_KEY,OMLX_API_KEY,server.envorauth.api_key
in oMLX's ownsettings.json; with none, the Overview says so. - A runtime that hangs costs one short wait. Every request of one poll
shares 1.5 s, so memory and paging keep being sampled; quit does not wait
for it. A runtime started after mlxtop shows within seconds, not 30.
localhostand kept-open connections work for oMLX. - No runtime reads
No LLM runtime detected, and one that stopped
answering readsUNREACHABLEwith its age. Every runtime's connection
changes are logged. - llama.cpp averages come from its counters, not gauges each scrape
resets; vLLM prefill leaves out cached tokens and is not called live;
mlx-serve restarts are seen; a server answering 401 or 503 on its second
port is found there. - GPU memory is compared with a limit only when one is known, and the
Linux memory panel no longer counts the page cache twice: cache and free
add up toMemAvailable.
Interface
- Signals restore the terminal.
kill,pkill,timeoutor a closed
session no longer leave the shell in raw mode; Ctrl-C quits from Help. - Small terminals show the verdict above the size note, and keys other
than quit wait until the terminal is larger. - The Journal reaches its oldest event when rows wrap; Down past the end
no longer stores an index Up has to unwind. - Replay lists every finding (it stopped after about 120 requests),
reports lines it could not read, and refuses files that are not
recordings. - Messages name what is wrong:
mlxtop dcotorsaysunknown command,
aconfig.jsonthat does not parse is reported once, and Settings keeps a
symlinked config linked and private. - The memory panel is laid out as the hardware is. Where the GPU has
no memory of its own (Apple Silicon, and NVIDIA's integrated GPUs such as
DGX Spark and Jetson) there is one pool and one bar: what the GPU holds
is the first part of RAM's wired memory, split into the model and the KV
cache and everything else where the runtime gives the model's size. Under
it are how near the GPU is to its limit, what is left (free memory and
cache, or the room under the GPU limit where that is less), each part's
size, the runtime's footprint and the largest other app, and the same
pool over time in dimmed colours. Where cards have their own memory,
VRAM and RAM each get a meter and a chart. The trace of what was left and
the sentence repeating the GPU meter are gone. - Each prompt shows the speed it generated at. Under every bar of the
prompt-load chart, below its size:24.5/s. The Overview's panel shows
it without being expanded; the count of generated tokens moves one row
down, on panels tall enough for both. - Processes show each process's footprint on macOS, beside its RSS and
in the title's total: a runtime's resident size leaves its GPU memory
out (7.0 GiB of RSS for a process holding 23.4 GiB). - Nothing is cut mid-word at 80 columns. The KV panel shortens its name
and keeps its reading (it readKV cache / model I/O %), the paging
footer and the empty-window message use a shorter wording that fits, and
any other bottom-edge summary drops whole fields. - Settings holds the inference engine's API key. It is typed and shown
as dots, saved to aconfig.jsonreadable by you only, and used at once.
MLXTOP_PROVIDER_API_KEYstill overrides it. The first time in a session
that the engine asks for a key mlxtop does not have, or refuses the one in
the file, mlxtop opens Settings on the key, ready to type; Esc carries on
without it. The Overview names a missing or refused key (No live rates: llama.cpp refused the API key · 4 for Settings). - One figure per fact. The header's
LLMand the memory panel'sby appboth give the runtime's footprint (the header gave its resident
size: 7.2 GiB beside 23.4 GiB). The Journal's first row,--onceand the
critical-pressure line give RAM in use and what is left, as the panel
does. The KV bar'sRAMandSSDshares add up to the title's hit rate
as printed (92% and 2% underKV hit 90%). An expanded memory chart's
title no longer gives a share "resident" over a panel that gives RAM by
kind (avg 98%over31.7 / 36.0 GiB). - Zero and a trickle stand on one baseline. A zero rate drew a dash
through the middle of its row and a tiny rate a sliver at the bottom, so
compression that came and went looked like a dashed line. Byte-rate axes
top out at round figures:1.5 MiB/s, not1.4 MiB/s. - Idle is said once. The empty generation and prefill panels read
No rate samples yet, the GPU panel's edge says nothing under0%, and an
empty panel carries notok/s. The SSD cache row keeps words when it is
short of room (1.5k loaded · 1.3k saved), not↑1.5k ↓1.3k ✕327. - A short compression or paging panel keeps a trickle small. At two
rows tall the panel scaled to its own peak, so 16 KiB/s of compression
drew full bars; it now scales against the warning rate, as the taller
chart does. - Smaller fixes:
CPU 0.0%, not-0.0%;1 request waiting; the paging
panel saysmemory is shortonly from the warning rate; the
discrete-GPU layout drops% residentandNo paging traffic.
Delivery
- Each release is installed by a machine with both install scripts and
updated from the previous one, on macOS, Linux and Windows. Jobs have
timeouts, the toolchain action is pinned, a final release with no
changelog section fails, and a CI job checks Rust 1.88.
Updating from 3.3.0 with mlxtop update
mlxtop update in 3.3.0 still works, but it downloads over the TLS crates
this release replaces. The install script or Homebrew avoid them. If you use
mlxtop update anyway, confirm afterwards with your system's own tools that
the installed binary is the published one (macOS on Apple Silicon, Linux):
mlxtop update
v=3.4.0
case "$(uname -s)-$(uname -m)" in
Darwin-arm64) t=aarch64-apple-darwin ;;
Linux-x86_64) t=x86_64-unknown-linux-musl ;;
Linux-aarch64) t=aarch64-unknown-linux-musl ;;
esac
a="mlxtop-$v-$t.tar.gz"; u="https://github.com/maximpri/mlxtop/releases/download/v$v"
d="$(mktemp -d)" && cd "$d" &&
curl -fsSLO "$u/$a" &&
curl -fsSL "$u/SHA256SUMS" | grep " $a\$" | shasum -a 256 -c - &&
tar -xzf "$a" &&
cmp "mlxtop-$v-$t/mlxtop" "$(command -v mlxtop)" &&
echo "OK: the installed mlxtop is the published $v"It prints OK only when the binary mlxtop update installed is byte for byte
the one in the release; otherwise reinstall with the install script. From 3.4.0
on, mlxtop update is safe again.