You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
mlx-serve is a supported backend (ADR-0031), adopted on the task bench and
a server measurement with the same Qwen 3.6 weights under mlx-serve and LM
Studio: the speed comes from the MTP head of mlx-serve's own model builds, not
the runtime; a 122k-token prompt completes. The ADR records the residual risk
of running it — chiefly that 26.10.1 listens on every interface with no key
by default — and that its default 2 GB prefix cache re-reads long sessions
every turn: with an 8 GB cap a 122k resend took 1.1 s instead of 207 s.
bench serve measures a model server: cold prompt reading per size,
reuse on a resend and the next turn, decode speed, alternating conversations
and two streams at once, logged raw; bench serve-report recounts a run.
[llm].provider = "mlxserve" for mlx-serve. The context window is
read from its /v1/models at startup, so leave [model].context_window at
0 for it — a set window skips the lookup. A [llm].model that list does not
carry is reported at startup: mlx-serve answers a chat request for any model
name with the model it has loaded, so a mistyped id otherwise runs silently
against whatever is resident.
Fixed
A completion of only blank lines is asked again, as an empty one is
(ADR-0007, amended by ADR-0031). Under LM Studio, Qwen 3.6 starts its answer
with the blank lines after its thinking; one bench run's entire final answer
was \n\n, which was returned as the answer.