Skip to content

v0.18.0

Choose a tag to compare

@magifd2 magifd2 released this 07 Oct 15:23
· 9 commits to main since this release

Added

  • mlx-serve is a supported backend (ADR-0031), adopted on the task bench and
    a server measurement with the same Qwen 3.6 weights under mlx-serve and LM
    Studio: the speed comes from the MTP head of mlx-serve's own model builds, not
    the runtime; a 122k-token prompt completes. The ADR records the residual risk
    of running it — chiefly that 26.10.1 listens on every interface with no key
    by default — and that its default 2 GB prefix cache re-reads long sessions
    every turn: with an 8 GB cap a 122k resend took 1.1 s instead of 207 s.
  • bench serve measures a model server: cold prompt reading per size,
    reuse on a resend and the next turn, decode speed, alternating conversations
    and two streams at once, logged raw; bench serve-report recounts a run.
  • [llm].provider = "mlxserve" for
    mlx-serve. The context window is
    read from its /v1/models at startup, so leave [model].context_window at
    0 for it — a set window skips the lookup. A [llm].model that list does not
    carry is reported at startup: mlx-serve answers a chat request for any model
    name with the model it has loaded, so a mistyped id otherwise runs silently
    against whatever is resident.

Fixed

  • A completion of only blank lines is asked again, as an empty one is
    (ADR-0007, amended by ADR-0031). Under LM Studio, Qwen 3.6 starts its answer
    with the blank lines after its thinking; one bench run's entire final answer
    was \n\n, which was returned as the answer.