rMLX 0.2.7
Constrained-decode hot-path and Gemma4-unified vision tuning. The json_schema
and json_object per-token allow-mask probes no longer deep-clone their grammar
across the ~152K-token vocab on every decode step, and a whitespace stall in
schema-constrained decode is fixed. Gemma4-unified gains a per-request image-token
budget. No breaking changes.
Added
- Per-request + CLI image-token budget for Gemma4-unified vision. A
image_max_tokensrequest field (and matching CLI flag) caps soft image
tokens per request; default 280, ceiling 1120. Lets callers trade vision
fidelity for prefill cost on the unified any-to-any path. (#181, closes #180)
Fixed
- Schema-constrained decode whitespace loop. Under
response_format: json_schema, enum / scalar leaves accepted insignificant whitespace in
states where it must be rejected (inside a literal, inside a string, at the
root scalar start), letting temp=0 decode loop on\n. The allow-mask now
matches whitespace per-leaf-state and rejects raw control chars (0x00..=0x1f)
inside strings. (#183)
Performance
json_schemaconstrained decode no longer deep-clones the schema per
vocab token. The allow-mask probe reset a scratchSchemaGrammar~152K
times per decode step, deep-copying the immutable parsed schema each reset.
The schema is now held behindArc(Object.props,Union,Array.items),
so entering a container/property/union branch is a refcount bump and the
per-token reset reuses buffers in place. Per-step cost on the production path
drops ~8–25× (was 20–40× heavier than thejson_objectengine; now
comparable). Tool / function-calling agents pay this directly. (#184,
closes #182)json_objectconstrained decode allow-mask reset is scratch-reused. The
JsonGrammarreset became a state copy +Vecclear/extend (the stack frame
isCopy) instead of a fresh clone per vocab token — ~2× on the per-step
probe. The two engines now share onefill_allow_maskkernel over a
ProbeGrammartrait. (#183)