Repository navigation
- picks up mlx-swift 0.33.0 (upToNextMinor)
New Models
MLXLLM
- Qwen 3.8 (
qwen3_8,qwen3_8_moe,qwen3_8_text), including MTP drafters (#545) - PrismML Bonsai 2 27B, ternary Hadamard-rotated Qwen 3.8 (
prism_hadamard_qwen35) (#630) - LiquidAI LFM3 (
lfm3) (#750, #756) - LiquidAI LFM2.5 dense and MoE checkpoints (
lfm2,lfm2_moe) and the LFM2 DSpark drafter (#528)- the DSpark drafter must be registered with
await LFM2DSparkRegistration.register()
- the DSpark drafter must be registered with
- JetBrains Mellum 2.1 (
mellum) (#697) - Nemotron 3.5 Lightning, Bolt (
nemotron_h_puzzle), and Super; dense Nemotron-H checkpoints (#712, #635)
MLXVLM
- Qwen 3.8 VLM (
qwen3_8,qwen3_8_moe) (#545) - Bonsai 2 VLM (
prism_hadamard_qwen35) (#630) - DiffusionGemma, a block-diffusion model (
diffusion_gemma) (#352)
MLXEmbedders
- EmbeddingGemma 2 (
embedding_gemma2,embedding_gemma2_text) (#684)
Tool Calling
- MiniCPM5 tool call format:
ToolCallFormat.miniCPM5(#637)
API Change
Source Breaking
MLXLMCommon.compile(shapeless:_:)overloads are removed (#720)- mlx-swift 0.33.0 now provides the same
@Sendableoverloads, so unqualifiedcompile(...)calls still build - only code that spells out
MLXLMCommon.compilebreaks
- mlx-swift 0.33.0 now provides the same
MTPDrafterModel.validateCompatibility(with:) throwsis a new protocol requirement with no default (#581)- custom MTP drafters must implement it
IntegrationTestModelschanged from anactorto astructandevictLLM(_:)was removed (#760)- loads go through
TestModelManager.shared, which keeps one model in memory at a time
- loads go through
Potentially Source Breaking
BaseLanguageModel.sanitize(weights:)andsanitize(weights:metadata:)nowthrows(#679)- conformers that do not throw still compile
- callers that go through the protocol need
try sanitizeonGemma3nText,Qwen35,Qwen35MoE, and the Qwen 3.5 MTP drafters (MLXLLM and MLXVLM) now throws, so direct callers needtryand subclass overrides needthrows
- New cases on public enums. Exhaustive
switchstatements need new arms:Generation.reasoning(String)(#563) andGeneration.steering(SteeringEvent)(#608)GenerateStopReason.steered(#608) and.interrupted(#798)ToolCallFormat.miniCPM5(#637)ModelAdapterError.unsupportedLayer(key:type:)(#803)KVCacheConfigurationError.invalidRewindCapacity,.incompatibleRewindCapacity(#622)ModelConfiguration.DirectoryError.invalidModelSubdirectory(#528)
- New
KVCache/BaseKVCachemembers, each with a default:currentKeyValues(),maxTrimCount,evictedTokenCount,serializedClassName,restore(state:metaState:)(#352, #622, #640, #701)- a
BaseKVCachesubclass that already declares one of these names must addoverride
- a
LoRALinear/DoRALinear/QLoRALinear/QDoRALinearnow trap unless the base layer is exactlyLinearorQuantizedLinear(#803)LoRAContainer.from(model:configuration:)throws.unsupportedLayerand leaves the model unchanged
Gemma4VisionConfigurationmoved from MLXVLM to MLXLMCommon. MLXVLM keeps a typealias (#684)ChatSession.respond(...)andsynchronize()are nownonisolated(nonsending)(#608)MLXLanguageModel.init(... load: ContainerLoader)is deprecated in favor of an overload that takesLoadProgressHandlers(#551, #745)- the
progressHandler:load overloads are deprecated in favor ofprogress: LoadProgressHandlers(#551)
Behavior Change
- Reasoning streams separately from the answer (#563)
- models with a
reasoningConfig(<think>families, DeepSeek-R1, Gemma 4, LFM2.5, Harmony, Onyx) now emit thinking as.reasoning, without the delimiters - string APIs (
respond,streamResponse) no longer include thinking text ChatSessionkeeps the thought and replays it asreasoning_content(#663)
- models with a
- Qwen 3.5 and Muse Glimmer honor a hard thinking budget (#734)
- Generation masks
suppress_tokensfromgeneration_config.jsonand multimodal placeholder tokens (#412) - Sampling defaults from
generation_config.jsonare exposed viaModelContainer.defaultGenerateParameters(#528) - Text-only prompts on several VLMs no longer carry an attention mask, so
ChatSessionreuses the KV cache across turns (#700) - Image prompts no longer promote Qwen2-VL, Qwen2.5-VL, FastVLM, and Mistral3 to float32 (#766)
- The DeepSeek-V3 MoE gate now matches mlx-lm, so it may route to different experts (#795)
- EmbeddingGemma uses bidirectional attention when the config asks for it. Embedding values change to match the reference (#654)
- Loading
- the loader skips checkpoint components that a model does not have, e.g. vision weights for a text model (#711)
- vision encoders for Qwen 3.5/3.8, LFM2-VL, and Muse Glimmer load on first media use (#580)
- MLX errors on loader threads are now thrown (#698)
- factories call
model.validate(tokenizer:), so a mismatched tokenizer fails at load time (#756) - weight loading reports progress per byte, in its own
weightsphase (#551)
- KV cache
savePromptCachethrows for a cache type with noserializedClassName. Before, it was silently saved as"KVCache"(#701)RotatingKVCacherewinds exact prefixes (#622), keeps the pinned prefix visible (#747), and sizes masks from the retained window (#706, #728)- prefill over a
QuantizedKVCacheuses the fused SDPA kernel (#703) - the MLX buffer cache is cleared after each prefill chunk (#710)
- Streaming detokenizers flush held text when generation ends (#670)
- Guided generation feeds the sampled token before the fast-forward tokens (#731) and puts the response schema in the prompt (#715)
- Logging goes through mlx-swift's
MLXLogger(#702, #790) LoRATrain.trainthrowsLoRATrainError.sampleTooShortinstead of crashing (#683)
New API
MLXLMCommon
ChatSession- mid-turn steering:
steer(_:policy:response:),canSteer(_:),ResponseID,latestResponse,SteeringEvent,SteeringPolicy,SteeringError(#608) interrupt(response:)ends a response at the next safe point and keeps the output (#798)fork(history:)copies the cache into a new session (#641)observe(to:using:)streams events through aGenerationTransformer, e.g..logProbabilities(topCount:)(#714)- resumable MTP speculative decoding:
SpeculativeDecodingConfig(mtpDrafter:blockSize:),SpeculativeDecodingFallbackReason(#581)
- mid-turn steering:
- Generation observability (#640)
- public
TokenLoopHandler,TextToolTokenLoopHandler,RawTokenLoopHandler,generateLoopTask(...) - token log probabilities:
GenerateParameters.logProbabilities,LogProbabilityTokenLoopHandler, also during speculative decoding (#774) GenerateCompletionInfo.evictedTokenCount,.reasoningTokenCount,.answerTokenCountPrefillParameters.prefixreports the reused cache prefix before prefill (#644)
- public
- Reasoning:
ReasoningChannel,ReasoningConfig.gemma4,ReasoningConfig.declaringBudgetTransition(_:),QwenReasoningProtocol.qwen35,OnyxReasoningProtocol(#563, #734) - Block diffusion:
BlockDiffusionLanguageModel,BlockDiffusionTokenIterator,BlockDiffusionParameters,GenerateParameters.diffusion,LMModelCapabilities(#352)generate(...)dispatches to the diffusion iterator automatically
- Model loading
BaseLanguageModel.prepareCheckpoint(_:),excludedCheckpointNamespaces,validate(tokenizer:)(#679, #580, #756)ModelCheckpoint,CheckpointComponent,CheckpointNameMapping(#679, #711)- on-demand components:
ModelComponent,ModelComponentsProviding,OnDemandComponent,ComponentLoading,ModelConfiguration.excludedComponents(#580) LoadProgressHandlers(download,weights) (#551)ModelConfiguration.modelSubdirectoryloads a model from a repository subfolder (#528)ModelTypeRegistry.contains(_:configuration:)for sharedmodel_types (#778)
- Memory reporting
- KV cache
LanguageModel.nextTokenLogits(_:cache:state:)computes only the last-position logits during prefill (#614)- Suppressed tokens:
SuppressedTokensProviding,SuppressTokensProcessor,GenerationConfigFile.suppressTokens(#412) Tokenizer.makeStreamingDetokenizer()(#619) andStreamingDetokenizer.finish()(#670)LogitProcessor.didEmit(token:)andfinalizeGeneration()(#640)- Hadamard-rotated quantization:
HadamardQuantizedLinear,HadamardQuantizedEmbedding, and supporting types (#630) - MTP:
ResumableMTPDrafterModel,MTPDrafterCompatibilityError,MTPDrafterModel.targetLayerIds(#581, #528)
MLXGuidedGeneration
GuidedGenerationLoop.run(...)accepts the caller'slogitProcessor:andlogitSampler:(#733)GrammarConstraint.freshMatcher()(#732)
MLXFoundationModels
MLXLanguageModel.prefillStepSize(#745)MLXLanguageModel.init(... progress:load:)reports load progress (#551)
MLXLLM / MLXVLM
Qwen35TextConfigurationgeometry is public and read-only (#737)Gemma4.hiddenStates(_:)/Gemma4Unified.hiddenStates(_:)via@_spi(GemmaEncoder)(#730)MultimodalEmbedding,MultimodalEmbeddingFactory,EmbeddingGemma2Embedding(#684, #692)
MLXScriptedLM (new library)
- Deterministic tokenizers and models for tests. No weights or downloads (#680, #701, #719)
ScriptedTokenizer,PseudoWordTokenizer,ScriptedChatTemplate,ChatMLTemplateScriptedLogitsModel,ForwardingLanguageModel,TranscriptKVCache,ScriptedScenario
What's Changed
- update documentation and skills by @davidkoski in #489
- Drop MTP weights from converted Qwen 3.5 VLM checkpoints by @sebastien-burel in #642
- Synchronize cross-SIMD WHT reads before shared-memory writes by @dbzoo in #650
- NemotronH: make MoE config keys optional so dense checkpoints load by @templetongroup in #635
- Declare the fused GDN projection as compile state so it frees with the model by @spokvulcan in #631
- Synchronize Metal before checking cached buffer memory by @aleroot in #667
- Recommend upToNextMinor in installation instructions by @mattt in #668
- Reduce Qwen3 reranking memory by projecting final tokens only by @aleroot in #621
- Fix Jina reranker cosine scoring precision (#651) by @aleroot in #666
- Make reranker cancellation test deterministic by @aleroot in #669
- Introduce component-aware checkpoint loading by @aleroot in #679
- Report KV cache memory in KVCacheStatus by @aleroot in #639
- Fix Gemma tool calls losing nested object and array arguments to raw strings by @Adi2K in #626
- Add generation observability by @aleroot in #640
- 🐞 Tokenizer: Flush held text when streaming generation ends by @aleroot in #670
- Fix GLM4MoELite quantized KV cache attention by @aleroot in #671
- perf(tools): scan only the chunk for a collecting tool call's end tag by @spokvulcan in #625
- MLXEmbedders: use bidirectional attention in EmbeddingGemma by @MBark286 in #654
- Split hidden states from logit projection for causal LMs by @aleroot in #672
- Conform Qwen3VL to PreparedInputSplitting by @NivDvir in #627
- Parse GLM-4-0414 markerless tool calls (#491) by @aleroot in #623
- Fix epsilon scaling in Qwen gated delta q/k normalization by @baozzz1 in #633
- Let a ChatSession fork its cache into a new conversation by @aleroot in #641
- Fix Qwen reasoning history reconstruction by @ngutech21 in #663
- Reduce unused vocabulary projections during Qwen prefill (~8% speedup) by @aleroot in #614
- scripted Tokenizer by @davidkoski in #680
- MLXEmbedders: build the README example mask from text lengths by @MBark286 in #685
- Reuse Gemma4 softcap fusion for text inference by @aleroot in #675
- Add compiled decode for Qwen3.5 VLM models by @ngutech21 in #662
- Load model components on demand, starting with MuseGlimmer's vision encoder by @aleroot in #580
- Generation: honor suppress_tokens and mask multimodal placeholder tokens by @fdagostino in #412
- Report the reused cache prefix before prefill by @aleroot in #644
- fix tests on main after PR merges by @davidkoski in #690
- Complete VLM processor TODOs by @RNT56 in #301
- Add EmbeddingGemma 2 full support by @aleroot in #684
- Add Qwen 3.8 support for text, VLM, and MTP speculative decoding by @aleroot in #545
- Fix the two tests failing on main by @spokvulcan in #696
- Allow tokenizers to provide streaming detokenizers by @aleroot in #619
- GuidedGenerationLoop: log diagnostics only when diagnostic log is enabled by @tseylerd in #693
- Throw MLX errors raised on the loader's threads by @AquiGorka in #698
- Fix exact prefix rewinds in rotating KV caches by @aleroot in #622
- Drop the attention mask from text-only VLM prompts by @aleroot in #700
- Omit empty text from message content parts by @aleroot in #699
- Support for scripted models (tests) by @davidkoski in #701
- fix(fusion): Preserve scaled NVFP4 projections by @achembarpu in #705
- Add Mellum 2.1 support by @aleroot in #697
- Read transformers 5 keys in LFM2-VL configs by @ykhrustalev in #695
- fix loratrain for tiny prompts by @davidkoski in #683
- Generate JSON properties in the schema's declared order by @ctymoszek in #718
- update for new TF32 comparison issues by @davidkoski in #721
- switch to MLXLogger by @davidkoski in #702
- add ScriptedLogitsModel by @davidkoski in #719
- 🐞 MLXVLM: Fix Qwen3VL attention masks after cache rotation by @aleroot in #706
- Expose request-scoped generation observations in ChatSession by @aleroot in #714
- Clear the MLX buffer cache after each prefill chunk by @aleroot in #710
- Gemma 4 VLM: expose hidden states for a prepared input (@_spi(GemmaEncoder)) by @jonpol01 in #730
- mlx-swift 0.33.0 by @davidkoski in #720
- Read Gemma 3 RoPE scaling from rope_parameters by @rchatham in #723
- Throw instead of trapping when Bert id2label keys collide by @krishhgg in #736
- FastVLM: Keep image sentinels out of the repetition-penalty history by @ronaldmannak in #735
- Fix attention masks sized from the absolute cache offset by @aleroot in #728
- Prefill over a quantized KV cache through the fused SDPA kernel by @aleroot in #703
- Add reasoning streaming events and support Gemma 4 channels by @abra-code in #563
- Support Nemotron 3.5 (Lightning, Bolt, Super) text models by @ancientcomputing in #712
- Expose Qwen 3.5 text geometry read-only by @ronaldmannak in #737
- Add GrammarConstraint.freshMatcher() by @quartzjer in #732
- Feed the sampled token before its fast-forward tokens by @quartzjer in #731
- MLXFoundationModels: let MLXLanguageModel set the prefill step size by @MBark286 in #745
- Report model used memory by @aleroot in #746
- Use the MLXFast gated delta kernel where mlx core supports it by @aleroot in #740
- Add MiniCPM5 Tool Call Parser by @ibrahimcetin in #637
- integration tests to run using xcode 27 (now that it is released) by @davidkoski in #743
- EmbeddingGemma 2: embed a batch of image sequences in one call by @adriangalilea in #729
- Show the response schema to the model during guided generation by @ctymoszek in #715
- RotatingKVCache: keep the pinned prefix visible in multi-token masks by @ahuzmeza in #747
- Fix EmbeddingGemma 2 image orientation and batch text embeddings by @aleroot in #692
- Add resumable Qwen MTP speculative decoding to ChatSession by @ngutech21 in #581
- Put images before text in GLM-OCR prompts by @mattt in #765
- Keep Nemotron-H Mamba activations in the input dtype by @Augustas11 in #754
- Fix the x86_64 build of the EmbeddingGemma 2 image processor by @aleroot in #763
- Decode surrogate pairs and the remaining Python escapes in PythonLiteralParser by @unaltolga in #769
- Report the progress of reading the weights while loading a model by @aleroot in #551
- Add production-ready LFM2.5 model support by @aleroot in #528
- Add mid-turn steering to ChatSession by @aleroot in #608
- Stop image prompts from running VLM language models in float32 by @aleroot in #766
- Add LiquidAI LFM3 model support by @pHequals7 in #750
- MLXVLM Qwen35: apply YaRN from rope_parameters on the VLM rotary by @xocialize in #761
- Report token log probabilities during speculative decoding by @aleroot in #774
- Batch greedy MTP verification into one GPU readback per round by @aleroot in #724
- Skip the checkpoint components a model does not have by @aleroot in #711
- Measure the memory of an EmbeddingGemma 2 embedding by @aleroot in #787
- Parse triple-quoted strings in PythonLiteralParser by @unaltolga in #786
- Fix the flaky usage test on the tool-calling path by @ahuzmeza in #784
- Add ModelTypeRegistry.contains(_:configuration:) for shared model types by @aleroot in #778
- Use MLXLogger for the model component warning by @aleroot in #790
- Serialize integration tests and their model loading. by @davidkoski in #760
- GLM-OCR: count the temporal patch in the pixel budget by @ahuzmeza in #783
- fix: render null tool-call arguments in chat history by @ykhrustalev in #780
- Base Gemma diffusion implementation by @aleroot in #352
- Add LiquidAI LFM3 text support with extended validation and regression tests by @alvarogarciapiz in #756
- Match the DeepSeek-V3 MoE gate to mlx-lm by @Augustas11 in #795
- Read r/u prefixes and adjacent string literals in PythonLiteralParser by @unaltolga in #799
- Keep the DeepSeek-V3 MoE output in the activation dtype by @Augustas11 in #793
- Fix GLM-OCR small images; match smart_resize rounding in QwenVL.targetSize by @sandeepvijayarao09 in #764
- Keep Qwen2-VL vision attention inside each image by @Augustas11 in #755
- steering follow up and chat interrupt API by @aleroot in #798
- feat(models): add Bonsai 2 27B support by @spokvulcan in #630
- Keep GPT-OSS activations in the input dtype (SwiGLU clip bounds) by @Augustas11 in #792
- Declare default(measured) budget transitions for Muse Glimmer and Qwen3.5 by @aleroot in #734
- Let guided generation use the caller's sampler and logit processor by @quartzjer in #733
- Refuse LoRA/DoRA on layers the adapters cannot reproduce (#800) by @aleroot in #803
- Size five float32 test comparisons for TF32 matmuls by @Augustas11 in #802
New Contributors
- @sebastien-burel made their first contribution in #642
- @dbzoo made their first contribution in #650
- @templetongroup made their first contribution in #635
- @mattt made their first contribution in #668
- @MBark286 made their first contribution in #654
- @baozzz1 made their first contribution in #633
- @tseylerd made their first contribution in #693
- @AquiGorka made their first contribution in #698
- @achembarpu made their first contribution in #705
- @jonpol01 made their first contribution in #730
- @rchatham made their first contribution in #723
- @krishhgg made their first contribution in #736
- @ancientcomputing made their first contribution in #712
- @quartzjer made their first contribution in #732
- @ibrahimcetin made their first contribution in #637
- @adriangalilea made their first contribution in #729
- @ahuzmeza made their first contribution in #747
- @Augustas11 made their first contribution in #754
- @unaltolga made their first contribution in #769
- @pHequals7 made their first contribution in #750
- @alvarogarciapiz made their first contribution in #756
- @sandeepvijayarao09 made their first contribution in #764
Full Changelog: 3.32.3...3.33.0