Skip to content

3.33.0

Latest

Choose a tag to compare

@davidkoski davidkoski released this 17 Nov 22:43
· 2 commits to main since this release
ca661dc

New Models

MLXLLM

  • Qwen 3.8 (qwen3_8, qwen3_8_moe, qwen3_8_text), including MTP drafters (#545)
  • PrismML Bonsai 2 27B, ternary Hadamard-rotated Qwen 3.8 (prism_hadamard_qwen35) (#630)
  • LiquidAI LFM3 (lfm3) (#750, #756)
  • LiquidAI LFM2.5 dense and MoE checkpoints (lfm2, lfm2_moe) and the LFM2 DSpark drafter (#528)
    • the DSpark drafter must be registered with await LFM2DSparkRegistration.register()
  • JetBrains Mellum 2.1 (mellum) (#697)
  • Nemotron 3.5 Lightning, Bolt (nemotron_h_puzzle), and Super; dense Nemotron-H checkpoints (#712, #635)

MLXVLM

  • Qwen 3.8 VLM (qwen3_8, qwen3_8_moe) (#545)
  • Bonsai 2 VLM (prism_hadamard_qwen35) (#630)
  • DiffusionGemma, a block-diffusion model (diffusion_gemma) (#352)

MLXEmbedders

  • EmbeddingGemma 2 (embedding_gemma2, embedding_gemma2_text) (#684)
    • multimodal (text, image, video, audio) embeddings via MultimodalEmbeddingFactory in MLXVLM (#684, #692, #729)

Tool Calling

  • MiniCPM5 tool call format: ToolCallFormat.miniCPM5 (#637)

API Change

Source Breaking

  • MLXLMCommon.compile(shapeless:_:) overloads are removed (#720)
    • mlx-swift 0.33.0 now provides the same @Sendable overloads, so unqualified compile(...) calls still build
    • only code that spells out MLXLMCommon.compile breaks
  • MTPDrafterModel.validateCompatibility(with:) throws is a new protocol requirement with no default (#581)
    • custom MTP drafters must implement it
  • IntegrationTestModels changed from an actor to a struct and evictLLM(_:) was removed (#760)
    • loads go through TestModelManager.shared, which keeps one model in memory at a time

Potentially Source Breaking

  • BaseLanguageModel.sanitize(weights:) and sanitize(weights:metadata:) now throws (#679)
    • conformers that do not throw still compile
    • callers that go through the protocol need try
    • sanitize on Gemma3nText, Qwen35, Qwen35MoE, and the Qwen 3.5 MTP drafters (MLXLLM and MLXVLM) now throws, so direct callers need try and subclass overrides need throws
  • New cases on public enums. Exhaustive switch statements need new arms:
    • Generation.reasoning(String) (#563) and Generation.steering(SteeringEvent) (#608)
    • GenerateStopReason.steered (#608) and .interrupted (#798)
    • ToolCallFormat.miniCPM5 (#637)
    • ModelAdapterError.unsupportedLayer(key:type:) (#803)
    • KVCacheConfigurationError.invalidRewindCapacity, .incompatibleRewindCapacity (#622)
    • ModelConfiguration.DirectoryError.invalidModelSubdirectory (#528)
  • New KVCache / BaseKVCache members, each with a default: currentKeyValues(), maxTrimCount, evictedTokenCount, serializedClassName, restore(state:metaState:) (#352, #622, #640, #701)
    • a BaseKVCache subclass that already declares one of these names must add override
  • LoRALinear / DoRALinear / QLoRALinear / QDoRALinear now trap unless the base layer is exactly Linear or QuantizedLinear (#803)
    • LoRAContainer.from(model:configuration:) throws .unsupportedLayer and leaves the model unchanged
  • Gemma4VisionConfiguration moved from MLXVLM to MLXLMCommon. MLXVLM keeps a typealias (#684)
  • ChatSession.respond(...) and synchronize() are now nonisolated(nonsending) (#608)
  • MLXLanguageModel.init(... load: ContainerLoader) is deprecated in favor of an overload that takes LoadProgressHandlers (#551, #745)
  • the progressHandler: load overloads are deprecated in favor of progress: LoadProgressHandlers (#551)

Behavior Change

  • Reasoning streams separately from the answer (#563)
    • models with a reasoningConfig (<think> families, DeepSeek-R1, Gemma 4, LFM2.5, Harmony, Onyx) now emit thinking as .reasoning, without the delimiters
    • string APIs (respond, streamResponse) no longer include thinking text
    • ChatSession keeps the thought and replays it as reasoning_content (#663)
  • Qwen 3.5 and Muse Glimmer honor a hard thinking budget (#734)
  • Generation masks suppress_tokens from generation_config.json and multimodal placeholder tokens (#412)
  • Sampling defaults from generation_config.json are exposed via ModelContainer.defaultGenerateParameters (#528)
  • Text-only prompts on several VLMs no longer carry an attention mask, so ChatSession reuses the KV cache across turns (#700)
  • Image prompts no longer promote Qwen2-VL, Qwen2.5-VL, FastVLM, and Mistral3 to float32 (#766)
    • GPT-OSS (#792), DeepSeek-V3 MoE (#793), and Nemotron-H Mamba (#754) also stay in the activation dtype
  • The DeepSeek-V3 MoE gate now matches mlx-lm, so it may route to different experts (#795)
  • EmbeddingGemma uses bidirectional attention when the config asks for it. Embedding values change to match the reference (#654)
  • Loading
    • the loader skips checkpoint components that a model does not have, e.g. vision weights for a text model (#711)
    • vision encoders for Qwen 3.5/3.8, LFM2-VL, and Muse Glimmer load on first media use (#580)
    • MLX errors on loader threads are now thrown (#698)
    • factories call model.validate(tokenizer:), so a mismatched tokenizer fails at load time (#756)
    • weight loading reports progress per byte, in its own weights phase (#551)
  • KV cache
    • savePromptCache throws for a cache type with no serializedClassName. Before, it was silently saved as "KVCache" (#701)
    • RotatingKVCache rewinds exact prefixes (#622), keeps the pinned prefix visible (#747), and sizes masks from the retained window (#706, #728)
    • prefill over a QuantizedKVCache uses the fused SDPA kernel (#703)
    • the MLX buffer cache is cleared after each prefill chunk (#710)
  • Streaming detokenizers flush held text when generation ends (#670)
  • Guided generation feeds the sampled token before the fast-forward tokens (#731) and puts the response schema in the prompt (#715)
  • Logging goes through mlx-swift's MLXLogger (#702, #790)
  • LoRATrain.train throws LoRATrainError.sampleTooShort instead of crashing (#683)

New API

MLXLMCommon

  • ChatSession
    • mid-turn steering: steer(_:policy:response:), canSteer(_:), ResponseID, latestResponse, SteeringEvent, SteeringPolicy, SteeringError (#608)
    • interrupt(response:) ends a response at the next safe point and keeps the output (#798)
    • fork(history:) copies the cache into a new session (#641)
    • observe(to:using:) streams events through a GenerationTransformer, e.g. .logProbabilities(topCount:) (#714)
    • resumable MTP speculative decoding: SpeculativeDecodingConfig(mtpDrafter:blockSize:), SpeculativeDecodingFallbackReason (#581)
  • Generation observability (#640)
    • public TokenLoopHandler, TextToolTokenLoopHandler, RawTokenLoopHandler, generateLoopTask(...)
    • token log probabilities: GenerateParameters.logProbabilities, LogProbabilityTokenLoopHandler, also during speculative decoding (#774)
    • GenerateCompletionInfo.evictedTokenCount, .reasoningTokenCount, .answerTokenCount
    • PrefillParameters.prefix reports the reused cache prefix before prefill (#644)
  • Reasoning: ReasoningChannel, ReasoningConfig.gemma4, ReasoningConfig.declaringBudgetTransition(_:), QwenReasoningProtocol.qwen35, OnyxReasoningProtocol (#563, #734)
  • Block diffusion: BlockDiffusionLanguageModel, BlockDiffusionTokenIterator, BlockDiffusionParameters, GenerateParameters.diffusion, LMModelCapabilities (#352)
    • generate(...) dispatches to the diffusion iterator automatically
  • Model loading
    • BaseLanguageModel.prepareCheckpoint(_:), excludedCheckpointNamespaces, validate(tokenizer:) (#679, #580, #756)
    • ModelCheckpoint, CheckpointComponent, CheckpointNameMapping (#679, #711)
    • on-demand components: ModelComponent, ModelComponentsProviding, OnDemandComponent, ComponentLoading, ModelConfiguration.excludedComponents (#580)
    • LoadProgressHandlers (download, weights) (#551)
    • ModelConfiguration.modelSubdirectory loads a model from a repository subfolder (#528)
    • ModelTypeRegistry.contains(_:configuration:) for shared model_types (#778)
  • Memory reporting
    • ModelMemory, ModelMemoryReporting, BaseLanguageModel.memory(), ModelContainer.memory(), EmbedderModelContainer.memory() (#746)
    • KVCacheStatus.memoryBytes, KVCacheLayerStatus.memoryBytes (#639)
  • KV cache
    • registerKVCacheType(_:) for custom prompt-cache types, AuxiliaryKVCache (#701)
    • RotatingKVCache(... rewindCapacity:), KVCacheConfiguration.Rewind, canTrimPromptCache(_:numTokens:) (#622)
    • RewindableConvolutionCache (#528)
  • LanguageModel.nextTokenLogits(_:cache:state:) computes only the last-position logits during prefill (#614)
  • Suppressed tokens: SuppressedTokensProviding, SuppressTokensProcessor, GenerationConfigFile.suppressTokens (#412)
  • Tokenizer.makeStreamingDetokenizer() (#619) and StreamingDetokenizer.finish() (#670)
  • LogitProcessor.didEmit(token:) and finalizeGeneration() (#640)
  • Hadamard-rotated quantization: HadamardQuantizedLinear, HadamardQuantizedEmbedding, and supporting types (#630)
  • MTP: ResumableMTPDrafterModel, MTPDrafterCompatibilityError, MTPDrafterModel.targetLayerIds (#581, #528)

MLXGuidedGeneration

  • GuidedGenerationLoop.run(...) accepts the caller's logitProcessor: and logitSampler: (#733)
  • GrammarConstraint.freshMatcher() (#732)

MLXFoundationModels

  • MLXLanguageModel.prefillStepSize (#745)
  • MLXLanguageModel.init(... progress:load:) reports load progress (#551)

MLXLLM / MLXVLM

  • Qwen35TextConfiguration geometry is public and read-only (#737)
  • Gemma4.hiddenStates(_:) / Gemma4Unified.hiddenStates(_:) via @_spi(GemmaEncoder) (#730)
  • MultimodalEmbedding, MultimodalEmbeddingFactory, EmbeddingGemma2Embedding (#684, #692)

MLXScriptedLM (new library)

  • Deterministic tokenizers and models for tests. No weights or downloads (#680, #701, #719)
    • ScriptedTokenizer, PseudoWordTokenizer, ScriptedChatTemplate, ChatMLTemplate
    • ScriptedLogitsModel, ForwardingLanguageModel, TranscriptKVCache, ScriptedScenario

What's Changed

  • update documentation and skills by @davidkoski in #489
  • Drop MTP weights from converted Qwen 3.5 VLM checkpoints by @sebastien-burel in #642
  • Synchronize cross-SIMD WHT reads before shared-memory writes by @dbzoo in #650
  • NemotronH: make MoE config keys optional so dense checkpoints load by @templetongroup in #635
  • Declare the fused GDN projection as compile state so it frees with the model by @spokvulcan in #631
  • Synchronize Metal before checking cached buffer memory by @aleroot in #667
  • Recommend upToNextMinor in installation instructions by @mattt in #668
  • Reduce Qwen3 reranking memory by projecting final tokens only by @aleroot in #621
  • Fix Jina reranker cosine scoring precision (#651) by @aleroot in #666
  • Make reranker cancellation test deterministic by @aleroot in #669
  • Introduce component-aware checkpoint loading by @aleroot in #679
  • Report KV cache memory in KVCacheStatus by @aleroot in #639
  • Fix Gemma tool calls losing nested object and array arguments to raw strings by @Adi2K in #626
  • Add generation observability by @aleroot in #640
  • 🐞 Tokenizer: Flush held text when streaming generation ends by @aleroot in #670
  • Fix GLM4MoELite quantized KV cache attention by @aleroot in #671
  • perf(tools): scan only the chunk for a collecting tool call's end tag by @spokvulcan in #625
  • MLXEmbedders: use bidirectional attention in EmbeddingGemma by @MBark286 in #654
  • Split hidden states from logit projection for causal LMs by @aleroot in #672
  • Conform Qwen3VL to PreparedInputSplitting by @NivDvir in #627
  • Parse GLM-4-0414 markerless tool calls (#491) by @aleroot in #623
  • Fix epsilon scaling in Qwen gated delta q/k normalization by @baozzz1 in #633
  • Let a ChatSession fork its cache into a new conversation by @aleroot in #641
  • Fix Qwen reasoning history reconstruction by @ngutech21 in #663
  • Reduce unused vocabulary projections during Qwen prefill (~8% speedup) by @aleroot in #614
  • scripted Tokenizer by @davidkoski in #680
  • MLXEmbedders: build the README example mask from text lengths by @MBark286 in #685
  • Reuse Gemma4 softcap fusion for text inference by @aleroot in #675
  • Add compiled decode for Qwen3.5 VLM models by @ngutech21 in #662
  • Load model components on demand, starting with MuseGlimmer's vision encoder by @aleroot in #580
  • Generation: honor suppress_tokens and mask multimodal placeholder tokens by @fdagostino in #412
  • Report the reused cache prefix before prefill by @aleroot in #644
  • fix tests on main after PR merges by @davidkoski in #690
  • Complete VLM processor TODOs by @RNT56 in #301
  • Add EmbeddingGemma 2 full support by @aleroot in #684
  • Add Qwen 3.8 support for text, VLM, and MTP speculative decoding by @aleroot in #545
  • Fix the two tests failing on main by @spokvulcan in #696
  • Allow tokenizers to provide streaming detokenizers by @aleroot in #619
  • GuidedGenerationLoop: log diagnostics only when diagnostic log is enabled by @tseylerd in #693
  • Throw MLX errors raised on the loader's threads by @AquiGorka in #698
  • Fix exact prefix rewinds in rotating KV caches by @aleroot in #622
  • Drop the attention mask from text-only VLM prompts by @aleroot in #700
  • Omit empty text from message content parts by @aleroot in #699
  • Support for scripted models (tests) by @davidkoski in #701
  • fix(fusion): Preserve scaled NVFP4 projections by @achembarpu in #705
  • Add Mellum 2.1 support by @aleroot in #697
  • Read transformers 5 keys in LFM2-VL configs by @ykhrustalev in #695
  • fix loratrain for tiny prompts by @davidkoski in #683
  • Generate JSON properties in the schema's declared order by @ctymoszek in #718
  • update for new TF32 comparison issues by @davidkoski in #721
  • switch to MLXLogger by @davidkoski in #702
  • add ScriptedLogitsModel by @davidkoski in #719
  • 🐞 MLXVLM: Fix Qwen3VL attention masks after cache rotation by @aleroot in #706
  • Expose request-scoped generation observations in ChatSession by @aleroot in #714
  • Clear the MLX buffer cache after each prefill chunk by @aleroot in #710
  • Gemma 4 VLM: expose hidden states for a prepared input (@_spi(GemmaEncoder)) by @jonpol01 in #730
  • mlx-swift 0.33.0 by @davidkoski in #720
  • Read Gemma 3 RoPE scaling from rope_parameters by @rchatham in #723
  • Throw instead of trapping when Bert id2label keys collide by @krishhgg in #736
  • FastVLM: Keep image sentinels out of the repetition-penalty history by @ronaldmannak in #735
  • Fix attention masks sized from the absolute cache offset by @aleroot in #728
  • Prefill over a quantized KV cache through the fused SDPA kernel by @aleroot in #703
  • Add reasoning streaming events and support Gemma 4 channels by @abra-code in #563
  • Support Nemotron 3.5 (Lightning, Bolt, Super) text models by @ancientcomputing in #712
  • Expose Qwen 3.5 text geometry read-only by @ronaldmannak in #737
  • Add GrammarConstraint.freshMatcher() by @quartzjer in #732
  • Feed the sampled token before its fast-forward tokens by @quartzjer in #731
  • MLXFoundationModels: let MLXLanguageModel set the prefill step size by @MBark286 in #745
  • Report model used memory by @aleroot in #746
  • Use the MLXFast gated delta kernel where mlx core supports it by @aleroot in #740
  • Add MiniCPM5 Tool Call Parser by @ibrahimcetin in #637
  • integration tests to run using xcode 27 (now that it is released) by @davidkoski in #743
  • EmbeddingGemma 2: embed a batch of image sequences in one call by @adriangalilea in #729
  • Show the response schema to the model during guided generation by @ctymoszek in #715
  • RotatingKVCache: keep the pinned prefix visible in multi-token masks by @ahuzmeza in #747
  • Fix EmbeddingGemma 2 image orientation and batch text embeddings by @aleroot in #692
  • Add resumable Qwen MTP speculative decoding to ChatSession by @ngutech21 in #581
  • Put images before text in GLM-OCR prompts by @mattt in #765
  • Keep Nemotron-H Mamba activations in the input dtype by @Augustas11 in #754
  • Fix the x86_64 build of the EmbeddingGemma 2 image processor by @aleroot in #763
  • Decode surrogate pairs and the remaining Python escapes in PythonLiteralParser by @unaltolga in #769
  • Report the progress of reading the weights while loading a model by @aleroot in #551
  • Add production-ready LFM2.5 model support by @aleroot in #528
  • Add mid-turn steering to ChatSession by @aleroot in #608
  • Stop image prompts from running VLM language models in float32 by @aleroot in #766
  • Add LiquidAI LFM3 model support by @pHequals7 in #750
  • MLXVLM Qwen35: apply YaRN from rope_parameters on the VLM rotary by @xocialize in #761
  • Report token log probabilities during speculative decoding by @aleroot in #774
  • Batch greedy MTP verification into one GPU readback per round by @aleroot in #724
  • Skip the checkpoint components a model does not have by @aleroot in #711
  • Measure the memory of an EmbeddingGemma 2 embedding by @aleroot in #787
  • Parse triple-quoted strings in PythonLiteralParser by @unaltolga in #786
  • Fix the flaky usage test on the tool-calling path by @ahuzmeza in #784
  • Add ModelTypeRegistry.contains(_:configuration:) for shared model types by @aleroot in #778
  • Use MLXLogger for the model component warning by @aleroot in #790
  • Serialize integration tests and their model loading. by @davidkoski in #760
  • GLM-OCR: count the temporal patch in the pixel budget by @ahuzmeza in #783
  • fix: render null tool-call arguments in chat history by @ykhrustalev in #780
  • Base Gemma diffusion implementation by @aleroot in #352
  • Add LiquidAI LFM3 text support with extended validation and regression tests by @alvarogarciapiz in #756
  • Match the DeepSeek-V3 MoE gate to mlx-lm by @Augustas11 in #795
  • Read r/u prefixes and adjacent string literals in PythonLiteralParser by @unaltolga in #799
  • Keep the DeepSeek-V3 MoE output in the activation dtype by @Augustas11 in #793
  • Fix GLM-OCR small images; match smart_resize rounding in QwenVL.targetSize by @sandeepvijayarao09 in #764
  • Keep Qwen2-VL vision attention inside each image by @Augustas11 in #755
  • steering follow up and chat interrupt API by @aleroot in #798
  • feat(models): add Bonsai 2 27B support by @spokvulcan in #630
  • Keep GPT-OSS activations in the input dtype (SwiGLU clip bounds) by @Augustas11 in #792
  • Declare default(measured) budget transitions for Muse Glimmer and Qwen3.5 by @aleroot in #734
  • Let guided generation use the caller's sampler and logit processor by @quartzjer in #733
  • Refuse LoRA/DoRA on layers the adapters cannot reproduce (#800) by @aleroot in #803
  • Size five float32 test comparisons for TF32 matmuls by @Augustas11 in #802

New Contributors

Full Changelog: 3.32.3...3.33.0