Skip to content

Version 2.9.36-experimental

Pre-release
Pre-release

Choose a tag to compare

@nieubank nieubank released this 18 Nov 14:38
· 1 commit to main since this release
acb606a

Windows ML 2.9.36 Experimental is the first public 2.9 experimental release. Tasks can now run a model you give them, such as published Hugging Face weights, with no pipeline or tokenizer to set up, on the new CGC backend, which compiles models into programs for a Direct3D 12 GPU. Text generation can now embed a statistical watermark in the text it samples, which a detector with your key can recognize later. This release also adds opt-in llama.cpp CUDA packages that run GGUF models on NVIDIA GPUs.

These APIs are still an early look and will change as they mature. The samples and documentation for this release are on the main branch.

Get started

We'd love your feedback on the API shape, the developer experience, and the models and scenarios that matter most to you. Open an issue to help shape our roadmap.

What's new since 2.7.2021-experimental

Models-backed Tasks

A Task can take a model your app loaded, in one of three ways.

  • Bring your own weights. Load a published model, a Hugging Face checkpoint folder of safetensors weights or a GGUF file, with LoadModelFromPath or LoadModelFromFile, and call SetModel. Windows ML plans it for the CGC backend, runs it on the GPU target you picked, and creates the tokenizer from the model's files. A DFlash 2 drafter checkpoint can be its draft model for speculative decoding.
  • Bring your own model. Describe the graphs you exported with SetComposition and a UNIFIED, EMBEDDING_DECODER or PREFILL_DECODE description.
  • llama.cpp, as an explicit choice. Load a GGUF file, and an optional draft model, with IWinMLLlamaModelLoader. Windows ML never moves a model to another engine on its own.

The new decision Task evaluates a state, such as a message or a support ticket, against typed questions and returns a probability for each option. Speech recognition can take a Whisper checkpoint folder.

Text watermarking

Text generation can embed a statistical watermark in the tokens it samples, and a detector that holds your key can later check whether text carries it.

  • The text stays the model's own. A keyed pseudorandom function of the preceding tokens picks among the candidates in proportion to their probabilities, so each token still follows the model's distribution and the text format doesn't change.
  • Three algorithms. Gumbel uses your key for every watermarked token. Dual-key Gumbel lets regenerated responses vary, with a weaker signal per token. SynthID uses tournament sampling over keyed layers.
  • Detection without the model. IWinMLTextGenerationWatermarkDetector scores generated token IDs with the same key and options, without the model or the prompt. A small p-value, such as below 1e-3, indicates the watermark.
  • Statistics and policy. IWinMLTextGenerationWatermarkResult counts the watermarked tokens and reports the key identifier, never the key. A request can opt out unless the configuration requires the watermark, and speculative decoding keeps it on every committed token.
  • From C++, C#, Python and the server. Set it with IWinMLTextGenerationWatermarkConfiguration, the C# sample wrapper, or set_watermark and detect_watermark in Python. WinMLServer.exe watermarks a model lane with --watermark gumbel, dual-key-gumbel or synthid and --watermark-key, and reports the watermarked token counts in its responses.

Only tokens sampled with a positive temperature carry the watermark, so greedy decoding, the default, doesn't. A watermark is cooperative provenance, not tamper evidence, and editing or paraphrasing weakens it. See Watermarking, and try the watermark scenario of the text generation features sample.

The CGC backend

  • Compiles a model into programs for a Direct3D 12 device and runs them on the GPU target your app picked. Models-backed Tasks run on it.
  • IWinMLModelCompiler writes CGC artifacts that pipelines and compositions load. A device-targeted .oir file loads fastest on the device that compiled it, and a durable .mlirbc file keeps loading across devices and driver updates.
  • A program cache, compiledProgramDirectory in the Task model options, keeps compiled programs, and optionally the prepared weights, for later processes.
  • The package brings the Direct3D 12 Agility SDK through Microsoft.Direct3D.D3D12 1.620.1, and deploys it and DirectML beside x64 and Arm64 apps. ARM64EC apps get no CGC payload.

Runtime API

  • IWinMLStageExecutionDiagnostics reports whether all, some or none of a stage's work runs on its target's device, by device and execution technology.
  • IWinMLStageHostBuffers binds app-owned host memory to several stage inputs and outputs in one call, without creating tensors, and IWinMLCallerBoundStageOptions declares that the app supplies device storage for every input and output of a stage.
  • IWinMLTensorHostReadback reads a tensor backed by a D3D12 buffer back to host memory.
  • IWinMLModelWeightFormats describes how a loaded model stores its weights.
  • IWinMLStreamingResourceMapReader lets a resource map reader supply each resource's digest and provide each resource when it's read, instead of keeping every resource available for the whole load.
  • The package includes ONNX Runtime 1.30.1.

llama.cpp

  • Microsoft.Windows.AI.MachineLearning.LibLlama.Core 0.7.0.107-experimental moves llama.cpp to build 11847, with backend API 3.
  • Apps that already call llama.cpp's own C API can set WinMLLinkLlamaUpstreamApi to true and build against the llama.cpp core in LibLlama.Core instead of shipping their own. The llama-upstream-api sample shows how.
  • New opt-in CUDA backend packages for NVIDIA GPUs. Microsoft.Windows.AI.MachineLearning.LibLlama brings the Core with LibLlama.Cuda.win-x64 and LibLlama.Cuda.win-arm64, and the C++ and C# samples deploy the backend with build.ps1 -LlamaCuda. It needs a driver that supports CUDA 13 and NVIDIA cuBLAS 13.7 or later, which the packages don't include. NVIDIA CUDA describes where the build finds cuBLAS and NVIDIA's license terms.
  • In Python, windowsml[llama] installs the CPU backend, windowsml-llama-cuda adds the CUDA backend and NVIDIA's nvidia-cublas wheel, and windowsml-llama installs windowsml with both backends.

Chat

  • Tool schemas keep the property order the client sent, so the prompt lists a tool's arguments as the client declared them.
  • A Qwen3-Coder style tool call that writes an optional argument before a required one now parses.
  • Qwen3 chat with thinking off streams its content per token.

Breaking changes

  • The Whisper-specific speech recognition names are renamed, with the same IIDs and behavior. IWinMLWhisperAutomaticSpeechRecognitionConfiguration is now IWinMLEncoderDecoderAutomaticSpeechRecognitionConfiguration, WINML_WHISPER_AUTOMATIC_SPEECH_RECOGNITION_BINDINGS is now WINML_ENCODER_DECODER_AUTOMATIC_SPEECH_RECOGNITION_BINDINGS, and IWinMLWhisperLogMelInput::TranscribeLogMel is now IWinMLAutomaticSpeechRecognitionEncoderInput::TranscribeEncoderInput. Rebuild code that calls SetBindings or GetBindings against the 2.9 headers.
  • The default GPU target no longer downloads WebGPU. On a PC with no installed GPU provider, a pipeline on that target fails to build with HRESULT_FROM_WIN32(ERROR_NOT_SUPPORTED). Pin WebGPU with IWinMLOrtCompatibility::CreateExecutionTarget, and Windows ML acquires it when the pipeline builds.
  • llama.cpp GPU backends built for 2.7 must be rebuilt for backend API 3, and W3_G32 GGUF files no longer load.

Known issues

  • Without SetLogicalCapacity, a session from published weights holds at most 8,192 tokens, even when the model supports more, and a prompt that doesn't fit fails with HRESULT_FROM_WIN32(ERROR_INSUFFICIENT_BUFFER). Call SetLogicalCapacity after SetModel for longer prompts.
  • With the WebGPU execution provider, a process can stop responding at exit after it prints its result, most often while the GPU is busy with other work.
  • With some GPU providers, a sample that exits normally can lose output that's redirected to a file or a pipe. Run it in a console to see its output.
  • A WebGPU session deletes the TensorRT for RTX execution provider's compiled context for the same model, so the next TensorRT for RTX session compiles it again.
  • With a CUDA backend built by Tools/llama instead of the windowsml-llama-cuda wheel, Python warns that the CUDA backend is unavailable, though the model still runs on CUDA.
  • On a PC without an NVIDIA GPU, a sample built with -LlamaCuda prints one ggml_cuda_init: failed to initialize CUDA line, then runs on the CPU.

The Runtime samples README lists more.

Packages

NuGet package Version Contents
Microsoft.Windows.AI.MachineLearning 2.9.36-experimental The Runtime and Task APIs, with ONNX Runtime 1.30.1
Microsoft.Windows.AI.MachineLearning.LibLlama.Core 0.7.0.107-experimental llama.cpp with the CPU backend
Microsoft.Windows.AI.MachineLearning.LibLlama.Cuda.win-x64 0.7.0.107-experimental The opt-in CUDA backend for x64, new
Microsoft.Windows.AI.MachineLearning.LibLlama.Cuda.win-arm64 0.7.0.107-experimental The opt-in CUDA backend for Arm64, new
Microsoft.Windows.AI.MachineLearning.LibLlama 0.7.0.107-experimental The Core with both CUDA backends, new
PyPI project Version Contents
windowsml 2.9.36a0 Windows ML for Python. The with-ort extra adds ONNX Runtime, and the llama extra adds the llama.cpp CPU backend
windowsml-llama-core 2.9.36a0 The llama.cpp CPU backend
windowsml-llama-cuda 2.9.36a0 The opt-in CUDA backend, new
windowsml-llama 2.9.36a0 windowsml with the CPU and CUDA backends, new

What's changed

Full changelog