Repository navigation
Version 2.9.36-experimental
Pre-releaseWindows ML 2.9.36 Experimental is the first public 2.9 experimental release. Tasks can now run a model you give them, such as published Hugging Face weights, with no pipeline or tokenizer to set up, on the new CGC backend, which compiles models into programs for a Direct3D 12 GPU. Text generation can now embed a statistical watermark in the text it samples, which a detector with your key can recognize later. This release also adds opt-in llama.cpp CUDA packages that run GGUF models on NVIDIA GPUs.
These APIs are still an early look and will change as they mature. The samples and documentation for this release are on the main branch.
Get started
- Install Microsoft.Windows.AI.MachineLearning 2.9.36-experimental from NuGet, or run
pip install "windowsml[with-ort]==2.9.36a0"for Python. - Run published weights with a Task in the Models-backed Task samples.
- Try chat, completion, speech recognition, decisions and speculative decoding in Task Studio, a WinUI 3 sample app.
- Watermark generated text and check it with a detector in the
watermarkscenario of the text generation features sample. - Call the Runtime and Task APIs from C# with the C# samples, which use CsWin32 bindings.
- Run GGUF models on an NVIDIA GPU with the opt-in CUDA backend.
- See how a model reaches ONNX Runtime, the CGC backend or llama.cpp in Backends and model formats, and browse the API reference.
We'd love your feedback on the API shape, the developer experience, and the models and scenarios that matter most to you. Open an issue to help shape our roadmap.
What's new since 2.7.2021-experimental
Models-backed Tasks
A Task can take a model your app loaded, in one of three ways.
- Bring your own weights. Load a published model, a Hugging Face checkpoint folder of safetensors weights or a GGUF file, with
LoadModelFromPathorLoadModelFromFile, and callSetModel. Windows ML plans it for the CGC backend, runs it on the GPU target you picked, and creates the tokenizer from the model's files. A DFlash 2 drafter checkpoint can be its draft model for speculative decoding. - Bring your own model. Describe the graphs you exported with
SetCompositionand aUNIFIED,EMBEDDING_DECODERorPREFILL_DECODEdescription. - llama.cpp, as an explicit choice. Load a GGUF file, and an optional draft model, with
IWinMLLlamaModelLoader. Windows ML never moves a model to another engine on its own.
The new decision Task evaluates a state, such as a message or a support ticket, against typed questions and returns a probability for each option. Speech recognition can take a Whisper checkpoint folder.
Text watermarking
Text generation can embed a statistical watermark in the tokens it samples, and a detector that holds your key can later check whether text carries it.
- The text stays the model's own. A keyed pseudorandom function of the preceding tokens picks among the candidates in proportion to their probabilities, so each token still follows the model's distribution and the text format doesn't change.
- Three algorithms. Gumbel uses your key for every watermarked token. Dual-key Gumbel lets regenerated responses vary, with a weaker signal per token. SynthID uses tournament sampling over keyed layers.
- Detection without the model.
IWinMLTextGenerationWatermarkDetectorscores generated token IDs with the same key and options, without the model or the prompt. A small p-value, such as below 1e-3, indicates the watermark. - Statistics and policy.
IWinMLTextGenerationWatermarkResultcounts the watermarked tokens and reports the key identifier, never the key. A request can opt out unless the configuration requires the watermark, and speculative decoding keeps it on every committed token. - From C++, C#, Python and the server. Set it with
IWinMLTextGenerationWatermarkConfiguration, the C# sample wrapper, orset_watermarkanddetect_watermarkin Python.WinMLServer.exewatermarks a model lane with--watermark gumbel,dual-key-gumbelorsynthidand--watermark-key, and reports the watermarked token counts in its responses.
Only tokens sampled with a positive temperature carry the watermark, so greedy decoding, the default, doesn't. A watermark is cooperative provenance, not tamper evidence, and editing or paraphrasing weakens it. See Watermarking, and try the watermark scenario of the text generation features sample.
The CGC backend
- Compiles a model into programs for a Direct3D 12 device and runs them on the GPU target your app picked. Models-backed Tasks run on it.
IWinMLModelCompilerwrites CGC artifacts that pipelines and compositions load. A device-targeted.oirfile loads fastest on the device that compiled it, and a durable.mlirbcfile keeps loading across devices and driver updates.- A program cache,
compiledProgramDirectoryin the Task model options, keeps compiled programs, and optionally the prepared weights, for later processes. - The package brings the Direct3D 12 Agility SDK through
Microsoft.Direct3D.D3D121.620.1, and deploys it and DirectML beside x64 and Arm64 apps. ARM64EC apps get no CGC payload.
Runtime API
IWinMLStageExecutionDiagnosticsreports whether all, some or none of a stage's work runs on its target's device, by device and execution technology.IWinMLStageHostBuffersbinds app-owned host memory to several stage inputs and outputs in one call, without creating tensors, andIWinMLCallerBoundStageOptionsdeclares that the app supplies device storage for every input and output of a stage.IWinMLTensorHostReadbackreads a tensor backed by a D3D12 buffer back to host memory.IWinMLModelWeightFormatsdescribes how a loaded model stores its weights.IWinMLStreamingResourceMapReaderlets a resource map reader supply each resource's digest and provide each resource when it's read, instead of keeping every resource available for the whole load.- The package includes ONNX Runtime 1.30.1.
llama.cpp
Microsoft.Windows.AI.MachineLearning.LibLlama.Core0.7.0.107-experimental moves llama.cpp to build 11847, with backend API 3.- Apps that already call llama.cpp's own C API can set
WinMLLinkLlamaUpstreamApitotrueand build against the llama.cpp core in LibLlama.Core instead of shipping their own. The llama-upstream-api sample shows how. - New opt-in CUDA backend packages for NVIDIA GPUs.
Microsoft.Windows.AI.MachineLearning.LibLlamabrings the Core withLibLlama.Cuda.win-x64andLibLlama.Cuda.win-arm64, and the C++ and C# samples deploy the backend withbuild.ps1 -LlamaCuda. It needs a driver that supports CUDA 13 and NVIDIA cuBLAS 13.7 or later, which the packages don't include. NVIDIA CUDA describes where the build finds cuBLAS and NVIDIA's license terms. - In Python,
windowsml[llama]installs the CPU backend,windowsml-llama-cudaadds the CUDA backend and NVIDIA'snvidia-cublaswheel, andwindowsml-llamainstallswindowsmlwith both backends.
Chat
- Tool schemas keep the property order the client sent, so the prompt lists a tool's arguments as the client declared them.
- A Qwen3-Coder style tool call that writes an optional argument before a required one now parses.
- Qwen3 chat with thinking off streams its content per token.
Breaking changes
- The Whisper-specific speech recognition names are renamed, with the same IIDs and behavior.
IWinMLWhisperAutomaticSpeechRecognitionConfigurationis nowIWinMLEncoderDecoderAutomaticSpeechRecognitionConfiguration,WINML_WHISPER_AUTOMATIC_SPEECH_RECOGNITION_BINDINGSis nowWINML_ENCODER_DECODER_AUTOMATIC_SPEECH_RECOGNITION_BINDINGS, andIWinMLWhisperLogMelInput::TranscribeLogMelis nowIWinMLAutomaticSpeechRecognitionEncoderInput::TranscribeEncoderInput. Rebuild code that callsSetBindingsorGetBindingsagainst the 2.9 headers. - The default GPU target no longer downloads WebGPU. On a PC with no installed GPU provider, a pipeline on that target fails to build with
HRESULT_FROM_WIN32(ERROR_NOT_SUPPORTED). Pin WebGPU withIWinMLOrtCompatibility::CreateExecutionTarget, and Windows ML acquires it when the pipeline builds. - llama.cpp GPU backends built for 2.7 must be rebuilt for backend API 3, and W3_G32 GGUF files no longer load.
Known issues
- Without
SetLogicalCapacity, a session from published weights holds at most 8,192 tokens, even when the model supports more, and a prompt that doesn't fit fails withHRESULT_FROM_WIN32(ERROR_INSUFFICIENT_BUFFER). CallSetLogicalCapacityafterSetModelfor longer prompts. - With the WebGPU execution provider, a process can stop responding at exit after it prints its result, most often while the GPU is busy with other work.
- With some GPU providers, a sample that exits normally can lose output that's redirected to a file or a pipe. Run it in a console to see its output.
- A WebGPU session deletes the TensorRT for RTX execution provider's compiled context for the same model, so the next TensorRT for RTX session compiles it again.
- With a CUDA backend built by
Tools/llamainstead of thewindowsml-llama-cudawheel, Python warns that the CUDA backend is unavailable, though the model still runs on CUDA. - On a PC without an NVIDIA GPU, a sample built with
-LlamaCudaprints oneggml_cuda_init: failed to initialize CUDAline, then runs on the CPU.
The Runtime samples README lists more.
Packages
| NuGet package | Version | Contents |
|---|---|---|
| Microsoft.Windows.AI.MachineLearning | 2.9.36-experimental | The Runtime and Task APIs, with ONNX Runtime 1.30.1 |
| Microsoft.Windows.AI.MachineLearning.LibLlama.Core | 0.7.0.107-experimental | llama.cpp with the CPU backend |
| Microsoft.Windows.AI.MachineLearning.LibLlama.Cuda.win-x64 | 0.7.0.107-experimental | The opt-in CUDA backend for x64, new |
| Microsoft.Windows.AI.MachineLearning.LibLlama.Cuda.win-arm64 | 0.7.0.107-experimental | The opt-in CUDA backend for Arm64, new |
| Microsoft.Windows.AI.MachineLearning.LibLlama | 0.7.0.107-experimental | The Core with both CUDA backends, new |
| PyPI project | Version | Contents |
|---|---|---|
| windowsml | 2.9.36a0 | Windows ML for Python. The with-ort extra adds ONNX Runtime, and the llama extra adds the llama.cpp CPU backend |
| windowsml-llama-core | 2.9.36a0 | The llama.cpp CPU backend |
| windowsml-llama-cuda | 2.9.36a0 | The opt-in CUDA backend, new |
| windowsml-llama | 2.9.36a0 | windowsml with the CPU and CUDA backends, new |
What's changed
- Add standalone optional model MSIX sample by mahabayana in #26
- Add Tools/capture-logs for Windows ML diagnostic traces by Siyuan (@sdotpeng) in #28
- Update the samples and documentation to Windows ML 2.9 experimental by Nick Eubanks (@nieubank) in #29
- Pin the samples to Windows ML 2.9.36 experimental by Nick Eubanks (@nieubank) in #36
- Add ONNX Runtime sample with the Windows ML CG Execution Provider by Zarana Kiran Desai (@zadesai) in #39
- Add C# Windows ML samples over the CsWin32 Runtime and Task bindings by adrastogi in #35
- Add Task Studio, a WinUI 3 sample app for the Windows ML Task APIs by Nick Eubanks (@nieubank) in #31
- Document how to recover the disk space the samples use by Nick Eubanks (@nieubank) in #33
- Add a Runtime sample that calls llama.cpp's API directly by Nick Eubanks (@nieubank) in #30
- Add the opt-in llama.cpp CUDA backend to the 2.9 GGUF samples by Nick Eubanks (@nieubank) in #40
- Wrap the rest of the Task API in the C# sample wrapper and add C# Task samples (re-land #38) by adrastogi in #41
- Port Task Studio to the shared C# sample wrapper by adrastogi in #43
- Update the samples and docs to Windows ML 2.9.36 experimental by Nick Eubanks (@nieubank) in #42
- Fix three Task Studio bugs and a leak in DXCoreAdapter.FindBest by adrastogi in #44
- Deploy the opt-in CUDA backend to the C# GGUF samples by Nick Eubanks (@nieubank) in #45