Repository navigation
Version 2.9.42-experimental
Pre-releaseWindows ML 2.9.42 Experimental updates 2.9.36 for long contexts. Text generation from published weights can now hold its attention key and value cache as 8-bit integers, which fits about twice the context in the same GPU memory, and a session whose context doesn't fit in dedicated GPU memory now fails when it's created, with an error that names the largest context that fits. Speculative decoding with a DFlash 2 drafter is faster, and prompts longer than about 104,000 tokens no longer fail.
These APIs are still an early look and will change as they mature. The samples and documentation for this release are on the main branch.
Get started
- Install Microsoft.Windows.AI.MachineLearning 2.9.42-experimental from NuGet, or use the attached "windowsml[with-ort]==2.9.42a0" for Python.
- Choose a session's key and value cache type with
SetKeyValueCacheType, described in Sessions from a model. - The 2.9.36-experimental release describes what's new in 2.9, including Models-backed Tasks, the CGC backend, text watermarking and the opt-in llama.cpp CUDA backend.
We'd love your feedback on the API shape, the developer experience, and the models and scenarios that matter most to you. Open an issue to help shape our roadmap.
What's new since 2.9.36-experimental
Key and value cache type
IWinMLModelTextGenerationConfiguration::SetKeyValueCacheTypechooses how a session from published weights holds its attention key and value cache, in 16-bit floating point (WINML_TEXT_GENERATION_KEY_VALUE_CACHE_TYPE_FLOAT16) or as 8-bit integers with static per-channel scales (INT8).DEFAULTis 16-bit floating point. Windows ML never picks a type from free memory.- An INT8 cache holds about twice the context in the same memory. On a 32 GB GPU, Qwen3.8-27B with NVFP4 weights and a DFlash 2 drafter reaches its full 262,144-token context with an INT8 cache, against 139,264 tokens in 16-bit floating point. In our tests INT8 matched 16-bit accuracy on HumanEval+ and MBPP+, 438 against 437 of 539 problems, decoded at the same speed or up to 7% faster, and prefilled 4 to 16% slower.
- INT8 needs the model's per-channel scales in a
kv_cache_scales.safetensorsfile beside the model. Without itCreateSessionfails withHRESULT_FROM_WIN32(ERROR_NOT_SUPPORTED). GetKeyValueCacheTypeon the configuration reports the type it requested, andIWinMLTextGenerationSession::GetKeyValueCacheTypereports the type a session holds. In Python, callset_key_value_cache_typewith aTextGenerationKeyValueCacheType, and readTextGenerationSession.key_value_cache_type.- A composition holds its stages' own state and a llama.cpp session holds 16-bit floating point, so
CreateSessionfails withHRESULT_FROM_WIN32(ERROR_NOT_SUPPORTED)for a type they can't hold.
Context that fits in GPU memory
- A session from published weights whose logical capacity doesn't fit in dedicated GPU memory now fails
CreateSessionwithHRESULT_FROM_WIN32(ERROR_NOT_ENOUGH_MEMORY), instead of spilling its cache into shared memory. The error message, also available throughGetErrorInfo, names the memory in use and free, the largest capacity that fits and, for 16-bit floating point, whether INT8 would fit. The configuration stays reusable, so an app can retry with a smaller capacity or INT8. - Qwen3.8-27B uses less GPU memory with a DFlash 2 drafter. Its output head shares the drafter's weights, which saves 0.67 GiB, and the drafter keeps a 4,096-position window of its cache instead of the whole context. Output is unchanged.
Faster speculative decoding
- Speculative decoding with a DFlash 2 drafter spends less host time in each round. On an RTX 5090, Qwen3.8-27B with NVFP4 weights decoded 33% faster with greedy decoding, the geometric mean over six prompts.
- A sampled speculative round with a top-k from 1 to 31 now reads each verified position's top candidates from the GPU instead of its full logits, with the same distribution. In the same test, sampled decoding was 66% faster with identical text. Greedy decoding, other top-k values, penalties and constraints work as before.
Fixes
- Prompts longer than about 104,000 tokens no longer fail generation. Prefill padded its last batch with zeros, which produced invalid logits at long positions.
WinMLServer
--kv-cache-type default|float16|int8, orkeyValueCacheTypein a recipe, sets the cache type of a--composition modellane. The catalog at/v1/modelspublishes it aswinml.key_value_cache_type, and aWINMLSERVER_KV_CACHE_TYPEline records the requested and held types at launch. A context whose cache doesn't fit in dedicated memory fails the launch withERROR_NOT_ENOUGH_MEMORY.--compiled-program-directorykeeps compiled CGC programs on disk, so the backend doesn't hold the weights it loads in shared memory and a longer context fits in dedicated memory.- When a request omits
parallel_tool_calls, a model lane that supports parallel tool calls now enables them, so a Qwen3.8 response with several tool calls parses instead of failing. - A failed launch step prints its error message, and a
WINMLSERVER_MEMORY_SPILLline reports a load that spilled into shared memory.
Packages
WinMLTokenizer.dllis built from the same onnxruntime-extensions tokenizer sources as in 2.9.36, now kept in the Windows ML repository. The package addsThirdPartyNotices.WinMLTokenizer.txtfor the components compiled into it.- ONNX Runtime stays at 1.30.1. llama.cpp stays at build 11847 with backend API 3, so CUDA backends built for 2.9.36 load unchanged.
Breaking changes
None. The new methods are added at the end of IWinMLModelTextGenerationConfiguration and IWinMLTextGenerationSession, so code built for 2.9.36 runs unchanged. A session from published weights whose capacity doesn't fit in dedicated GPU memory now fails CreateSession, as described above, where 2.9.36 ran it with part of its cache in shared memory.
Known issues
- INT8 needs a
kv_cache_scales.safetensorsfile with the model's per-channel scales, and published checkpoints, such as NVIDIA's Qwen3.8-27B, don't include one. - The Task API doesn't report how much dedicated and shared GPU memory a session uses.
- Without
SetLogicalCapacity, a session from published weights holds at most 8,192 tokens, even when the model supports more, and a prompt that doesn't fit fails withHRESULT_FROM_WIN32(ERROR_INSUFFICIENT_BUFFER). CallSetLogicalCapacityafterSetModelfor longer prompts. - With the WebGPU execution provider, a process can stop responding at exit after it prints its result, most often while the GPU is busy with other work.
- With some GPU providers, a sample that exits normally can lose output that's redirected to a file or a pipe. Run it in a console to see its output.
- A WebGPU session deletes the TensorRT for RTX execution provider's compiled context for the same model, so the next TensorRT for RTX session compiles it again.
- With a CUDA backend built by
Tools/llamainstead of thewindowsml-llama-cudawheel, Python warns that the CUDA backend is unavailable, though the model still runs on CUDA. - On a PC without an NVIDIA GPU, a sample built with
-LlamaCudaprints oneggml_cuda_init: failed to initialize CUDAline, then runs on the CPU.
The Runtime samples README lists more.
Packages
| NuGet package | Version | Contents |
|---|---|---|
| Microsoft.Windows.AI.MachineLearning | 2.9.42-experimental | The Runtime and Task APIs, with ONNX Runtime 1.30.1 |
| Microsoft.Windows.AI.MachineLearning.LibLlama.Core | 0.7.0.113-experimental | llama.cpp with the CPU backend |
| Microsoft.Windows.AI.MachineLearning.LibLlama.Cuda.win-x64 | 0.7.0.113-experimental | The opt-in CUDA backend for x64 |
| Microsoft.Windows.AI.MachineLearning.LibLlama.Cuda.win-arm64 | 0.7.0.113-experimental | The opt-in CUDA backend for Arm64 |
| Microsoft.Windows.AI.MachineLearning.LibLlama | 0.7.0.113-experimental | The Core with both CUDA backends |
| PyPI project | Version | Contents |
|---|---|---|
| windowsml | 2.9.42a0 | Windows ML for Python. The with-ort extra adds ONNX Runtime, and the llama extra adds the llama.cpp CPU backend |
| windowsml-llama-core | 2.9.42a0 | The llama.cpp CPU backend |
| windowsml-llama-cuda | 2.9.42a0 | The opt-in CUDA backend |
| windowsml-llama | 2.9.42a0 | windowsml with the CPU and CUDA backends |
What's changed
- Pin the samples to Windows ML 2.9.42 experimental by Nick Eubanks (@nieubank) in #46