Skip to content

Version 2.9.42-experimental

Pre-release
Pre-release

Choose a tag to compare

@nieubank nieubank released this 20 Nov 17:10
c780909

Windows ML 2.9.42 Experimental updates 2.9.36 for long contexts. Text generation from published weights can now hold its attention key and value cache as 8-bit integers, which fits about twice the context in the same GPU memory, and a session whose context doesn't fit in dedicated GPU memory now fails when it's created, with an error that names the largest context that fits. Speculative decoding with a DFlash 2 drafter is faster, and prompts longer than about 104,000 tokens no longer fail.

These APIs are still an early look and will change as they mature. The samples and documentation for this release are on the main branch.

Get started

We'd love your feedback on the API shape, the developer experience, and the models and scenarios that matter most to you. Open an issue to help shape our roadmap.

What's new since 2.9.36-experimental

Key and value cache type

  • IWinMLModelTextGenerationConfiguration::SetKeyValueCacheType chooses how a session from published weights holds its attention key and value cache, in 16-bit floating point (WINML_TEXT_GENERATION_KEY_VALUE_CACHE_TYPE_FLOAT16) or as 8-bit integers with static per-channel scales (INT8). DEFAULT is 16-bit floating point. Windows ML never picks a type from free memory.
  • An INT8 cache holds about twice the context in the same memory. On a 32 GB GPU, Qwen3.8-27B with NVFP4 weights and a DFlash 2 drafter reaches its full 262,144-token context with an INT8 cache, against 139,264 tokens in 16-bit floating point. In our tests INT8 matched 16-bit accuracy on HumanEval+ and MBPP+, 438 against 437 of 539 problems, decoded at the same speed or up to 7% faster, and prefilled 4 to 16% slower.
  • INT8 needs the model's per-channel scales in a kv_cache_scales.safetensors file beside the model. Without it CreateSession fails with HRESULT_FROM_WIN32(ERROR_NOT_SUPPORTED).
  • GetKeyValueCacheType on the configuration reports the type it requested, and IWinMLTextGenerationSession::GetKeyValueCacheType reports the type a session holds. In Python, call set_key_value_cache_type with a TextGenerationKeyValueCacheType, and read TextGenerationSession.key_value_cache_type.
  • A composition holds its stages' own state and a llama.cpp session holds 16-bit floating point, so CreateSession fails with HRESULT_FROM_WIN32(ERROR_NOT_SUPPORTED) for a type they can't hold.

Context that fits in GPU memory

  • A session from published weights whose logical capacity doesn't fit in dedicated GPU memory now fails CreateSession with HRESULT_FROM_WIN32(ERROR_NOT_ENOUGH_MEMORY), instead of spilling its cache into shared memory. The error message, also available through GetErrorInfo, names the memory in use and free, the largest capacity that fits and, for 16-bit floating point, whether INT8 would fit. The configuration stays reusable, so an app can retry with a smaller capacity or INT8.
  • Qwen3.8-27B uses less GPU memory with a DFlash 2 drafter. Its output head shares the drafter's weights, which saves 0.67 GiB, and the drafter keeps a 4,096-position window of its cache instead of the whole context. Output is unchanged.

Faster speculative decoding

  • Speculative decoding with a DFlash 2 drafter spends less host time in each round. On an RTX 5090, Qwen3.8-27B with NVFP4 weights decoded 33% faster with greedy decoding, the geometric mean over six prompts.
  • A sampled speculative round with a top-k from 1 to 31 now reads each verified position's top candidates from the GPU instead of its full logits, with the same distribution. In the same test, sampled decoding was 66% faster with identical text. Greedy decoding, other top-k values, penalties and constraints work as before.

Fixes

  • Prompts longer than about 104,000 tokens no longer fail generation. Prefill padded its last batch with zeros, which produced invalid logits at long positions.

WinMLServer

  • --kv-cache-type default|float16|int8, or keyValueCacheType in a recipe, sets the cache type of a --composition model lane. The catalog at /v1/models publishes it as winml.key_value_cache_type, and a WINMLSERVER_KV_CACHE_TYPE line records the requested and held types at launch. A context whose cache doesn't fit in dedicated memory fails the launch with ERROR_NOT_ENOUGH_MEMORY.
  • --compiled-program-directory keeps compiled CGC programs on disk, so the backend doesn't hold the weights it loads in shared memory and a longer context fits in dedicated memory.
  • When a request omits parallel_tool_calls, a model lane that supports parallel tool calls now enables them, so a Qwen3.8 response with several tool calls parses instead of failing.
  • A failed launch step prints its error message, and a WINMLSERVER_MEMORY_SPILL line reports a load that spilled into shared memory.

Packages

  • WinMLTokenizer.dll is built from the same onnxruntime-extensions tokenizer sources as in 2.9.36, now kept in the Windows ML repository. The package adds ThirdPartyNotices.WinMLTokenizer.txt for the components compiled into it.
  • ONNX Runtime stays at 1.30.1. llama.cpp stays at build 11847 with backend API 3, so CUDA backends built for 2.9.36 load unchanged.

Breaking changes

None. The new methods are added at the end of IWinMLModelTextGenerationConfiguration and IWinMLTextGenerationSession, so code built for 2.9.36 runs unchanged. A session from published weights whose capacity doesn't fit in dedicated GPU memory now fails CreateSession, as described above, where 2.9.36 ran it with part of its cache in shared memory.

Known issues

  • INT8 needs a kv_cache_scales.safetensors file with the model's per-channel scales, and published checkpoints, such as NVIDIA's Qwen3.8-27B, don't include one.
  • The Task API doesn't report how much dedicated and shared GPU memory a session uses.
  • Without SetLogicalCapacity, a session from published weights holds at most 8,192 tokens, even when the model supports more, and a prompt that doesn't fit fails with HRESULT_FROM_WIN32(ERROR_INSUFFICIENT_BUFFER). Call SetLogicalCapacity after SetModel for longer prompts.
  • With the WebGPU execution provider, a process can stop responding at exit after it prints its result, most often while the GPU is busy with other work.
  • With some GPU providers, a sample that exits normally can lose output that's redirected to a file or a pipe. Run it in a console to see its output.
  • A WebGPU session deletes the TensorRT for RTX execution provider's compiled context for the same model, so the next TensorRT for RTX session compiles it again.
  • With a CUDA backend built by Tools/llama instead of the windowsml-llama-cuda wheel, Python warns that the CUDA backend is unavailable, though the model still runs on CUDA.
  • On a PC without an NVIDIA GPU, a sample built with -LlamaCuda prints one ggml_cuda_init: failed to initialize CUDA line, then runs on the CPU.

The Runtime samples README lists more.

Packages

NuGet package Version Contents
Microsoft.Windows.AI.MachineLearning 2.9.42-experimental The Runtime and Task APIs, with ONNX Runtime 1.30.1
Microsoft.Windows.AI.MachineLearning.LibLlama.Core 0.7.0.113-experimental llama.cpp with the CPU backend
Microsoft.Windows.AI.MachineLearning.LibLlama.Cuda.win-x64 0.7.0.113-experimental The opt-in CUDA backend for x64
Microsoft.Windows.AI.MachineLearning.LibLlama.Cuda.win-arm64 0.7.0.113-experimental The opt-in CUDA backend for Arm64
Microsoft.Windows.AI.MachineLearning.LibLlama 0.7.0.113-experimental The Core with both CUDA backends
PyPI project Version Contents
windowsml 2.9.42a0 Windows ML for Python. The with-ort extra adds ONNX Runtime, and the llama extra adds the llama.cpp CPU backend
windowsml-llama-core 2.9.42a0 The llama.cpp CPU backend
windowsml-llama-cuda 2.9.42a0 The opt-in CUDA backend
windowsml-llama 2.9.42a0 windowsml with the CPU and CUDA backends

What's changed

Full changelog