Skip to content

Foundry Local v2.4.0

Choose a tag to compare

View all tags
@baijumeswani baijumeswani released this 10 Nov 21:19
v2.4.0
3901b30
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

🚀 Foundry Local v2.4.0

Foundry Local v2.4.0 adds ranking and typed-decision models alongside generation, transcription, and embeddings. You can now score candidates and make structured decisions on-device with the same model lifecycle, local catalog, and SDK session patterns you already use. These features are available in C++, C#, JavaScript, Python, and Rust.

This release also adds AMD GPU execution-provider support, updates ONNX Runtime GenAI to 0.19.0, verifies model downloads, and adds request cancellation in every language SDK.

🆕 What's New

🏆 Ranking and typed-decision models

  • Two new local-catalog tasks: text-ranking and typed-decision. Register a multi-component package (for example, CLM for ranking or KEV for decisions) in the local catalog, load it, and run it locally on CPU or CUDA.
  • New HTTP endpoints: POST /catalogs/local/v1/rank and POST /catalogs/local/v1/systemone. Pass the registered model ID or alias in model.
  • New RankingSession and DecisionSession APIs in every SDK: C++, C#, JavaScript, Python, and Rust. They return typed results and use the same Session/Request/Response ABI as the other sessions. You can reuse one session across many requests.
  • Ranking results include each candidate's id and zero-based index, plus a probability relative to the submitted candidate set. Ties keep input order.
  • Decision results support choice criteria, which keep your IDs and order, and ordered score criteria, which return a probability-weighted score that can be fractional.
  • Built-in safeguards:
    • Packages are validated, including manifests, tasks, providers, and paths. Paths cannot escape the package root, including through symlinks.
    • Concurrent cold loads of the same model are deduplicated.
    • A memory budget is reserved before loading. Configure it with NonGenerativeMemoryBudgetBytes.
    • NonGenerativeReadinessConcurrency limits concurrent initialization per provider.
    • Unload and unregister wait for active requests to finish.
with RankingSession(ranking_model) as session:
    ranking = session.rank(RankingRequest(
        context={"weather": "rain"},
        question="Which activity is better?",
        answers=["picnic", "museum"],
    ))
    print(ranking.ranked)

See the package guide and each SDK's README for complete examples.

🟥 AMD GPU execution provider

  • Foundry Local now recognizes AMDGPUExecutionProvider and can select it for models that support it on AMD GPUs.

🛑 Request cancellation in every SDK

  • C#: CancellationToken works on typed, compatibility, and streaming calls.
  • JavaScript: Pass an AbortSignal.
  • Python: Pass a cancel_event.
  • Rust: Dropping a stream or future cancels the request.

Canceled requests roll back their history, and you can reuse them.

⚡ Improvements

  • ONNX Runtime GenAI 0.19.0, with refreshed CUDA EP bundles for Windows and Linux on x64 and ARM64.
  • Verified model downloads: each 2 MiB range is checked with Azure Storage MD5 and pinned to the blob ETag. Corrupt chunks are downloaded again on resume. Incomplete downloads are never treated as cached.
  • Better automatic model selection: variants that need an unavailable execution provider are skipped when choosing defaults. They remain listed, and you can still select them explicitly.
  • Embedding token usage is now reported in native responses, OpenAI JSON, and the HTTP endpoint.
  • Better tool calling: the tool protocol is inferred from the model's rendered chat template.
  • Model context length and capabilities (for example, reasoning and tool-calling) are restored from catalog tags when structured metadata is missing.
  • Image inputs are documented for the JSON APIs and native SDKs.
  • WinML updated to 2.5.96.

🐛 Fixed

  • Qwen2.5 required and named tool choices no longer return HTTP 500.
  • image_url vision inputs now work in Chat Completions.
  • The Microsoft.AI.Foundry.Local.Runtime NuGet package no longer overrides the consuming project's RuntimeIdentifier.
  • Fixed a startup race in the shared host.

⚠️ Behavior Changes

  • The telemetry opt-out environment variable is now ORT_DISABLE_TELEMETRY=1, matching ONNX Runtime. It replaces ORT_TELEMETRY_DISABLED. Update scripts and deployments that set the old variable.

Note

CUDA and cuDNN runtimes must be installed separately; Foundry Local does not download them.

  • Windows/Linux x64: CUDA 12 and compatible cuDNN 9 runtimes.
  • Windows/Linux ARM64: CUDA 13 and compatible cuDNN 9 runtimes.

On Windows, add the runtime DLL directories to PATH. On Linux, configure the system library search path or LD_LIBRARY_PATH.
Configure library search paths before starting your application.
Windows x64 applications must additionally call AddDllDirectory for the CUDA runtime directory before CUDA execution-provider registration; PATH alone is insufficient for the GenAI CUDA bridge. This additional step is not required for the shipped Windows ARM64 bridge.

📥 Source Code

GitHub automatically includes source archives with the v2.4.0 release:

📚 Resources
📖 Documentation https://aka.ms/foundry-local-docs
🐙 GitHub https://github.com/microsoft/Foundry-Local
🧪 Samples https://github.com/microsoft/Foundry-Local/tree/main/samples
💬 Discord https://aka.ms/foundry-local-discord

💙 Feedback

File issues at https://github.com/microsoft/Foundry-Local/issues and include the SDK version, platform, and relevant diagnostics.

Full Changelog: v2.3.1...v2.4.0

Assets 2
Loading