2.9.1: Confidential computing and security enhancements

2.9.1 Release Contributors (PR Count Order)

Total PRs counted in this release: 73

• Zhihong Zhang — 28 PRs
• Isaac Yang — 22 PRs
• Holger Roth — 10 PRs
• Chester Chen — 8 PRs
• Yuan-Ting Hsieh (謝沅廷) — 2 PRs
• Peter Cnudde — 2 PRs
• EmmaHu — 1 PR

---

🎉 Welcome First-Time Contributors!

• EmmaHu — 1 PR

---

Feature Highlights

NVIDIA FLARE 2.9.1 updates the confidential computing deployment stack to follow the NVIDIA confidential computing reference architecture. Confidential computing remains a technical preview and entirely optional. Changes that affect all deployments include stricter control-request authorization, checkpoint path and Dashboard restrictions, removal of the legacy push-based tensor-streaming API, updated security dependencies, configuration-format deprecations, and reliability fixes.

• Confidential computing (Technical Preview): one provisioning interface for bare-metal confidential VMs (CVMs), Confidential Containers (CoCo) and Azure; modes can be mixed in one federation.
  • CVM Builder builds and approves a generic confidential VM once per CPU/GPU profile, then seals each application's image, code, configuration and secrets into a separate encrypted vault, so application updates reuse the approved VM. It supports Intel TDX and AMD SEV-SNP, CPU-only or with NVIDIA confidential GPUs. Vault keys are released only after attestation passes, and workloads are re-checked periodically and quarantined on failure.
  • Confidential Containers (CoCo) run FLARE participants as Kubernetes workloads inside confidential VMs, including mixed TDX and SEV-SNP participants. Peers verify each other against the Trustee attestation service.
  • Azure: refreshed confidential VM and container guides; the single Azure Container Instances (ACI) server deployment now enforces the administrator-approved confidential computing policy and verifies a signed external startup kit.
• Security hardening: client administration and job-control requests are checked against the authenticated caller, route and job; multiprocess sub-workers enforce the site's component authorization; PTFileModelPersistor checkpoint paths must stay within the application directory; the Dashboard rejects client props; jobs that carry their own executable content in launcher specifications (BYOC) are blocked in confidential computing; and certifi and uv minimum versions are raised following the eMudhra/emSign CA incident.
• Tensor streaming removal: the legacy push-based nvflare.app_opt.tensor_stream API and Recipe.enable_tensor_streaming() are removed; use the pull-based Tensor Downloader for large-model transfer.
• Deprecation: HOCON (.conf) and OmegaConf YAML (.yml, .yaml) runtime and job configuration are deprecated and will be removed in 2.10.0; use JSON. Provisioning files such as project.yml are not affected.

Existing confidential computing and ACI deployments must be reprovisioned; there is no in-place upgrade. See the migration guide before upgrading.

See the full 2.9.1 release note: https://nvflare.readthedocs.io/en/2.9.1/release_notes/flare_291.html

---

What's Changed

• Added Confidential Containers (CoCo) provisioning and attestation, Intel TDX and CPU-only runtimes, mixed TDX/SEV-SNP workflows, and a simplified workflow for existing clusters and Trustee by @IsaacYangSLA in #5284, #5344, #5351, #5362, #5439, #5557, #5635
• Added the two-stage CVM Builder for Intel TDX and AMD SEV-SNP, unified confidential computing provisioning, vault capacity preflight, and simplified deployment workflows and diagnostics by @nvidianz and @emma-hu123 in #5312, #5372, #5411, #5582, #5660, #5667, #5673
• Hardened confidential computing attestation and GPU quarantine (terminal quarantine after revocation, quarantine ordering, vault revocation, cold-boot clock skew, portable SEV-SNP guest policy, Trustee trust-vector and cross-vault policy fixes) by @nvidianz and @IsaacYangSLA in #5353, #5409, #5410, #5412, #5414, #5415, #5420, #5438, #5457, #5463, #5464, #5477
• Hardened CVM build input validation and acceptance checks, including checkpoint path confinement by @nvidianz and @IsaacYangSLA in #5427, #5441, #5458, #5459, #5473
• Blocked launcher-defined BYOC jobs in confidential computing and enforced approved ACI policies and signed external startup kits by @nvidianz and @IsaacYangSLA in #5417, #5744
• Bound client and server job-control requests to the authenticated caller, route and job, and scoped server-parent commands by job and caller role by @pcnudde and @nvidianz in #5709, #5710, #5746
• Fixed multiprocess sub-worker initialization and enforced site component authorization for runtime builds by @YuanTingHsieh in #5644
• Blocked client props in Dashboard client creation and PATCH requests by @IsaacYangSLA in #5298
• Required patched certifi and uv versions by @nvidianz in #5727
• Removed the legacy push-based tensor-streaming API and example by @chesterxgchen in #5718
• Fixed admin connector inspection reply decoding, async gRPC startup URLs, FedAvg partial checkpoints, recipe client log streaming in server jobs, job-log uploads after failed deletion, and subprocess session startup by @nvidianz in #5394, #5567, #5633, #5663, #5665, #5675
• Published NVFlare Agent Skills with a shared catalog bundle and refreshed NVSkills benchmark provenance by @holgerroth in #5260, #5287, #5303, #5304, #5306, #5307, #5308, #5309, #5378
• Refreshed confidential computing documentation: architecture overview, unified deployment guides, Azure overview, CoCo executor limitation, and peer-attestation guarantees by @nvidianz, @chesterxgchen, and @IsaacYangSLA in #5440, #5504, #5527, #5530, #5693, #5694
• Added the 2.9.1 release notes and migration guide, updated Hello World requirements, fixed the documentation build, Read the Docs build timeout, website version dropdown, and release-branch documentation links by @chesterxgchen, @holgerroth, and @IsaacYangSLA in #5507, #5510, #5628, #5723, #5758, #5762, #5764
• Backported CI stability, dependency, Docker and workflow fixes, and stabilized tests by @YuanTingHsieh and @IsaacYangSLA in #5388, #5403, #5482, #5484, #5711

Full Changelog: 2.9.0...2.9.1