2.9.0rc1: Major features and various bug fixes

What's Changed

• Bump jupyterlab from 4.5.6 to 4.5.7 in /research/fsi-fraud-detection by @dependabot[bot] in #4502
• Clear retained training result after accept events by @pcnudde in #4520
• Refresh stale integration tests by @YuanTingHsieh in #4505
• Align CLI Python support with 3.10-3.14 by @pcnudde in #4521
• [doc] Add LLM-friendly docs outputs by @pcnudde in #4519
• Add run-scoped tensor offload temp cleanup by @pcnudde in #4522
• Add Auto-FL literature events and local compute budget search by @holgerroth in #4523
• Bump next from 15.5.14 to 15.5.15 in /examples/tutorials/self-paced-training/part-5_federated_learning_applications_in_industries/chapter-11_federated_learning_in_healthcare_lifescience/11.3_medical_devices/code/visualization/frontend/analytics-dashboard by @dependabot[bot] in #4436
• Bump mako from 1.3.10 to 1.3.11 in /research/fsi-fraud-detection by @dependabot[bot] in #4453
• Bump astro from 5.16.3 to 6.1.6 in /web by @dependabot[bot] in #4463
• Bump jupyter-server from 2.17.0 to 2.18.0 in /research/fsi-fraud-detection by @dependabot[bot] in #4524
• Bump postcss from 8.5.6 to 8.5.10 in /examples/tutorials/self-paced-training/part-5_federated_learning_applications_in_industries/chapter-11_federated_learning_in_healthcare_lifescience/11.3_medical_devices/code/visualization/frontend/analytics-dashboard by @dependabot[bot] in #4478
• Avoid k8s pod name collisions across sites by @pcnudde in #4544
• Add cleanup parameter to recipe Run.get_result() (default True) by @nvshaxie in #4541
• Bump postcss from 8.5.6 to 8.5.14 in /web by @dependabot[bot] in #4489
• Clarify simulator and API doc signposts [skip ci] by @YuanTingHsieh in #4546
• Throttle Auto-FL literature review triggers by @holgerroth in #4543
• Validate k8s deploy namespace by @YuanTingHsieh in #4545
• Warn on deploy prepare storage rewrites by @YuanTingHsieh in #4548
• Clarify Flower example version compatibility by @holgerroth in #4557
• Bump mistune from 3.2.0 to 3.2.1 in /research/fsi-fraud-detection by @dependabot[bot] in #4531
• Bump mako from 1.3.11 to 1.3.12 in /research/fsi-fraud-detection by @dependabot[bot] in #4536
• Bump gitpython from 3.1.47 to 3.1.50 in /research/fsi-fraud-detection by @dependabot[bot] in #4565
• Bump urllib3 from 2.6.3 to 2.7.0 in /research/fsi-fraud-detection by @dependabot[bot] in #4570
• Bump torch from 2.6.0 to 2.8.0 in /examples/advanced/medgemma by @dependabot[bot] in #4413
• Harden job metadata path containment by @pcnudde in #4573
• Add adaption example to Auto_FL by @ZiyueXu77 in #4560
• [main] Fix StorageException after every job from empty SystemLogStreamer context by @nvidianz in #4584
• Cherry pick pr 4527 4526 4552 4562 4575 4571 4579 by @IsaacYangSLA in #4592
• Align black target-version with supported Python range by @cyyever in #4566
• Cherry-pick PR #4558 to main by @nvidianz in #4580
• [main] Cherry pick PR 4586 by @nvidianz in #4594
• Port selected 2.8 fixes to main by @YuanTingHsieh in #4595
• Fix Auto-FL research license headers by @pcnudde in #4599
• Make K8s server service name configurable by @pcnudde in #4598
• Add premerge license check by @pcnudde in #4600
• Bind auth tokens to runtime origins by @pcnudde in #4597
• Port PRs 4588-4591 to main by @holgerroth in #4603
• Fix aborted job download race by @pcnudde in #4604
• Mock more classes/functions by @IsaacYangSLA in #4608
• Use uv for pre-merge unit test installs by @pcnudde in #4612
• Normalize job not found admin errors by @pcnudde in #4621
• Address comments (from cherry-pick PR #4618) by @IsaacYangSLA in #4619
• Fix PT persistor bootstrap from empty checkpoint by @holgerroth in #4610
• Fix admin CLI login rejection handling by @pcnudde in #4625
• Only run pre-merge build-wheel in one environment by @IsaacYangSLA in #4620
• CPU-only torch backend for github pre-merge workflow by @IsaacYangSLA in #4615
• Allow CellPipe stream aliases for auth binding by @pcnudde in #4628
• Map running job delete to JOB_NOT_DONE by @YuanTingHsieh in #4632
• Fix aborted job status publication race by @YuanTingHsieh in #4633
• Enable Blossom CI after PR merge by @pcnudde in #4636
• Revert post-merge Blossom CI trigger by @pcnudde in #4637
• Adding no-BYOC support for Flower integration by @d0uwe in #4514
• Add FedBPT Job API entry point by @holgerroth in #4634
• Port integration CI layout updates from 2.8 by @YuanTingHsieh in #4631
• Add 2.8 release notes and NVFlare CLI tutorial by @chesterxgchen in #4643
• Harden NVFlare Docker image base by @pcnudde in #4638
• Bump idna from 3.11 to 3.15 in /research/fsi-fraud-detection by @dependabot[bot] in #4640
• Add K8s image pull secret support by @pcnudde in #4647
• Bump fast-uri from 3.1.0 to 3.1.2 in /web by @dependabot[bot] in #4564
• Cherry-pick [2.8]: Add more details on helm chart and k8s deployment guides [skip… by @IsaacYangSLA in https://github.com//pull/4651
• Cherry-pick [2.8]: Fix session.list_job_components() error (#4622) by @IsaacYangSLA in #4650
• Port numpy trainer logging fix from 2.8 by @YuanTingHsieh in #4653
• Port Docker deployment docs update from 2.8 by @YuanTingHsieh in #4654
• Cherry-pick CC runtime workspace fix to main by @nvidianz in #4660
• Remove LGPL frontend dependencies by @chesterxgchen in #4667
• Replace scikit-survival (GPL-3.0) with download_data.py in KM examples by @chesterxgchen in #4666
• Add tensor disk offload to 2.8.0 release notes by @chesterxgchen in #4669
• Fix NVFlare CLI docs by @chesterxgchen in #4671
• fix fd leak in GenerateDataFromFile on send error by @cyyever in #4577
• Add new examples to tutorial catalog by @chesterxgchen in #4673
• Fix fstat fd leak in GenerateDataFromFile and add success-path test by @chesterxgchen in #4677
• Validate launcher_spec image field is a string by @chesterxgchen in #4678
• Restore devops examples under examples/devops by @pcnudde in #4682
• Update multi-study documentation links by @chesterxgchen in #4686
• modernize syntax via pyupgrade --py310-plus by @cyyever in #4578
• Don't use kagglehub==1.0.1 by @IsaacYangSLA in #4692
• cert: make project_profile.yaml server.admin_port optional by @chesterxgchen in #4700
• Fix CLI docs: provisioning paths, admin_port, K8s status, ephemeral_storage by @chesterxgchen in #4695
• Fix monitoring pair duration after missing end event by @chesterxgchen in #4697
• Improve security by loading classes based on BYOC flag and class lists by @IsaacYangSLA in #4684
• Handle corner cases by @IsaacYangSLA in #4704
• Simplify Docker runtime image by @pcnudde in #4702
• [main] Cherry-pick #4708: Fix finished download ref retry handling by @chesterxgchen in #4712
• [main] Cherry-pick #4710: Validate client API launcher resend settings by @chesterxgchen in #4713
• [main] Avoid duplicate timeout validation by @chesterxgchen in #4717
• Update docs for large-model streaming reliability by @chesterxgchen in #4715
• [main] Fix FOBS externalizer shared-state race by @nvidianz in #4718
• Harden frame length validation in network drivers by @cyyever in #4687
• Avoid name collision in Dockerfile by @IsaacYangSLA in #4726
• [main] Port 2.8 large-model client API fixes by @chesterxgchen in #4729
• [main] Port 2.8 dashboard container launch fixes by @chesterxgchen in #4730
• Align component allow-list files with 2.8 by @IsaacYangSLA in #4733
• Harden built-in artifact writer paths by @pcnudde in #4745
• Document Kubernetes API CA validation by @pcnudde in #4742
• [main] Migrate built-in job configs to fully qualified class paths by @IsaacYangSLA in #4746
• [main] Cherry-pick from [2.8]: Lower a log level on one specific message (#4674) by @IsaacYangSLA in #4731
• Port curated allow-list updates to main by @chesterxgchen in #4748
• Add Q2 webinar recording by @holgerroth in #4750
• Guard blob receive size before allocation by @fallintoplace in #4735
• Bump next from 15.5.15 to 15.5.18 in /examples/tutorials/self-paced-training/part-5_federated_learning_applications_in_industries/chapter-11_federated_learning_in_healthcare_lifescience/11.3_medical_devices/code/visualization/frontend/analytics-dashboard by @dependabot[bot] in #4572
• Bump astro from 6.1.6 to 6.1.10 in /web by @dependabot[bot] in #4587
• Bump devalue from 5.6.4 to 5.8.1 in /web by @dependabot[bot] in #4609
• Document and scripts for OpenShift deployment by @IsaacYangSLA in #4679
• fix(edge): create Android SGD optimizer once and correct proxy port docs by @AlexanderSanin in #4723
• Align research README and contributor template with current layouts by @holgerroth in #4757
• [main] Cherry-pick FLARE-2976 MPM return-code fix by @nvidianz in #4766
• [main] Cherry-pick BYOC clone metadata preservation by @nvidianz in #4765
• [main] Cherry-pick FLARE-2977 BYOC decomposer gating by @nvidianz in #4751
• [main] Cherry-pick FOBS externalizer constructor-args fix by @nvidianz in #4752
• [main] Cherry-pick CC class allow-list updates by @nvidianz in #4763
• Add tensor disk offload memory chart to release notes by @chesterxgchen in #4771
• Suppress empty error log stream warning by @nvidianz in #4768
• Fix web deploy npm install by @holgerroth in #4764
• Use Node 22 for web deploy by @holgerroth in #4776
• Guard code-execution sinks out of the provisioned class_allow_list by @pcnudde in #4747
• Accept path alias in recipe model configs by @chesterxgchen in #4777
• feat(job_config): add fail_fast parameter to FedJob for development mode by @AlexanderSanin in #4688
• Align FLModel metrics across FL workflows by @chesterxgchen in #4778
• Upgrade NeMo PEFT Nemotron example by @holgerroth in #4773
• Handle K8s pending resource timeouts by @IsaacYangSLA in #4770
• Drop duplicate task submissions after cleanup by @chesterxgchen in #4772
• Bind CellNet mTLS endpoint identity by @nvidianz in #4759
• Add Kubernetes ConfigMap/Secret staging for NVFlare deploy by @IsaacYangSLA in #4760
• fix(dxo_filter): correct error message in DXOFilter.init validation by @AlexanderSanin in #4775
• Fix simulator job cell without server config by @nvidianz in #4781
• Fix admin CellNet server identity mapping by @pcnudde in #4786
• Use ConfigMap/Secret staging in multicloud Kubernetes example by @pcnudde in #4782
• Add FedTaskRecipe for one-round tasks by @holgerroth in #4593
• Remove legacy NeMo PEFT package by @holgerroth in #4783
• Add recipe metrics artifacts writer by @chesterxgchen in #4784
• Add tensor disk offload support for FedOpt by @pcnudde in #4794
• Reliable Messaging by @nvidianz in #4785
• feat(flower): run superlink in job run directory by @Zare2001 in #4762
• fixes nightly build: add exact CSE simulator allow list by @pcnudde in #4800
• Support study-specific K8s job pod templates by @IsaacYangSLA in #4804
• Add progress-aware streaming wait policy by @chesterxgchen in #4736
• Make recipe --export/--export-dir import safe by @chesterxgchen in #4808
• fix(analytix): correct typo in convert_data_type that silently drops TORCH_TB→WANDB metrics by @AlexanderSanin in #4803
• Fix: Handle missing resource module on Windows by @mrityunjay2627 in #4807
• Deprecate legacy training APIs by @holgerroth in #4641
• Add tensor disk offload support for Scaffold by @pcnudde in #4799
• Add Codecov coverage upload by @YuanTingHsieh in #4657
• Reject cyclic error results before conversion by @YuanTingHsieh in #4658
• Add NVFLARE agent skills and CLI by @chesterxgchen in #4798
• Name CellPipe cells hierarchically for mTLS identity resolution by @nvidianz in #4801
• Filter analysis files from released agent skills by @chesterxgchen in #4810
• Bump torch from 2.5.0 to 2.12.0 in /examples/tutorials/self-paced-training/part-5_federated_learning_applications_in_industries/chapter-12_federated_learning_in_financial_services/12.1_end_to_end_federated_fraud_detection by @dependabot[bot] in #4796
• Make FSI fraud uv lock a reference snapshot by @holgerroth in #4816
• Verify received client deploy app bytes by @pcnudde in #4817
• FLARE-2986 Use signer identity for client app authz by @pcnudde in #4819
• Bind START_JOB metadata to deployed metadata by @pcnudde in #4818
• Compute BYOC locally before allow-list bypass by @pcnudde in #4820
• Cache CIFAR-10 data for integration tests by @pcnudde in #4828
• Remove hard-coded simulator IPC password by @nvidianz in #4822
• Cherry-pick selected 2.8 fixes by @YuanTingHsieh in #4829
• Update NeMo SFT example for Nemotron 3 by @holgerroth in #4779
• Create output directory for NeMo SFT predictions by @holgerroth in #4831
• Fix Docker job image editable install by @YuanTingHsieh in #4830
• Remove legacy NeMo prompt learning example by @holgerroth in #4832
• Stabilize reverse result upload stall test by @YuanTingHsieh in #4833
• Enforce BYOC for job-selected launcher content by @pcnudde in #4839
• Add per-site recipe config helper by @IsaacYangSLA in #4792
• Fix Scaffold tensor controls for BF16/CUDA by @pcnudde in #4840
• Allow admin certificate chains by @pcnudde in #4827
• Provision default component policy for SimEnv by @pcnudde in #4841
• Constrain broken kagglesdk versions by @YuanTingHsieh in #4847
• Make reverse upload stall test deterministic by @YuanTingHsieh in #4842
• Allow edge result reports without cookies by @fallintoplace in #4824
• Add configurable root CA validity by @pcnudde in #4848
• Add recipe metadata helper by @YuanTingHsieh in #4849
• Use resolved swarm client defaults by @fallintoplace in #4826
• Fix feature election test collection by @pcnudde in #4859
• Add multicloud E2E verification by @YuanTingHsieh in #4850
• Export flare.get_task_name from nvflare.client by @YuanTingHsieh in #4854
• docs: document recipe api by @YuanTingHsieh in #4843
• Write Client API config files with owner-only permissions by @YuanTingHsieh in #4855
• Avoid spurious stream ACK teardown errors by @nvidianz in #4844
• Harden stream teardown notifications and peer retry windows by @nvidianz in #4861
• Add torchrun CPU Client API integration test by @YuanTingHsieh in #4834
• Raise unit test coverage above 60% by @pcnudde in #4860
• Add Client API execution modes design docs and F3 aggregate transfer outcome by @YuanTingHsieh in #4853
• Add Client API Cell protocol vocabulary (client/cell/defs.py) by @YuanTingHsieh in #4856
• Download CIFAR-10 from a mirror first in integration test prep by @YuanTingHsieh in #4864
• Fix CellPipe topology FQCN naming by @nvidianz in #4835
• NVFLARE agent skills: PyTorch/Lightning conversion, orient, diagnose — with lint, eval, packaging, and security hardening by @chesterxgchen in #4837
• Add recipe-k8s example: submit a Recipe job to clients in two k8s clusters by @IsaacYangSLA in #4871
• Add per-site client tracking and harden recipe client placement by @YuanTingHsieh in #4862
• Expose Swarm and Cyclic parameters by @nvkevlu in #4868
• Add cleanup for staged Kubernetes resources by @IsaacYangSLA in #4877
• Add ClientAPIExecutor with frozen backend spec and working in_process backend by @YuanTingHsieh in #4857
• Reject symlinks in signed folders by @pcnudde in #4879
• [Main] Fix admin leaf validation for custom roles and legacy certificates by @nvidianz in #4880
• Add study_runtime.yaml v2 per-study runtime config for job launchers by @pcnudde in #4872
• Validate WEIGHT_DIFF recipe configurations and add DP example by @nvkevlu in #4874
• Fix flaky job_runner completion-loop tests by scoping the time.sleep patch by @pcnudde in #4882
• [main] Skip auth filter for cellnet protocol-level bye messages by @nvshaxie in #4863
• Fix global model discovery for job downloads by @nvkevlu in #4884
• Persist study registry to workspace root for read-only staged local dirs by @pcnudde in #4886
• Add recipe secret parameter safeguards by @IsaacYangSLA in #4883
• Add NVSkills CI request workflow by @chesterxgchen in #4892
• Fix Docker job image build: remove stale skill_manifest.py COPY by @nvshaxie in #4893
• Replace public Recipe.job usage with purpose-built Recipe APIs by @nvkevlu in #4873
• CellPipe follow-ups: review cleanups, VIA_ROOT routing fall-through, route-loop guard by @nvidianz in #4867
• [Main] ccwf swarm: fix SwarmServerConfig.min_clients default + start_task_timeout by @nvidianz in #4878
• Add F3 payload layer: receiver-confirmed completion, receiver budgets, awaitable transfer facade by @YuanTingHsieh in #4865
• Fix FedJob export of parent helper modules by @nvkevlu in #4885
• Fix job log tail truncation by @nvkevlu in #4889
• Fix abort for registered STARTING jobs by @pcnudde in #4904
• Fix per-version averaging in BuffModelManager. by @kminhta in #4900
• Tighten resource spec validation by @pcnudde in #4909
• Preserve abort requests during client job launch by @pcnudde in #4910
• Refactor deploy tooling into Docker and Kubernetes backends by @pcnudde in #4911
• Add comprehensive job statistics reporting by @IsaacYangSLA in #4907
• Add class allow-list migration controls (cherry-pick #4888) by @nvidianz in #4905
• Make Recipe job storage private by @nvkevlu in #4903
• Add Auto-FL agent skill by @holgerroth in #4780
• Add Auto-FL paper reference by @holgerroth in #4913
• Implement per-site configuration for FedAvg and FedEval recipes by @nvkevlu in #4902
• Run both Docker example clients in containers by @YuanTingHsieh in #4912
• [Auto-FL] Diagnose outdated nvflare when trial result root is unresolved by @holgerroth in #4917
• Add nvflare-fed-stats agent skill and dataset-target classification in agent inspect by @chesterxgchen in #4890
• Deliver job process bootstrap credentials via environment instead of command line by @pcnudde in #4929
• Redact client auth token from log messages by @pcnudde in #4932
• Refresh skills for NVSkills CI bootstrap by @chesterxgchen in #4931
• Redact remaining client auth token log sites by @pcnudde in #4933
• [Auto-FL] Add literature-batch and diversification eval cases by @holgerroth in #4918
• [Auto-FL] Clarify key_metric direction and warn on loss-like metrics by @holgerroth in #4939
• Support scalar-like analytics values by @YuanTingHsieh in #4683
• Validate edge device ID headers by @fallintoplace in #4825
• Bump torch to 2.12.0 and torchvision to 0.27.0 in /examples/advanced/medgemma by @dependabot[bot] in #4809
• [FLARE-2672] Prevent timestamp-based federated event loss by @YuanTingHsieh in #4916
• Refine NVSkills validation findings by @chesterxgchen in #4940
• [Auto-FL] Document run-to-run variance and aggregation-order nondeterminism by @holgerroth in #4928
• Add automatic Lightning SCAFFOLD support by @holgerroth in #4838
• [Auto-FL] Declare canonical selection metric and fix cross-site fallback extraction by @holgerroth in #4927
• [Auto-FL] Expose explicit budget/baseline/abandoned accounting in campaign state by @holgerroth in #4926
• [Auto-FL] Improve nvflare-autofl discoverability and routing by @holgerroth in #4925
• [Auto-FL] Refresh campaign state when mutable settings change before action gating by @holgerroth in #4945
• [Auto-FL] Reduce cross-site fallback scores with a defined per-site mean by @holgerroth in #4944
• [Auto-FL] Improve skill quality guidance by @holgerroth in #4946
• [Auto-FL] Remove synthetic data injection by @holgerroth in #4950
• [Auto-FL] Restrict campaigns to score maximization by @holgerroth in #4951
• Add Slurm job launcher by @pcnudde in #4930
• Add integration coverage for recipe examples by @YuanTingHsieh in #4915
• Include site names in Slurm job names by @pcnudde in #4952
• Add shared-file transport driver for cell communication by @pcnudde in #4947
• Collab API: pythonic collaborative FL programming by @nvidianz in #4901
• [Auto-FL] Add report skill by @holgerroth in #4845
• Add HuggingFace Trainer Client API by @chesterxgchen in #4948
• Add the external_process execution mode (backend + trainer engine) by @YuanTingHsieh in #4906
• Fix agent inspection and NVFLARE skill routing by @chesterxgchen in #4955
• Redact headers from unknown Cell reply warnings by @pcnudde in #4968
• Add simple Collab split learning example by @holgerroth in #4958
• Add client-local require_signed_jobs policy by @pcnudde in #4956
• Clarify unreleased example version guidance by @holgerroth in #4969
• Remove stale Torch dependencies from framework-neutral examples by @YuanTingHsieh in #4970
• Make Slurm CLI timeouts configurable by @pcnudde in #4961
• [Auto-FL] Improve report skill quality guidance by @holgerroth in #4957
• Reject shared-file transport in container deploy prepare by @pcnudde in #4972
• Document shared-file restriction for container deploys by @pcnudde in #4977
• Reorganize Collab examples by difficulty by @holgerroth in #4960
• Remove deprecated HUB support by @YuanTingHsieh in #4971
• Add automatic FedProx support to PyTorch Lightning by @holgerroth in #4895
• Fix explicit client topology in secure XGBoost examples by @holgerroth in #4980
• Fix client topology in Collab examples by @holgerroth in #4978
• Fix in-process Client API framework decomposer registration by @YuanTingHsieh in #4981
• Fix Swarm external-process result relay by @YuanTingHsieh in #4982
• Add launcher-owned multi-node Slurm training by @pcnudde in #4962
• Optimize F3 tensor download throughput: pipelining, prefetch, and TCP_NODELAY by @nvidianz in #4973
• Add FedCE research recipe support by @nvkevlu in #4922
• Fix Slurm shared-file container mount by @pcnudde in #4987
• [FLARE-3030] Prevent secret leaks in CLI errors by @pcnudde in #4979
• [Auto-FL] Harden trial process cleanup by @holgerroth in #4984
• Fix FedJob subdirectory sibling imports by @nvkevlu in #4959
• Add Hugging Face Trainer conversion skill and routing by @chesterxgchen in #4967
• Fix Collab call identity contracts by @nvidianz in #4988
• Stabilize external process backend shutdown test by @YuanTingHsieh in #4992
• [Auto-FL] Guard against semantic metric drift by @holgerroth in #4998
• Add MetricsArtifactWriter to default component allowlist by @YuanTingHsieh in #4991
• [doc] Document Slurm worker image workflow by @pcnudde in #4996
• [FLARE-3068] Allow Slurm studies to override Python path by @pcnudde in #4997
• Distinguish Slurm infrastructure failures by @pcnudde in #4986
• Enable tensor disk offload for swarm learning by @YuanTingHsieh in #4965
• [Auto-FL] Clarify metric repair restart workflow by @holgerroth in #4999
• [Auto-FL] Harden final report outcomes by @holgerroth in #5002
• Synchronize F3 streaming parameters by @nvidianz in #4995
• Fix external trainer cleanup on abort by @YuanTingHsieh in #4994
• Split Client API tests across deployment modes by @YuanTingHsieh in #4990
• Clarify live log persistence message by @YuanTingHsieh in #4993
• Fix Collab execution in POC and simulation by @nvidianz in #5003
• Fix concise simulator training logs by @YuanTingHsieh in #5001
• Document custom type packaging across job apps by @YuanTingHsieh in #4914
• Fix obsolete POC setup command in tutorial setup_poc.ipynb by @nvshaxie in #5009
• Handle FedCE client participation edge cases by @ZiyueXu77 in #5014
• Raise default streaming blob size to 4 GiB by @pcnudde in #5015
• Fail jobs when result publication fails by @pcnudde in #5010
• Add random forest example to CI by @YuanTingHsieh in #5000
• Atomically persist PyTorch checkpoints by @pcnudde in #5011
• Fix NVFlare versioning in Docker builds by @pcnudde in #5004
• Fix Recipe API usage in secure XGBoost examples by @ZiyueXu77 in #5012
• Simplify agent inspector architecture by @chesterxgchen in #5008
• Scan bundled NVFlare skills with public SkillSpector by @chesterxgchen in #4941
• Fix Agent Inspector denied job authority routing by @chesterxgchen in #5020
• Add Client API attach execution mode by @YuanTingHsieh in #4989
• Constrain XGBoost below 3.4 for column-split compatibility by @pcnudde in #5026
• Make job elapsed timezone-safe by @pcnudde in #5016
• Finish concrete Recipe entrypoint and documentation cleanup by @nvkevlu in #4983
• Bump svgo in /web by @dependabot[bot] in #4942
• Preserve successful Slurm results during heartbeat cleanup by @pcnudde in #5019
• [FLARE-3030] Address secret-redaction follow-up by @pcnudde in #5025
• Fix HCI session expiry during active result downloads by @pcnudde in #5017
• Add PyTorch async Collab example by @ZiyueXu77 in #4975
• Bump postcss from 8.5.14 to 8.5.25 in /web by @dependabot[bot] in #5023
• Bump fast-uri from 3.1.2 to 3.1.5 in /web by @dependabot[bot] in #5024
• Reject empty CollabRecipe job names by @YuanTingHsieh in #5034
• Fix Docker socket access on macOS by @YuanTingHsieh in #5029
• Fix admin result downloads available across retries by @nvkevlu in #5033
• Fix Slurm Apptainer PATH and GPU environment by @pcnudde in #5038
• Fix stale global failure count in FedStats by @chesterxgchen in #5022
• Fix F3 reassembly buffer sizing under load by @nvidianz in #5039
• Preserve reportable Slurm worker exit codes by @pcnudde in #5037
• Warn that recipe export arguments are reserved by @chesterxgchen in #5045
• Verify task result retries after stream failure by @nvidianz in #5046
• Expose key_metric_mode in FedAvg recipes by @chesterxgchen in #4953
• Restore Docker DinD compatibility by @YuanTingHsieh in #5043
• Document Client API Attach and migrate examples by @YuanTingHsieh in #5030
• Fix persistent trainer cleanup on abort by @YuanTingHsieh in #5027
• Fix K8s child failure propagation by @pcnudde in #5047
• Fix Lightning DDP result sending from nonzero ranks by @YuanTingHsieh in #5040
• Align Client API pass-through and Attach trust boundaries for 2.9 by @YuanTingHsieh in #5042
• Use standard Recipe export CLI in examples by @chesterxgchen in #5048
• Add SkillEvaluator Tier 1 gate for bundled skills by @chesterxgchen in #5054
• FLARE-3096: Downgrade harmless abort and fail logs to INFO by @nvidianz in #5051
• Stop warning on the negated key metric the warning itself recommends by @chesterxgchen in #4985
• Support bounded training factories in agent inspection by @chesterxgchen in #5065
• Fix shared-file Client API attach for Slurm containers by @pcnudde in #5049
• Clarify Swarm initial checkpoint path handling by @YuanTingHsieh in #5052
• Fix TASK_READY heartbeat expiry during materialization by @pcnudde in #5067
• Remove brittle Slurm version gate by @pcnudde in #5071
• Correct runtime and CLI documentation inconsistencies by @nvkevlu in #5066
• Bound permanent workspace archival retries by @pcnudde in #5069
• Allow Kubernetes launchers to read pod events by @IsaacYangSLA in #5078
• test: bypass MKLDNN in Codex ARM sandbox by @pcnudde in #5053
• Document Podman support for OpenShift image builds by @IsaacYangSLA in #5077
• Fix OpenShift CRC parent pod scheduling by @IsaacYangSLA in #5076
• Fix Docker version validation for development tags by @YuanTingHsieh in #5070
• Fix pass-through tensor download retention by @YuanTingHsieh in #5075
• Add concrete FedBPT recipe entrypoint by @nvkevlu in #5013
• Fail fast when streamed blobs exceed size limit by @nvidianz in #5068
• Strengthen conversion skills for FedAvg and data locality by @chesterxgchen in #5081
• Fix stale Recipe CLI documentation commands by @ZiyueXu77 in #5064
• Update request-nvskills-ci.yml by @chesterxgchen in #5087
• Add hello-world Agent Skills examples by @chesterxgchen in #5086
• Refactor metrics transport to use direct Cell by @YuanTingHsieh in #5080
• Fix workspace archive cleanup retries by @YuanTingHsieh in #5091
• Correct wheel build and package naming instructions by @nvkevlu in #5101
• Fix example checkout navigation after cloning NVFlare by @nvkevlu in #5100
• Document high-level API selection by @chesterxgchen in #5095
• Preserve Slurm worker failure reasons by @pcnudde in #5094
• Require BYOC for Slurm node commands by @pcnudde in #5104
• Wait for client terminal outcomes before finalizing jobs by @pcnudde in #5072
• Add federated LLM SFT example for Collab API by @ZiyueXu77 in #5005
• Add portable CPU and memory job resources by @pcnudde in #5092
• Document basic job execution workflow by @YuanTingHsieh in #5089
• Align job configuration defaults with runtime by @nvkevlu in #5105
• Document CSV dataset mounts for Docker jobs by @Roshni-Ranjita in #5074
• Fix external Client API receive guard by @YuanTingHsieh in #5090
• Update LLM HF example for multi-node Slurm by @YuanTingHsieh in #5031
• Restore LLM tutorial export directories by @ZiyueXu77 in #5102
• Route cross-client authentication through server by @YuanTingHsieh in #5096
• Fix Dashboard documentation and legacy page routing by @nvkevlu in #5106
• Clarify Hugging Face sharding support status by @chesterxgchen in #5113
• Require explicit NVSkills CI requests by @chesterxgchen in #5118
• Relocate agent skill eval suites by @chesterxgchen in #5110
• Harden job-controlled Docker launcher options by @YuanTingHsieh in #5108
• Remove legacy Client API execution stack by @YuanTingHsieh in #5044
• Preserve workspace sources during archival by @YuanTingHsieh in #5111
• Clean up clients after server job failure by @pcnudde in #5117
• Consolidate skill security validation under SkillEvaluator by @chesterxgchen in #5114
• Align remote job log configuration controls by @pcnudde in #5120
• Use loopback for SimEnv/PocEnv listeners by @pcnudde in #5122
• Preserve server transit across relay routes by @YuanTingHsieh in #5119
• Disable inbound admin services in Simulator by @nvkevlu in #5125
• Fix Docker example IPC configuration by @YuanTingHsieh in #5130
• Fix portable resource spec regressions by @pcnudde in #5133
• Add Client API attach acceptance coverage and docs by @YuanTingHsieh in #5129
• Correct documented F3 streaming defaults by @nvkevlu in #5103
• Harden confidential computing authorizers by @IsaacYangSLA in #5131
• Revert "Harden confidential computing authorizers" by @IsaacYangSLA in #5141
• Enable Kubernetes internal CellNet mTLS by @pcnudde in #5142
• Enable Slurm internal CellNet mTLS by @pcnudde in #5144
• Bind Collab calls to authenticated job participants by @nvidianz in #5079
• Surface client job log configuration failures by @chesterxgchen in #5138
• Surface resource manager admission errors by @pcnudde in #5143
• Expose key_metric_mode in legacy PT/TF job config wrappers by @holgerroth in #5140
• Validate job-controlled Docker launcher values by @YuanTingHsieh in #5135
• Keep resource manager exceptions site-local by @pcnudde in #5148
• Fix result transfer waiter polling race by @YuanTingHsieh in #5134
• Enable Docker internal CellNet mTLS by @YuanTingHsieh in #5147
• Fix Lightning best-model metric reporting by @chesterxgchen in #5145
• Clarify Hugging Face evaluation metric requirement by @chesterxgchen in #5152
• Clarify train_with_evaluation example comments by @chesterxgchen in #5151
• Harden resource admission failure handling by @pcnudde in #5149
• Materialize external-process payloads for client filters by @YuanTingHsieh in #5136
• Fix preflight command documentation by @nvkevlu in #5154
• Reject unsafe FedJob names by @nvidianz in #5063
• Fix safe default component allow-list gaps by @IsaacYangSLA in #5157
• Align Network Attach profile with internal mTLS defaults by @pcnudde in #5159
• [Auto-FL] Harden candidate accounting by @holgerroth in #5160
• Bind localhost TCP listeners to loopback by @YuanTingHsieh in #5146
• Fix Client API attach integration test isolation by @YuanTingHsieh in #5167
• Harden agent skill policies and metadata by @chesterxgchen in #5139
• Fix POC tutorial CLI commands by @nvkevlu in #5155
• Fix external_process listener with fixed listening_host by @pcnudde in #5166
• Add public PR milestone and Jira linking guidance by @holgerroth in #5172
• Harden Client API and Swarm abort cleanup by @YuanTingHsieh in #5097
• Handle dict metrics in SimpleIntimeModelSelector by @holgerroth in #5150
• Authenticate Cell messages with AES-GCM by @IsaacYangSLA in #5158
• Bound non-aggregator Swarm client job RSS by @YuanTingHsieh in #5168
• Harden PyTorch-family conversion skills and artifact resolution by @chesterxgchen in #5170
• Honor configured internal listener host by @nvidianz in #5163
• Avoid optional framework imports in recipe CLI metadata by @chesterxgchen in #5175
• Add realistic SMILES workloads to conversion examples by @chesterxgchen in #5176
• Drop late streamed replies without routing errors by @nvidianz in #5165
• Harden certless admin sessions and listener binding by @nvidianz in #5171
• Add NVFlare Day 2026 link by @holgerroth in #5178
• Remove stale package version fallback by @nvkevlu in #5153
• Bump monai from 1.5.2 to 1.6.0 in /research/condist-fl by @dependabot[bot] in #5007
• Bump transformers from 5.0.0 to 5.5.0 in /research/fed-bpt by @dependabot[bot] in #4921
• Bump torch from 2.8.0 to 2.13.0 in /examples/advanced/qwen3-vl by @dependabot[bot] in #4920
• Treat duplicate mandatory clients as one required site by @YuanTingHsieh in #5161
• Bump torch from 2.12.0 to 2.13.0 in /examples/advanced/medgemma by @dependabot[bot] in #4963
• Enable best-model selection for Swarm recipes by @chesterxgchen in #5162
• Clarify Swarm best-model metric documentation [doc] [skip-ci] by @chesterxgchen in #5180
• Fix external-process authentication race by @YuanTingHsieh in #5107
• Align agent skills with current specification by @chesterxgchen in #5181
• Upgrade CIFAR-10 SplitNN example to the Collab API by @ZiyueXu77 in #5050
• Add Q3 2026 webinar recording by @holgerroth in #5184
• [Auto-FL] Add native metric direction support by @holgerroth in #5169
• Allow client-to-client Collab calls through transit cells by @nvidianz in #5185
• Make blocking Collab group results re-iterable by @nvidianz in #5183
• Avoid per-stream error context scans by @nvidianz in #5190
• Fix external trainer settlement and receiver parsing by @YuanTingHsieh in #5179
• Add synchronous CIFAR-10 Collab examples by @ZiyueXu77 in #5036
• Migrate tutorials from deprecated job CLI by @nvkevlu in #5156
• [Auto-FL] Harden skill CI contracts by @holgerroth in #5182
• Clarify launcher default inheritance for server jobs by @pcnudde in #5193
• Pass NVFlare base version into Docker example builds by @YuanTingHsieh in #5196
• [Auto-FL] Report direction-adjusted literature improvement by @holgerroth in #5186

New Contributors

• @fallintoplace made their first contribution in #4735
• @AlexanderSanin made their first contribution in #4723
• @mrityunjay2627 made their first contribution in #4807
• @Roshni-Ranjita made their first contribution in #5074

Full Changelog: 2.8.0rc1...2.9.0rc1