Skip to content

26.11

Latest
Latest

Choose a tag to compare

View all tags
@dholt dholt released this 13 Nov 18:58
· 4 commits to master since this release
26.11
c8858fa
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

DeepOps 26.11 release notes

DeepOps 26.11 is tagged from master; there is no separate release branch. It
covers the changes since 26.09.

Highlights

Model workflows (all opt-in)

  • Shared model cache. Download an approved Hugging Face model once onto
    existing NFS storage with the open-source hf CLI; Slurm jobs and Kubernetes
    pods read pinned snapshots read-only. Includes an offline cache validator (#1421, #1440, #1441).
    Nothing is downloaded or mounted automatically and it is not part of either
    cluster playbook. See docs/model-workflows/shared-cache.md.
  • Single-node vLLM quickstart. Serve a cached model on one GPU (loopback
    only) and check a completion with a standard-library validator. vLLM is
    pinned to 0.32.0, which includes upstream security fixes (#1422, #1439, #1442). It needs the
    Python 3.12 development headers and a C compiler on the serving host. See
    docs/model-workflows/vllm-quickstart.md.
  • Fine-tuning readiness validator. scripts/validation/validate_finetune.py
    answers ready / not ready / bad input for one named Slurm node, a fully cached
    model and a digest-pinned container image. It is a preparation check only; it
    never reports that training worked (#1423). See docs/model-workflows/finetune-readiness.md.
  • Skills. New operator skills under skills/ (and a root AGENTS.md index),
    written for AI agents and equally usable by people, cover model-workflow
    routing, cache management, single-node serving, fine-tuning readiness, and
    optional KAI Scheduler / Run:ai deployment (#1421, #1423, #1425).

Monitoring: kube-prometheus-stack 92.14.0

The monitoring stack moves from 88.5.4 (shipped in 26.09) to 92.14.0 (#1437, upgrade guide in
#1438). Existing installs need action; see Upgrade notes. Also in this
release: deploy_monitoring.sh now exits non-zero when the Helm release is not
deployed (#1430), and the GPU Nodes dashboard resolves GPU and host exporters
independently (#1431).

Component version refresh (#1428)

Slurm 26.05.5, GPU Operator v26.7.2, NVIDIA device plugin and GPU feature
discovery 0.20.1, Network Operator 26.7.0, MIG Manager 0.15.1, Prometheus 3.15.0,
Grafana 13.2.4, Alertmanager 0.34.1, Node Exporter 1.12.1, container registry
3.1.2 and registry proxy 0.7.0. Offline (air-gap) documentation and artifacts
were aligned with the new pins. The NVIDIA driver role is pinned to v2.4.0
(#1427; see Upgrade notes).

Job-scoped Slurm epilog cleanup (#1433)

The Slurm epilog now cleans up per job instead of per user, so it no longer
disturbs another running job of the same user on the node. It is on by default
and has a documented switch back; see Upgrade notes.

Run:ai and KAI Scheduler (optional, off by default)

A new runai_mode setting (#1425, #1432):

  • kai: installs the open-source KAI Scheduler for fair-share queues, gang
    scheduling and fractional-GPU pods. Supported.
  • platform: installs the full Run:ai control plane and cluster (charts 3.0.20).
    Experimental. Live-tested on a single node.
  • The control plane can be reached through a LoadBalancer address or through the
    opt-in Gateway API. Both were tested with MetalLB on a single node; reaching
    the address from another machine was not part of that test.
  • scripts/validation/validate_runai.py checks an installation.

See docs/k8s-cluster/runai.md.

Other changes

Slurm and Pyxis

  • Opt-in Lua support for building Slurm (slurm_build_lua, default off) (#1417).
  • Opting out of the Pyxis user-namespace override now removes the override
    DeepOps itself added, leaving other site settings alone (fixes #1412) (#1420).
  • Epilog tests now run in CI, skipped fixtures are rejected, and the epilog's
    mount-point decoder handles escaped spaces and tabs correctly (#1419).
  • Documentation for removing a manual or single-node Slurm install (#1414).

Kubernetes and GPUs

  • verify_gpu.sh stops with a clear message when no node advertises GPU
    capacity (#1418), and now prints the logs of every GPU test pod, not only the
    first (#1443).
  • nvidia-gpu-tests: optional gpu_test_dcgm_fail_on_error (default off) makes
    the play fail when the DCGM diagnostic fails (#1436).
  • The Kubernetes setup guide no longer tells you to enable the Kubernetes
    Dashboard, which the pinned Kubespray version does not install (#1424).
  • The Run:ai documentation now refers to gpu_operator_chart_version instead of
    a hard-coded version (#1429).

Setup and virtual lab

  • scripts/setup.sh now installs your config/requirements.yml; a misspelled
    file name had made it skip that step silently (#1435).
  • The Vagrant virtual lab defaults to Ubuntu 22.04 (#1415).

Upgrade notes

MIG: the hooks file is now opt-in (#1416)

mig_manager_hooks now defaults to empty and DeepOps no longer creates a hooks
file, so the MIG playbook runs without custom hooks. If your inventory still sets
/etc/nvidia-mig-manager/hooks.yaml, provide that file on every target or clear
the variable; an explicitly configured missing or invalid file is still an error.
Drain GPU workloads before changing MIG configuration, or provide site hooks that
stop GPU clients. See docs/slurm-cluster/nvml.md.

NVIDIA driver role v2.4.0 (#1427)

On Ubuntu, when nvidia_driver_branch is not set, the package fallback now
selects driver branch 580 (it was 515). Set nvidia_driver_branch explicitly
to keep another branch. On EL9 and newer, the role installs the kernel-modules
package matching the newest installed kernel-core; make sure your mirror carries
it, or set nvidia_driver_ensure_kernel_modules: false and manage the modules
yourself. DeepOps installs this role from Ansible Galaxy.

Slurm: epilog cleanup no longer touches your other running jobs

When a Slurm job ended, the DeepOps epilog cleaned up everything the user had
on that node, which could kill or delete files from another job of the same user
that was still running (or had just started) there. As of this release the
cleanup is job-scoped: the epilog checks the node itself, leaves other jobs'
processes and files alone, and only does the full per-user cleanup when the
user's last job on the node ends. This is the new default
(slurm_epilog_cleanup_scope: job) and needs no action on most sites; it
requires cgroup-based process tracking, which DeepOps already configures.

To keep the previous behaviour for this release, set
slurm_epilog_cleanup_scope: user in config/group_vars/slurm-cluster.yml and
re-run the Slurm playbook. That setting is deprecated and will be removed in
27.01. Also fixed: the epilog's SSH-access cleanup matched account names as
patterns and could remove the access entry of a different user whose name
contained the job user's name. See docs/slurm-cluster/epilog-job-scoped-cleanup.md.
(Reported in #1407.)

Monitoring: kube-prometheus-stack 92.14.0 (action needed for existing installs)

The monitoring stack moves from kube-prometheus-stack 88.5.4 (the 26.09
default) to 92.14.0 (Prometheus Operator v0.94.2) and now requires Kubernetes 1.25 or newer.
Control-plane scraping stays authenticated but now uses the Prometheus pod's
own automatically rotated service-account identity instead of a stored,
non-expiring one. Component mirror variables (PROMETHEUS_OPER_REPO, PROMETHEUS_REPO, ALERTMANAGER_REPO,
GRAFANA_REPO, GRAFANA_WATCHER_REPO) now map to the chart's effective image
keys. Fresh installs need no extra steps. (#1437)

Existing monitoring installs must update the CRDs first. Re-running the
monitoring installer does not upgrade an existing release (it skips a healthy
one and stops on an unhealthy one), and Helm does not upgrade CRDs. Follow
docs/k8s-cluster/monitoring-upgrade-92.md (written for 88.6.5; it applies
unchanged to 88.5.4, which ships the same CRDs and Prometheus Operator v0.93.1): back up your Helm values and monitoring resources privately, apply the 10 CRDs bundled
with chart 92.14.0 using server-side apply, confirm every CRD reports operator
version 0.94.2, then run the Helm upgrade with your reviewed saved values. Do
not use the installer's -d delete option to upgrade or roll back. Expect a
one-time restart of Prometheus, Alertmanager and the operator, and plan a
maintenance window. Sites with an external scraper (such as Grafana Alloy) or
custom Grafana, image-mirror or RBAC values should review the guide's
"Security and scheduling changes" section before upgrading.

Not tested in this release

  • MIG configuration with MIG Manager 0.15.1 was not tested on real MIG hardware
    in 26.11; the MIG playbook was checked for syntax only.
  • Network Operator 26.7.0 secondary-network traffic and RDMA were not tested
    in 26.11.
  • The Operator-managed driver 580.126.20 was not tested in 26.11; GPU testing
    used a driver preinstalled on the host.

Acknowledgments

Thank you to @100milliongold and @100-JM for their contributions to DeepOps, including the GPU, Slurm, platform-compatibility, and deployment fixes, and for the detailed reports and testing that helped identify these issues.

Thank you also to @0z5a (#1443), @MaxFreedomPollard (#1435) and @saqibkh (#1436), who contributed fixes in this release.

Contributors

saqibkh, 100milliongold, and 3 other contributors
Assets 2
Loading