26.09

DeepOps 26.09 is a stabilization release. It makes the Slurm and Pyxis paths
work with a custom Slurm prefix and current Ubuntu releases, replaces two
third-party dependencies with DeepOps-owned roles, adds two operator skills,
and updates the main GPU software dependencies.

Acknowledgments

Thank you to @100milliongold and @100-JM for their contributions to DeepOps,
including the GPU, Slurm, platform-compatibility, and deployment fixes, and for
the detailed reports and testing that helped identify these issues. Thanks also
to @MaxFreedomPollard for strengthening the Slurm epilog test suite.

Slurm and Pyxis

• Fixed Pyxis builds and runtime with a custom slurm_install_prefix: the
  SPANK header include path resolves (#1388), and the Slurm client tools are
  on the slurmd PATH so Enroot hooks can find scontrol (#1411).
• Enroot works on Ubuntu 23.10 and later, where AppArmor restricts unprivileged
  user namespaces by default. The Pyxis role installs a profile scoped to
  enroot-nsenter; a host-wide opt-out is available but off by default
  (#1392).
• Updated Slurm to 26.05.3 (#1380).
• Replaced the third-party Slurm exporter with a DeepOps-owned implementation
  that keeps the existing dashboard metrics, builds from a current base image,
  restarts correctly when Docker restarts, and mounts the Slurm client tools
  onto the container PATH (#1363, #1365, #1366, #1395).
• Login-node GPU hiding uses ssh.service on Debian-family systems (#1389),
  and nvidia-smi tasks in the playbooks run outside the SSH cgroup so they
  survive the login GPU guard (#1397).
• Exclusive-job detection in run-parts.sh works with a custom prefix and
  recent Slurm releases (#1391).
• The GPU power/clock helpers batch their queries and parallelize writes
  (#1393).
• The Slurm epilog cleanup is scoped to the job that ended, spares users listed
  in localusers.backup, and leaves mount points alone even when their names
  contain glob characters or a trailing newline (#1404, #1408). The test suite
  covers the cleanup exit status and these pathname cases (#1410).
• The PAM module directory is derived from the target architecture (#1398).

GPU and platform compatibility

• DGX systems transition from legacy nv_peer_mem to the in-tree
  nvidia_peermem module when it is available (#1390).
• The DCGM role no longer replaces the DCGM package that DGX OS already ships
  (#1394).
• The MIG manager package and the MOFED ISO filename are selected from the
  target architecture (#1399, #1400).
• The Enroot RPM distribution tag is derived from the version being installed,
  and the Enroot/Pyxis example versions were refreshed (#1401, #1402).
• NHC carries the sshd process-title colon fix to all Ubuntu releases
  (#1406).

Provisioning and setup

• Replaced the unmaintained external MAAS role with a smaller DeepOps-owned
  controller role (#1367).
• Added a provision-with-maas operator skill covering tags, dynamic inventory,
  deployment, validation, reruns, and cleanup (#1377).
• Fixed scripts/setup.sh for non-interactive callers (#1374).
• Made the memory fact locale-safe and consistently integer-valued (#1364).

Kubernetes and component updates

• GPU Operator v26.7.0 (#1381).
• NVIDIA Kubernetes device plugin and GPU Feature Discovery 0.20.0
  (#1382, #1383).
• kube-prometheus-stack 88.5.4 (#1385).
• NVIDIA Network Operator 26.4.1 (#1387).
• Spack v1.2.2 (#1384).

Agent and operator workflows

• Added a deploy-airgapped skill with repository and image mirroring,
  deployment, validation, and recovery steps (#1376).
• Slurm and Kubernetes validators report stable per-node state and GPU detail
  in their JSON output (#1378).
• Replaced the stale container image mirroring guide with a current Skopeo-based
  workflow and a manual fallback (#1375).
• Removed tracked Python bytecode and added repository ignore rules (#1379).

Retired components

• Removed the archived ELK logging playbook, Open OnDemand installer,
  Singularity wrapper, and vestigial OpenShift client role (#1368, #1370,
  #1371, #1372).
• Vendored the small first-party utility roles used by DeepOps so installs no
  longer depend on mutable remote role state (#1369, #1373).

Validation notes

• Release QA ran on a fresh Ubuntu 24.04 Slurm deployment with GPU scheduling,
  login isolation, monitoring/exporter health, a Pyxis/Enroot GPU container,
  and exclusive/nonexclusive prolog selection.
• Kubernetes/GPU Operator and MAAS 3.7 on Ubuntu 24.04 are included in the live
  release gates. The DeepOps 23.08 upgrade baseline is checked on Ubuntu 22.04;
  direct upgrades stop at the documented Kubespray Calico minimum-version
  boundary.
• Multi-node scheduling remains a known validation gap and is carried into the
  next release. DGX OS package convergence is covered by source and CI checks
  but still needs a dedicated live DGX-system proof.