Skip to content

Release v0.7.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 12 Oct 21:04
· 29 commits to main since this release
v0.7.0
This tag was signed with the committer’s verified signature.
ndipebot Ebot Ndip-Agbor
GPG key ID: 75363CC079EC9202
Verified
Learn about vigilant mode.
a0193ff
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

NVCRE v0.7.0

The NVIDIA Cluster Readiness Engine (NVCRE) is a Kubernetes controller that certifies GPU clusters before production use. It runs real training and communication workloads across topology-aware node groups, measures goodput and bandwidth, detects hardware failures, and reports every failed node with a reason. See the README for the full feature list.

Highlights

This release focuses on three new places to certify (Nscale NKS, x86 HGX B300, and OCI GB200), running one job of a chosen size with Unpinned placement, and telling a blocked run apart from a running one.

Nscale NKS, HGX B300, and OCI GB200 are certifiable. Nscale NKS clusters, detected from the nscale:// providerID, claim GPUs and RDMA NICs through DRA ResourceClaimTemplates, run an InfiniBand NCCL profile that leaves NCCL_NET to the HPC-X plugin so SHARP stays on, and use topology.nks.nscale.com/accelerator-domain where other platforms use the GFD clique label; nvcrectl cluster info counts GPUs from ResourceSlices on nodes that advertise no nvidia.com/gpu (#380, #419). The x86 HGX B300 (NVIDIA-B300-SXM6-AC) gets eight-GPU, eight-NIC defaults, an on-prem override that tolerates the GPU taint and requests RDMA devices, and a GPUArchitectureDefaults Warning whenever an architecture silently falls back to four GPUs (#464). OCI GB200 no longer requests nvidia.com/mlnxnics by default, renders the RoCEv2 GID index its queue pairs need, and reaches host-exposed RDMA devices through rdmaDeviceAccess: hostPath (#443, #467); OCI GB300 ships an NCCL topology file so each GPU pairs with its own rail (#458). GCP H100 reads the GKE network names from the nodes and picks the NCCL and training images and the tcpxo-daemon from the installed TCPXO plugin version, validated on an AICR A3 Mega cluster at plugin v1.0.15 and v1.0.16 with no overrides; an explicit image still replaces the chosen one (#437, #445).

One job of exactly the size you asked for (#454). orchestration.placement: Unpinned on a WorkloadRun, or on a hand-written Workflow, runs a single job of numNodes nodes and leaves the rest of the target untouched, with the scheduler placing the pods inside the target. The size is honored or the run fails; it is never clamped or auto-selected. In both modes the target selector now travels onto the pods as a required node affinity, and a group retry is no longer lost to a stale cache read. A Certification has no placement option on purpose: its verdict covers every node it targets.

A blocked run is no longer reported as running (#371). A Job whose pods the scheduler has rejected for longer than spec.schedulingStallGraceSeconds (default five minutes) reports WorkloadSchedulingBlocked with the scheduler's own diagnosis, the Workflow mirrors it as JobSchedulingBlocked, and the report prints a Blocked: line. The blocked interval pauses timeoutPerJob and the stall budgets, so a run no longer times out against hardware that never ran. startupStallTimeoutSeconds on the Certification sets the first-step budget the training entries previously hardcoded at 1200 s (#394).

Controller correctness

  • MPI workers request SYS_CHROOT alongside IPC_LOCK, and their readiness probe opens a real loopback SSH session. On CRI-O every SSH session died preauth while the listener stayed up and mpirun failed with the generic ORTE banner; an admission policy that pins the capability list needs to allow the new one (#466)
  • Checkpointed training runs complete on clusters that enforce RBAC: the manager role gains update on PersistentVolumeClaims, which setting the Job as the checkpoint PVC's owner requires (#463)
  • nvcre_job_status reports failed for a Job that timed out, is built at scrape time from the informer cache like the other status gauges, and the timeout write keeps InProgress, Succeeded, and Failed exclusive (#448)

Measurement

  • BandwidthMeasurement.status.transport records which NCCL network carried the collective, IB or Socket, and the report and --results-file show it per category, group, and clique. Record-only; no verdict reads it (#444)

Supply chain and release integrity

  • The release verify gate refuses legacy signature formats instead of letting cosign fall back to them, attest.yml refuses a cosign_version below v3.1.3 (GHSA-fx35-mq7g-6g98), and the image and binaries are built with Go 1.27.2 and golang.org/x/net v0.60.0 (#369, #460, #459)

Breaking changes

  • OCI GB200 defaults to mlnxPerNode: 0, and mlnxPerNode: 0 now omits the nvidia.com/mlnxnics request instead of rendering a count of zero. A site that runs the SR-IOV device plugin sets its own count. An unchanged manifest on OCI GB200 renders no NIC request and no k8s.v1.cni.cncf.io/networks annotation (#443)
  • openstack:// providerIDs resolve to onprem, and Nscale detection requires nscale://. A cluster that v0.6.0 detected as Nscale through the nscale.com/rdmashare allocatable now gets the on-prem overrides (#380)
  • WorkloadRun MPI ranks receive the base NCCL environment and spec.env through mpirun -x, and NCCL_DEBUG_SUBSYS=NET,INIT is no longer set by default; set it in spec.env to keep the filtered log. An unchanged MPI WorkloadRun runs with NCCL settings its ranks never saw before (#380)
  • Workload pods carry the target selector and the discovered nvidia.com/gpu.product values as a required node affinity. A node relabeled after discovery no longer qualifies and its pod stays Pending (#454)
  • training/nemotron5-8b, nemotron5-56b, and communication/nccl-loopback render timeoutPerJob (24h and 1h) where they rendered none, and nvcrectl certification render names Workflows with the options suffix the controller uses. Scripts that matched the shorter rendered names need the new ones (#467)
  • The chart must move with the controller. The PVC update grant lives in manager-role.yaml, so a v0.7.0 manager under the v0.6.0 role fails checkpointed runs the way v0.6.0 did (#463)

Thanks to @asivanadi0, @kaynetu, @mkjpryor, @mlsorensen, @ndipebot, @ntheanh201, and @wussh.

Install nvcrectl

The installer is signed. Download it with its bundle, verify, then run — piping it into a shell executes it before anything has checked it.

REL=https://github.com/NVIDIA/cluster-readiness-engine/releases/download/v0.7.0

curl -fsSLO "${REL}/installer"
curl -fsSLO "${REL}/installer.sigstore.json"

cosign verify-blob-attestation \
  --bundle installer.sigstore.json \
  --type https://slsa.dev/provenance/v1 \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.7.0" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  installer

bash installer -v v0.7.0
Without cosign 
curl -fsSL https://github.com/NVIDIA/cluster-readiness-engine/releases/download/v0.7.0/installer | bash -s -- -v v0.7.0

This runs the installer before anything verifies it. TLS proves you reached GitHub; it proves nothing about the asset if the release itself was tampered with. Use it only where installing cosign first is genuinely impractical.

# newest stable release; `releases/latest` never resolves to a pre-release
curl -fsSL https://github.com/NVIDIA/cluster-readiness-engine/releases/latest/download/installer | bash

Helm Chart

This release's chart is published at digest sha256:6326d30c3443d27d62d039882dd080f5d73cab5b690db61dd791a8e72644d8b3. The signature covers that digest; the version tag is mutable, so verify and install the same digest rather than verifying a digest and installing a tag.

CHART=ghcr.io/nvidia/cluster-readiness-engine
DIGEST=sha256:6326d30c3443d27d62d039882dd080f5d73cab5b690db61dd791a8e72644d8b3

# 1. Verify the chart. Requires cosign; the package is public, so no login is needed.
cosign verify \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.7.0" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "${CHART}@${DIGEST}"

# 2. Pull that exact digest, then install the file you just verified.
helm pull "oci://${CHART}@${DIGEST}"
helm install nvcre ./cluster-readiness-engine@${DIGEST/:/-}.tgz \
  --namespace nvcre --create-namespace

Installing with --version v0.7.0 instead resolves the tag at install time, which is not provably the digest you verified.

helm install --verify does not check this signature. That flag looks for a PGP .prov file, a mechanism unrelated to Sigstore, and we publish no .prov files.

Controller image

A different artifact from the chart, verified separately. The release built and signed these exact digests:

Subject Digest
index (multi-platform) sha256:4eaff9d047e7826aa212e6f1be21014189f8661e2ab464998427cce35e0aa0de
linux/amd64 sha256:43d02389881201a888ed43726555079f53c9579b7d85008c5c91f54aefa9e1f6
linux/arm64 sha256:fda0f176aa4e5ceec0c61e92500159dd1584504e1c22867dbc9bb49c00beea69

The index carries the signature and the SLSA provenance; each platform digest carries a signature and its own CycloneDX SBOM. Pin the index digest rather than the tag:

IMAGE=ghcr.io/nvidia/cluster-readiness-engine/manager
INDEX=sha256:4eaff9d047e7826aa212e6f1be21014189f8661e2ab464998427cce35e0aa0de

cosign verify \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.7.0" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "${IMAGE}@${INDEX}"

nvcrectl CLI

Download the binary for your platform from the assets below and place it on your $PATH.

Platform Binary
Linux amd64 nvcrectl-linux-amd64
Linux arm64 nvcrectl-linux-arm64
macOS amd64 nvcrectl-darwin-amd64
macOS arm64 (Apple Silicon) nvcrectl-darwin-arm64

Verify your download

Every asset ships with a Sigstore bundle, <asset>.sigstore.json, carrying a SLSA Build Provenance v1 statement. Verify the artifact itself, not a checksum served from the same place as the artifact:

REL=https://github.com/NVIDIA/cluster-readiness-engine/releases/download/v0.7.0
ASSET=nvcrectl-linux-amd64   # or installer, or any published asset

# The bundle is a separate asset; fetch both.
curl -fsSLO "${REL}/${ASSET}"
curl -fsSLO "${REL}/${ASSET}.sigstore.json"

cosign verify-blob-attestation \
  --bundle "${ASSET}.sigstore.json" \
  --type https://slsa.dev/provenance/v1 \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.7.0" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "${ASSET}"

Each nvcrectl-* binary also ships a CycloneDX SBOM, <binary>.cyclonedx.json. There are two bundles involving it, one segment apart in name, and they prove different things:

Bundle Proves
<binary>.cyclonedx.json.sigstore.json the SBOM file is the one we published and was not altered
<binary>.cyclonedx.sigstore.json the SBOM describes that binary — it is bound to the binary as its subject

Before feeding an SBOM to a scanner, check both. The binding proves an SBOM describes the binary; it says nothing about the bytes of the file on your disk, which is a separate download.

cosign verify-blob-attestation \
  --bundle "${ASSET}.cyclonedx.sigstore.json" \
  --type cyclonedx \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.7.0" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "${ASSET}"

# 2. the SBOM file you downloaded is the one that was published
cosign verify-blob-attestation \
  --bundle "${ASSET}.cyclonedx.json.sigstore.json" \
  --type https://slsa.dev/provenance/v1 \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.7.0" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "${ASSET}.cyclonedx.json"

Only the first would let a tampered local .cyclonedx.json reach your scanner: the binding attestation carries its own copy of the SBOM and never looks at your file.

checksums.txt is still published and still works, but it is not the verification story: it is served from the same origin as the assets and is itself unsigned, so it detects corruption rather than tampering.

sha256sum --check --ignore-missing checksums.txt

Third-Party Notices

Third-party Go module licenses are listed in THIRD_PARTY_NOTICES.md and attached as a release asset.

Support

What's Changed

Dependency Updates

  • build(deps): bump github.com/modelcontextprotocol/go-sdk from 1.7.0 to 1.8.0 in the go-dependencies group by @dependabot[bot] in #477
  • build(deps): bump the actions group with 6 updates by @dependabot[bot] in #478
  • build(deps-dev): bump fern-api from 5.140.0 to 5.144.3 in /fern in the fern group by @dependabot[bot] in #479

Other Changes

  • chore(fern): register v0.6.0 by @github-actions[bot] in #436
  • ci: finish #270 verify-gate residuals (step-disabled + legacy reject) by @asivanadi0 in #369
  • fix(catalog): detect GKE TCPXO networks, align NCCL env, and pin CUDA 12 for GCP H100 by @kaynetu in #437
  • docs(gcp): point the GCP H100 CUDA image workaround at categories[].options.image by @ndipebot in #442
  • fix(catalog): stop requesting mlnxnics on OCI GB200 and honor mlnxPerNode: 0 by @ndipebot in #443
  • feat(certification): expose startupStallTimeoutSeconds as a category option by @wussh in #394
  • feat(controller): surface scheduling stalls as WorkloadSchedulingBlocked by @ntheanh201 in #371
  • feat(catalog): pick GCP H100 images from the detected TCPXO plugin version by @kaynetu in #445
  • fix(metrics): report timed-out Jobs as failed on nvcre_job_status (ADR-086) by @kaynetu in #448
  • build: upgrade Go 1.27.0 → 1.27.2 and golang.org/x/net to v0.60.0 by @kaynetu in #459
  • fix: count DRA GPUs in cluster info by @mkjpryor in #419
  • feat: support Nscale NKS clusters on the DRA stack by @mkjpryor in #380
  • feat(placement): run one job of the requested size under Unpinned by @ndipebot in #454
  • Record NCCL network transport on BandwidthMeasurement by @asivanadi0 in #444
  • docs(designs): add ADR-092 for per-namespace controller write access by @kaynetu in #457
  • ci: reject cosign_version pins below v3.1.3 by @asivanadi0 in #460
  • docs(designs): add ADR-090 for sharded component results and ADR-091 for a per-GPU GEMM entry by @ndipebot in #453
  • fix(catalog): ship the NCCL topology file and stop forcing DMA-BUF off for OCI GB300 by @mlsorensen in #458
  • fix(rbac): grant the manager update on PersistentVolumeClaims by @kaynetu in #463
  • feat(catalog): support the x86 HGX B300 architecture and warn on unknown GPU defaults by @ndipebot in #464
  • fix(platform): request SYS_CHROOT on MPI workers and probe sshd with a real session by @ndipebot in #466
  • fix(catalog): render OCI RoCE env on GB200 and add rdmaDeviceAccess for hostPath RDMA by @ndipebot in #467
  • docs: prepare the v0.7.0 release by @ndipebot in #482

Full Changelog: v0.6.0...v0.7.0

Contributors

mkjpryor, mlsorensen, and 6 other contributors
Assets 27
Loading