Skip to content

Release v0.6.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 05 Oct 21:36
· 33 commits to main since this release
v0.6.0
This tag was signed with the committer’s verified signature.
ndipebot Ebot Ndip-Agbor
GPG key ID: 75363CC079EC9202
Verified
Learn about vigilant mode.
5d0cea5
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

NVCRE v0.6.0

The NVIDIA Cluster Readiness Engine (NVCRE) is a Kubernetes controller that certifies GPU clusters before production use. It runs real training and communication workloads across topology-aware node groups, measures goodput and bandwidth, detects hardware failures, and reports every failed node with a reason. See the README for the full feature list.

Highlights

This release focuses on certifying clusters that carry no GPU product label, a read-only MCP server for certification state, and keeping the EFA fabric on AWS H100 and GB200.

GPU architecture no longer depends on a node label (#379). Every architecture decision NVCRE makes used to read nvidia.com/gpu.product, which GPU Feature Discovery writes. A cluster that claims GPUs through Dynamic Resource Allocation runs no GFD, so no node carried the label and every certification failed with cannot determine GPU architecture. NVCRE now reads the architecture from the gpu.nvidia.com ResourceSlices the DRA driver publishes, normalized to the exact string GFD would have written, so a mixed GFD and DRA fleet still passes the homogeneity check. The manager role gains get/list/watch on resource.k8s.io resourceslices; a missing grant leaves those nodes unlabeled rather than failing the reconcile. Offline nvcrectl certification render and workloadrun render, which have no cluster to read slices from, take --gpu-arch.

Agents can read certification state over MCP (#372). nvcrectl mcp serve exposes four read-only tools over stdio: list_categories, get_certification_status, get_certification_report, and list_failed_nodes. No tool creates, mutates, or triggers a run, and a test pins that surface so a fifth tool fails the build. Every verdict is projected from the same code that backs nvcrectl certification report, so the two cannot disagree about whether a run passed or which nodes failed.

AWS H100 and GB200 keep the EFA userspace stack (#434). Two override paths matched platform: aws with no GPU guard, so every AWS architecture removed /opt/amazon and unset NCCL_NET_PLUGIN before the workload started. The EFA devices stayed attached and unused while NCCL fell back to TCP over eth0. The cleanup is now scoped to GB300, which is RoCE and has no EFA devices; H100 and GB200 get FI_PROVIDER=efa, FI_EFA_USE_DEVICE_RDMA=1, FI_EFA_FORK_SAFE=1, and IPC_LOCK on both the catalog and WorkloadRun paths.

Certification coverage

  • GCP RTX PRO 6000 Blackwell Server Edition overlays for NCCL all-reduce, all-gather, all-to-all, and Nemotron 5 8B, with an eight-GPU architecture default. Smaller G4 node shapes need an explicit gpusPerNode (#396)
  • Under KAI Scheduler the MPI launcher gets no dependsOn gate and the JobSet no startupPolicy, so both sub-groups are populated and the gang is admitted. The ordering barrier moves into the launcher pod, which waits for every worker to answer sshd. Non-KAI paths render byte-identically (#395)

Controller correctness

  • Four controllers treated a cache miss as proof a child was deleted and acted on it destructively. Each tier now confirms absence against an uncached reader first, so a Workflow that is still running and still holding GPUs is not written off (#387, #370)
  • Bandwidth threshold verdicts are judged on the launcher's complete log, read once when the Job ends, rather than on a partial NCCL size sweep. The rows missed were the largest message sizes, where peak bandwidth is measured, so a healthy group could fail ThresholdViolation or send adaptive fault isolation down its healthy half (#418)
  • cleanupPVForPVC reads the PVC through the uncached reader. The cached read started a cluster-wide PVC informer the manager role has no watch for, so every PVC in the cluster was relisted on a loop after the first cleanup (#425)
  • WorkloadRun.spec.env values survive platform overrides that set TrainJob.trainer.env, restoring the documented user-wins precedence (#422)

Measurement and observability

  • nvcre_certification_status and nvcre_workflow_status gauges, with the same 0/1-per-status encoding as nvcre_job_status. A scrape-time collector reads them from the manager's informer cache on the elected leader, so no reconciler write is needed to keep them current (#414)
  • The ServiceMonitor keeps the controller's own namespace and job labels instead of letting Prometheus rename them to exported_*, and metrics.serviceMonitor.labels, .interval, and .scrapeTimeout are now Helm values. The discovery label was previously hardcoded to release: prometheus (#413)

Other improvements

  • nvcrectl certification run --category no longer persists the GPU product it detected into spec.target.nodeSelector. Only nvidia.com/gpu.present: "true" is stored, so node discovery is not pinned to what nodes reported at creation time (#378)
  • --timeout is validated only when --wait is set, and a value below 1s is rejected before the run is created. A tiny or non-positive timeout previously cancelled the watch immediately, which looked like a stalled run (#411)
  • The release dispatch policy tests exercise the release-tag resolver against a matching tag, a branch ref, and a different tag before crediting its shell guard, and follow local reusable-workflow calls when checking dispatch callers. A guard written as : # exit 1, or an attest call hidden one hop away, is no longer accepted (#359)
  • CodeQL analyzes pull_request, so fork PRs are scanned, and the Fern docs CLI is installed from a lockfile pinned by integrity hash rather than latest, which the release publish path resolved at tag time (#393)
  • Deployment docs cover preloading images when no registry is reachable (#399)

Breaking changes

  • The NCCL bandwidth gauges are now nvcre_nccl_algbw_gbs and nvcre_nccl_busbw_gbs. The previous _gbps names incorrectly implied gigabits; the values have always been GB/s from nccl-tests. The old names stay dual-registered as deprecated aliases for one minor release and are removed in the following minor, so a dashboard that is not changed keeps working today but should move to the _gbs names now (#412)
  • The ServiceMonitor sets honorLabels: true by default. A dashboard built on exported_namespace or exported_job stops matching after the chart upgrade; set metrics.serviceMonitor.honorLabels=false to keep the previous labels (#413)
  • nodeHealthMonitor, goodputMeasurement, and bandwidthMeasurement on a Job can no longer be added or removed after creation. Each was already immutable once set; the presence rule closes the absent-to-present and present-to-absent gap. Existing objects are unaffected and an update that does not touch these fields still succeeds, but a patch that adds one of them to a live Job is now rejected at admission (#402)

Thanks to @asivanadi0, @awolde, @efegokdemir, @gphelps-dev, @kaynetu, @lalitadithya, @MaxFreedomPollard, @mchmarny, @mkjpryor, @ndipebot, @ntheanh201, and @wussh.

Install nvcrectl

The installer is signed. Download it with its bundle, verify, then run — piping it into a shell executes it before anything has checked it.

REL=https://github.com/NVIDIA/cluster-readiness-engine/releases/download/v0.6.0

curl -fsSLO "${REL}/installer"
curl -fsSLO "${REL}/installer.sigstore.json"

cosign verify-blob-attestation \
  --bundle installer.sigstore.json \
  --type https://slsa.dev/provenance/v1 \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.6.0" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  installer

bash installer -v v0.6.0
Without cosign 
curl -fsSL https://github.com/NVIDIA/cluster-readiness-engine/releases/download/v0.6.0/installer | bash -s -- -v v0.6.0

This runs the installer before anything verifies it. TLS proves you reached GitHub; it proves nothing about the asset if the release itself was tampered with. Use it only where installing cosign first is genuinely impractical.

# newest stable release; `releases/latest` never resolves to a pre-release
curl -fsSL https://github.com/NVIDIA/cluster-readiness-engine/releases/latest/download/installer | bash

Helm Chart

This release's chart is published at digest sha256:af20cf1d7827a35e60a539ef9f7fc8445e0def61f345e8bdb030afc76f01f6cf. The signature covers that digest; the version tag is mutable, so verify and install the same digest rather than verifying a digest and installing a tag.

CHART=ghcr.io/nvidia/cluster-readiness-engine
DIGEST=sha256:af20cf1d7827a35e60a539ef9f7fc8445e0def61f345e8bdb030afc76f01f6cf

# 1. Verify the chart. Requires cosign; the package is public, so no login is needed.
cosign verify \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.6.0" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "${CHART}@${DIGEST}"

# 2. Pull that exact digest, then install the file you just verified.
helm pull "oci://${CHART}@${DIGEST}"
helm install nvcre ./cluster-readiness-engine@${DIGEST/:/-}.tgz \
  --namespace nvcre --create-namespace

Installing with --version v0.6.0 instead resolves the tag at install time, which is not provably the digest you verified.

helm install --verify does not check this signature. That flag looks for a PGP .prov file, a mechanism unrelated to Sigstore, and we publish no .prov files.

Controller image

A different artifact from the chart, verified separately. The release built and signed these exact digests:

Subject Digest
index (multi-platform) sha256:8008c6f9f7e19b317caf0f91afab06b88027c14d7dd03e4173bd49cd1f3cd27a
linux/amd64 sha256:80efd1b6b2ac99b710432c4dbfbeaaf069a6fec53e00bb6894f581e7f0d6f807
linux/arm64 sha256:1f2ef6945689d10a34e1d6b74f9a6ee52a410698231f317bd1c7d1548df19b98

The index carries the signature and the SLSA provenance; each platform digest carries a signature and its own CycloneDX SBOM. Pin the index digest rather than the tag:

IMAGE=ghcr.io/nvidia/cluster-readiness-engine/manager
INDEX=sha256:8008c6f9f7e19b317caf0f91afab06b88027c14d7dd03e4173bd49cd1f3cd27a

cosign verify \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.6.0" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "${IMAGE}@${INDEX}"

nvcrectl CLI

Download the binary for your platform from the assets below and place it on your $PATH.

Platform Binary
Linux amd64 nvcrectl-linux-amd64
Linux arm64 nvcrectl-linux-arm64
macOS amd64 nvcrectl-darwin-amd64
macOS arm64 (Apple Silicon) nvcrectl-darwin-arm64

Verify your download

Every asset ships with a Sigstore bundle, <asset>.sigstore.json, carrying a SLSA Build Provenance v1 statement. Verify the artifact itself, not a checksum served from the same place as the artifact:

REL=https://github.com/NVIDIA/cluster-readiness-engine/releases/download/v0.6.0
ASSET=nvcrectl-linux-amd64   # or installer, or any published asset

# The bundle is a separate asset; fetch both.
curl -fsSLO "${REL}/${ASSET}"
curl -fsSLO "${REL}/${ASSET}.sigstore.json"

cosign verify-blob-attestation \
  --bundle "${ASSET}.sigstore.json" \
  --type https://slsa.dev/provenance/v1 \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.6.0" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "${ASSET}"

Each nvcrectl-* binary also ships a CycloneDX SBOM, <binary>.cyclonedx.json. There are two bundles involving it, one segment apart in name, and they prove different things:

Bundle Proves
<binary>.cyclonedx.json.sigstore.json the SBOM file is the one we published and was not altered
<binary>.cyclonedx.sigstore.json the SBOM describes that binary — it is bound to the binary as its subject

Before feeding an SBOM to a scanner, check both. The binding proves an SBOM describes the binary; it says nothing about the bytes of the file on your disk, which is a separate download.

cosign verify-blob-attestation \
  --bundle "${ASSET}.cyclonedx.sigstore.json" \
  --type cyclonedx \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.6.0" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "${ASSET}"

# 2. the SBOM file you downloaded is the one that was published
cosign verify-blob-attestation \
  --bundle "${ASSET}.cyclonedx.json.sigstore.json" \
  --type https://slsa.dev/provenance/v1 \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.6.0" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "${ASSET}.cyclonedx.json"

Only the first would let a tampered local .cyclonedx.json reach your scanner: the binding attestation carries its own copy of the SBOM and never looks at your file.

checksums.txt is still published and still works, but it is not the verification story: it is served from the same origin as the assets and is itself unsigned, so it detects corruption rather than tampering.

sha256sum --check --ignore-missing checksums.txt

Third-Party Notices

Third-party Go module licenses are listed in THIRD_PARTY_NOTICES.md and attached as a release asset.

Support

What's Changed

Dependency Updates

  • build(deps): bump anchore/sbom-action from 0.24.2 to 0.24.3 in the actions group by @dependabot[bot] in #431

Other Changes

  • fix(controller): prevent false WorkloadRun failure on Workflow cache miss by @kaynetu in #370
  • chore(fern): register v0.5.0 by @github-actions[bot] in #382
  • ci: resolve the actionable code-scanning alerts by @ndipebot in #393
  • fix(cli): stop persisting detected GPU product into Certification nodeSelector by @mkjpryor in #378
  • fix(controller): confirm a missing child against the API server before acting on it by @ndipebot in #387
  • fix(controller): judge bandwidth on the complete final log, never a partial sweep by @mchmarny in #418
  • feat: add GCP RTX PRO 6000 NVCRE overlays by @awolde in #396
  • chore: add e2e tests by @lalitadithya in #427
  • test: exercise release ref guards and follow attest wrappers by @MaxFreedomPollard in #359
  • fix(cli): validate --timeout only with --wait, floor at 1s by @asivanadi0 in #411
  • fix(metrics): rename NCCL bandwidth gauges to GB/s-accurate names by @asivanadi0 in #412
  • fix(helm): honor metric labels on ServiceMonitor and expose scrape knobs by @asivanadi0 in #413
  • fix: preserve WorkloadRun user env precedence by @efegokdemir in #422
  • docs: add ADR-086 for lifecycle status metrics and timeout exclusivity by @kaynetu in #423
  • fix(controller): read the PVC in cleanupPVForPVC from the API server by @gphelps-dev in #425
  • feat: resolve GPU architecture without the nvidia.com/gpu.product label by @mkjpryor in #379
  • feat(mcp): add a read-only MCP server for certification state by @ntheanh201 in #372
  • fix(mpi): admit the KAI gang by dropping the launcher gate and waiting in the pod by @wussh in #395
  • docs(deployment): preloading images when no registry is reachable by @wussh in #399
  • fix(api): enforce presence immutability for optional JobSpec fields by @efegokdemir in #402
  • feat(metrics): scrape Certification and Workflow status from the informer cache by @asivanadi0 in #414
  • fix: keep the EFA stack for AWS H100 and GB200 on both the catalog and WorkloadRun paths by @ndipebot in #434
  • docs: prepare the v0.6.0 release by @ndipebot in #435

New Contributors

Full Changelog: v0.5.0...v0.6.0

Contributors

mchmarny, mkjpryor, and 11 other contributors
Assets 27
Loading