Skip to content

Release v0.10.0

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 02 Nov 21:28
· 1 commit to main since this release
v0.10.0
This tag was signed with the committer’s verified signature.
ndipebot Ebot Ndip-Agbor
GPG key ID: 75363CC079EC9202
Verified
Learn about vigilant mode.
bc2f6dc
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

NVCRE v0.10.0

The NVIDIA Cluster Readiness Engine (NVCRE) is a Kubernetes controller that certifies GPU clusters before production use. It runs real training and communication workloads across topology-aware node groups, measures goodput and bandwidth, detects hardware failures, and reports every failed node with a reason. See the README for the full feature list.

Highlights

This release focuses on RDMA networking that finds your cluster's own layout, installing on the Kubernetes versions the docs promise, and failure logs that name the right pod.

OCI RDMA networks are detected, not assumed — the controller no longer attaches workload pods to a hard-coded network-operator/sriov-net NetworkAttachmentDefinition. It finds the one annotated k8s.v1.cni.cncf.io/resourceName: nvidia.com/mlnxnics, whatever its name and namespace, so a cluster laid out differently, such as AICR's nvidia-network-operator, attaches its RDMA network without a hand-made copy. Detection runs only for a category whose rendered Workflow attaches the network. A match in the Certification's namespace wins; several equivalent candidates resolve to the first in sorted order and emit a SecondaryNetworkDetection event; none, several that differ, or no NetworkAttachmentDefinition API fails the Certification with SecondaryNetworkDetectionFailed before any Workflow is created, with a message saying what to do. nvcrectl certification render --dry-run detects the same way (ADR-103, #569).

The Certification CRD installs on Kubernetes 1.29 through 1.33 — every earlier release shipped one those API servers refuse (estimated rule cost exceeds budget), so nvcrectl setup init stopped at [helm] apply CRD certifications.nvcre.nvidia.com and a helm install of either chart failed the same way, although the install guide said 1.29+. Through 1.33 the API server costs any CEL rule on a resource.Quantity as a 3 MiB string whatever its maxLength, so no CEL rule reads a quantity any more: cpu and memory carry a pattern instead, and the controller and nvcrectl reject a request above its limit before any Workflow is created. make test-crd-compat now installs every chart CRD on each Kubernetes minor from 1.29 up (ADR-105, #533).

Other improvements

  • A Job that times out records the MPI launcher's log in failureLog, instead of the first running worker's sshd output with no NCCL in it. Workloads without a launcher still record their first running node pod (#568).
  • The AWS GB300 NCCL categories pin public.ecr.aws/hpc-cloud/nccl-tests to a versioned tag instead of :latest, so a moved upstream tag no longer changes what a certification runs (#574).
  • nvcrectl certification render --dry-run without --platform takes GPU and NIC counts from the platform it detects on the target nodes, as the controller does. Before, OCI L40S, GB200 and GB300, and Nscale GB300 previewed the architecture defaults, such as eight NICs on OCI GB200 where the controller requests none (#569).

Thanks to @asivanadi0, @kangclzjc, @kaynetu, @mlsorensen, and @rorajani.

Install nvcrectl

The installer is signed. Download it with its bundle, verify, then run — piping it into a shell executes it before anything has checked it.

REL=https://github.com/NVIDIA/cluster-readiness-engine/releases/download/v0.10.0

curl -fsSLO "${REL}/installer"
curl -fsSLO "${REL}/installer.sigstore.json"

cosign verify-blob-attestation \
  --bundle installer.sigstore.json \
  --type https://slsa.dev/provenance/v1 \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.10.0" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  installer

bash installer -v v0.10.0
Without cosign 
curl -fsSL https://github.com/NVIDIA/cluster-readiness-engine/releases/download/v0.10.0/installer | bash -s -- -v v0.10.0

This runs the installer before anything verifies it. TLS proves you reached GitHub; it proves nothing about the asset if the release itself was tampered with. Use it only where installing cosign first is genuinely impractical.

# newest stable release; `releases/latest` never resolves to a pre-release
curl -fsSL https://github.com/NVIDIA/cluster-readiness-engine/releases/latest/download/installer | bash

Helm Chart

This release's chart is published at digest sha256:798c3c6e98d3c99c44dc2fced05644c51fee5336525f2b12d5af372bf13514ff. The signature covers that digest; the version tag is mutable, so verify and install the same digest rather than verifying a digest and installing a tag.

CHART=ghcr.io/nvidia/cluster-readiness-engine
DIGEST=sha256:798c3c6e98d3c99c44dc2fced05644c51fee5336525f2b12d5af372bf13514ff

# 1. Verify the chart. Requires cosign; the package is public, so no login is needed.
cosign verify \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.10.0" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "${CHART}@${DIGEST}"

# 2. Pull that exact digest, then install the file you just verified.
helm pull "oci://${CHART}@${DIGEST}"
helm install nvcre ./cluster-readiness-engine@${DIGEST/:/-}.tgz \
  --namespace nvcre --create-namespace

Installing with --version v0.10.0 instead resolves the tag at install time, which is not provably the digest you verified.

helm install --verify does not check this signature. That flag looks for a PGP .prov file, a mechanism unrelated to Sigstore, and we publish no .prov files.

CRDs chart

The seven NVCRE CRDs are also published as their own chart, cluster-readiness-engine-crds, at digest sha256:2e32849da7946faa563df3ca72ec8de119de071eda8a2b1f0d4a1ffcbeffec53. Its CRDs are rendered from templates/, so helm upgrade keeps them current. Install it beside the main chart with crds.enabled=false there, and upgrade it first. Verify it the same way:

CRDS_CHART=ghcr.io/nvidia/cluster-readiness-engine-crds
CRDS_DIGEST=sha256:2e32849da7946faa563df3ca72ec8de119de071eda8a2b1f0d4a1ffcbeffec53

cosign verify \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.10.0" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "${CRDS_CHART}@${CRDS_DIGEST}"

In the main chart the CRDs moved from crds/ to the crds subchart, charts/crds/crds/. helm show crds returns them as before; tooling that reads crds/ from the unpacked chart must switch to it.

Controller image

A different artifact from the chart, verified separately. The release built and signed these exact digests:

Subject Digest
index (multi-platform) sha256:bda49e1068d95286258aea203cea154d631eafaaade10f8b2d36fa5e3c684bbb
linux/amd64 sha256:33803eacf4943e17b5957003b6df3b4fb7cdb270f8da83cff49a79ffb76d7c0b
linux/arm64 sha256:fba9771db643f1d38c6aa87780266b8255a176cdd444f33634b59121c0c98bd0

The index carries the signature and the SLSA provenance; each platform digest carries a signature and its own CycloneDX SBOM. Pin the index digest rather than the tag:

IMAGE=ghcr.io/nvidia/cluster-readiness-engine/manager
INDEX=sha256:bda49e1068d95286258aea203cea154d631eafaaade10f8b2d36fa5e3c684bbb

cosign verify \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.10.0" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "${IMAGE}@${INDEX}"

nvcrectl CLI

Download the binary for your platform from the assets below and place it on your $PATH.

Platform Binary
Linux amd64 nvcrectl-linux-amd64
Linux arm64 nvcrectl-linux-arm64
macOS amd64 nvcrectl-darwin-amd64
macOS arm64 (Apple Silicon) nvcrectl-darwin-arm64

Verify your download

Every asset ships with a Sigstore bundle, <asset>.sigstore.json, carrying a SLSA Build Provenance v1 statement. Verify the artifact itself, not a checksum served from the same place as the artifact:

REL=https://github.com/NVIDIA/cluster-readiness-engine/releases/download/v0.10.0
ASSET=nvcrectl-linux-amd64   # or installer, or any published asset

# The bundle is a separate asset; fetch both.
curl -fsSLO "${REL}/${ASSET}"
curl -fsSLO "${REL}/${ASSET}.sigstore.json"

cosign verify-blob-attestation \
  --bundle "${ASSET}.sigstore.json" \
  --type https://slsa.dev/provenance/v1 \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.10.0" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "${ASSET}"

Each nvcrectl-* binary also ships a CycloneDX SBOM, <binary>.cyclonedx.json. There are two bundles involving it, one segment apart in name, and they prove different things:

Bundle Proves
<binary>.cyclonedx.json.sigstore.json the SBOM file is the one we published and was not altered
<binary>.cyclonedx.sigstore.json the SBOM describes that binary — it is bound to the binary as its subject

Before feeding an SBOM to a scanner, check both. The binding proves an SBOM describes the binary; it says nothing about the bytes of the file on your disk, which is a separate download.

cosign verify-blob-attestation \
  --bundle "${ASSET}.cyclonedx.sigstore.json" \
  --type cyclonedx \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.10.0" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "${ASSET}"

# 2. the SBOM file you downloaded is the one that was published
cosign verify-blob-attestation \
  --bundle "${ASSET}.cyclonedx.json.sigstore.json" \
  --type https://slsa.dev/provenance/v1 \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.10.0" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "${ASSET}.cyclonedx.json"

Only the first would let a tampered local .cyclonedx.json reach your scanner: the binding attestation carries its own copy of the SBOM and never looks at your file.

checksums.txt is still published and still works, but it is not the verification story: it is served from the same origin as the assets and is itself unsigned, so it detects corruption rather than tampering.

sha256sum --check --ignore-missing checksums.txt

Third-Party Notices

Third-party Go module licenses are listed in THIRD_PARTY_NOTICES.md and attached as a release asset.

Support

What's Changed

Dependency Updates

  • build(deps): bump the docker-images group with 2 updates by @dependabot[bot] in #577
  • build(deps): bump the kubernetes group with 4 updates by @dependabot[bot] in #578
  • build(deps-dev): bump fern-api from 5.163.1 to 5.172.0 in /fern in the fern group by @dependabot[bot] in #579

Other Changes

  • chore(fern): register v0.9.0 by @github-actions[bot] in #567
  • docs(readme): reword the node health monitoring feature bullet by @kaynetu in #570
  • fix(api): install the Certification CRD on Kubernetes 1.29 through 1.33 (ADR-105) by @kangclzjc in #533
  • fix: pin the AWS GB300 nccl-tests image to a versioned tag by @rorajani in #574
  • feat(controller): detect the OCI RDMA NetworkAttachmentDefinition (ADR-103) by @mlsorensen in #569
  • fix(controller): record the MPI launcher's log when a Job times out (#546) by @asivanadi0 in #568
  • docs(designs): add ADR-106 for workload pods that are placed but never become Ready by @asivanadi0 in #556
  • docs: prepare the v0.10.0 release by @kaynetu in #582

New Contributors

Full Changelog: v0.9.0...v0.10.0

Contributors

mlsorensen, dependabot, and 4 other contributors
Assets 27
Loading