Skip to content

Release v0.5.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 28 Sep 17:40
· 108 commits to main since this release
v0.5.0
This tag was signed with the committer’s verified signature.
ndipebot Ebot Ndip-Agbor
GPG key ID: 75363CC079EC9202
Verified
Learn about vigilant mode.
d3e0fd0
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

NVCRE v0.5.0

The NVIDIA Cluster Readiness Engine (NVCRE) is a Kubernetes controller that certifies GPU clusters before production use. It runs real training and communication workloads across topology-aware node groups, measures goodput and bandwidth, detects hardware failures, and reports every failed node with a reason. See the README for the full feature list.

Highlights

This release focuses on scheduler queue placement on the submitted workload, certifying cordoned nodes, and phase transitions as Kubernetes Events.

The queue label reaches the object your scheduler actually reads (#353). NVCRE previously wrote the gang-scheduler queue onto the runtime Job and pod templates only. It now writes it onto the submitted workload object as well, where Kueue and KAI Scheduler look for it before any pod exists. The key is gangScheduler.queueLabelKey, which defaults to kai.scheduler/queue and is set to runai/queue on a Run:ai cluster. The same change adds workloadMetadata.labels for arbitrary labels on that object, set globally at spec.workloadMetadata or per category at spec.categories[].options.workloadMetadata.

Cordoned nodes can be certified (#358). Add a target.taintSelectors entry for the node.kubernetes.io/unschedulable taint and a run will schedule onto nodes you have cordoned, which is how you burn in hardware held out of production. NVCRE drops the hardware-failure check for that run, because a cordon is an operator decision rather than evidence of a fault.

Every phase transition is an Event (#349). Certification, Workflow, Job, and WorkloadRun each emit a Kubernetes Event when a phase becomes true, so kubectl describe shows a run's history without anyone reading controller logs. A transition into the tier's Failed phase is a Warning; every other phase is Normal.

Other improvements

  • nvcrectl setup init no longer mistakes a read-only JobSet consumer role, such as GKE's system:clustermetrics, for a leftover JobSet controller. That misclassification blocked Kubeflow Trainer installation (#368)
  • The reusable-workflow signing boundary is enforced by tests rather than by convention. Weakening attest.yml's sole-signer boundary, its trusted-context provenance predicate, or its builder-identity guard now fails make test (#308)
  • Commits must name a human author, and agent attribution trailers are not accepted (#366)

Thanks to @asivanadi0, @kaynetu, @natherz97, and @ndipebot.

Install nvcrectl

The installer is signed. Download it with its bundle, verify, then run — piping it into a shell executes it before anything has checked it.

REL=https://github.com/NVIDIA/cluster-readiness-engine/releases/download/v0.5.0

curl -fsSLO "${REL}/installer"
curl -fsSLO "${REL}/installer.sigstore.json"

cosign verify-blob-attestation \
  --bundle installer.sigstore.json \
  --type https://slsa.dev/provenance/v1 \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.5.0" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  installer

bash installer -v v0.5.0
Without cosign 
curl -fsSL https://github.com/NVIDIA/cluster-readiness-engine/releases/download/v0.5.0/installer | bash -s -- -v v0.5.0

This runs the installer before anything verifies it. TLS proves you reached GitHub; it proves nothing about the asset if the release itself was tampered with. Use it only where installing cosign first is genuinely impractical.

# newest stable release; `releases/latest` never resolves to a pre-release
curl -fsSL https://github.com/NVIDIA/cluster-readiness-engine/releases/latest/download/installer | bash

Helm Chart

This release's chart is published at digest sha256:6e67c6e81708b38cc1b90f01dbdc9b1496dc4dec721ddd2641d45fe08a16e760. The signature covers that digest; the version tag is mutable, so verify and install the same digest rather than verifying a digest and installing a tag.

CHART=ghcr.io/nvidia/cluster-readiness-engine
DIGEST=sha256:6e67c6e81708b38cc1b90f01dbdc9b1496dc4dec721ddd2641d45fe08a16e760

# 1. Verify the chart. Requires cosign; the package is public, so no login is needed.
cosign verify \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.5.0" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "${CHART}@${DIGEST}"

# 2. Pull that exact digest, then install the file you just verified.
helm pull "oci://${CHART}@${DIGEST}"
helm install nvcre ./cluster-readiness-engine@${DIGEST/:/-}.tgz \
  --namespace nvcre --create-namespace

Installing with --version v0.5.0 instead resolves the tag at install time, which is not provably the digest you verified.

helm install --verify does not check this signature. That flag looks for a PGP .prov file, a mechanism unrelated to Sigstore, and we publish no .prov files.

Controller image

A different artifact from the chart, verified separately. The release built and signed these exact digests:

Subject Digest
index (multi-platform) sha256:8578ca5aa37d8e52ffb266cc8384a6f22a2d94e063b71a0b133cbca5f82224e2
linux/amd64 sha256:38dc0fa1427301072da9e89b04af5313f933fc15f29768438d787953b8f49af8
linux/arm64 sha256:bf8104f42fc71a87b6f84f02296eeeb0a304da0c68ca2e6dac91d7b708ec86f2

The index carries the signature and the SLSA provenance; each platform digest carries a signature and its own CycloneDX SBOM. Pin the index digest rather than the tag:

IMAGE=ghcr.io/nvidia/cluster-readiness-engine/manager
INDEX=sha256:8578ca5aa37d8e52ffb266cc8384a6f22a2d94e063b71a0b133cbca5f82224e2

cosign verify \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.5.0" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "${IMAGE}@${INDEX}"

nvcrectl CLI

Download the binary for your platform from the assets below and place it on your $PATH.

Platform Binary
Linux amd64 nvcrectl-linux-amd64
Linux arm64 nvcrectl-linux-arm64
macOS amd64 nvcrectl-darwin-amd64
macOS arm64 (Apple Silicon) nvcrectl-darwin-arm64

Verify your download

Every asset ships with a Sigstore bundle, <asset>.sigstore.json, carrying a SLSA Build Provenance v1 statement. Verify the artifact itself, not a checksum served from the same place as the artifact:

REL=https://github.com/NVIDIA/cluster-readiness-engine/releases/download/v0.5.0
ASSET=nvcrectl-linux-amd64   # or installer, or any published asset

# The bundle is a separate asset; fetch both.
curl -fsSLO "${REL}/${ASSET}"
curl -fsSLO "${REL}/${ASSET}.sigstore.json"

cosign verify-blob-attestation \
  --bundle "${ASSET}.sigstore.json" \
  --type https://slsa.dev/provenance/v1 \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.5.0" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "${ASSET}"

Each nvcrectl-* binary also ships a CycloneDX SBOM, <binary>.cyclonedx.json. There are two bundles involving it, one segment apart in name, and they prove different things:

Bundle Proves
<binary>.cyclonedx.json.sigstore.json the SBOM file is the one we published and was not altered
<binary>.cyclonedx.sigstore.json the SBOM describes that binary — it is bound to the binary as its subject

Before feeding an SBOM to a scanner, check both. The binding proves an SBOM describes the binary; it says nothing about the bytes of the file on your disk, which is a separate download.

cosign verify-blob-attestation \
  --bundle "${ASSET}.cyclonedx.sigstore.json" \
  --type cyclonedx \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.5.0" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "${ASSET}"

# 2. the SBOM file you downloaded is the one that was published
cosign verify-blob-attestation \
  --bundle "${ASSET}.cyclonedx.json.sigstore.json" \
  --type https://slsa.dev/provenance/v1 \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.5.0" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "${ASSET}.cyclonedx.json"

Only the first would let a tampered local .cyclonedx.json reach your scanner: the binding attestation carries its own copy of the SBOM and never looks at your file.

checksums.txt is still published and still works, but it is not the verification story: it is served from the same origin as the assets and is itself unsigned, so it detects corruption rather than tampering.

sha256sum --check --ignore-missing checksums.txt

Third-Party Notices

Third-party Go module licenses are listed in THIRD_PARTY_NOTICES.md and attached as a release asset.

Support

What's Changed

Dependency Updates

  • build(deps): bump the kubernetes group with 4 updates by @dependabot[bot] in #373
  • build(deps): bump the actions group with 3 updates by @dependabot[bot] in #374

Other Changes

  • chore(fern): register v0.4.0 by @github-actions[bot] in #364
  • feat: implement ADR-080 phase-transition events by @kaynetu in #349
  • docs+test: correct ADR-074 D2 and gate reusable-workflow boundary by @asivanadi0 in #308
  • feat: declarative labels for generated workload objects by @kaynetu in #353
  • docs: require human commit authorship and ban agent attribution by @ndipebot in #366
  • fix: ignore read-only JobSet consumer roles in ownership scan by @kaynetu in #368
  • feat: implement support for cordoned node selection by @natherz97 in #358
  • docs: prepare the v0.5.0 release by @ndipebot in #381

Full Changelog: v0.4.0...v0.5.0

Contributors

ndipebot, dependabot, and 3 other contributors
Assets 27
Loading