Skip to content

Release v0.4.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 21 Sep 20:47
· 80 commits to main since this release
v0.4.0
This tag was signed with the committer’s verified signature.
ndipebot Ebot Ndip-Agbor
GPG key ID: 75363CC079EC9202
Verified
Learn about vigilant mode.
d221662
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

NVCRE v0.4.0

The NVIDIA Cluster Readiness Engine (NVCRE) is a Kubernetes controller that certifies GPU clusters before production use. It runs real training and communication workloads across topology-aware node groups, measures goodput and bandwidth, detects hardware failures, and reports every failed node with a reason. See the README for the full feature list.

Highlights

This release makes nvcrectl setup safe alongside an existing JobSet installation, closes a release-identity gap in the signing pipeline, and ships admission-policy samples for verifying the controller image at deploy time. Nothing changed in the catalog, the controllers, or the CRD schemas.

Setup coexists with your JobSet controller (#346). nvcrectl setup init now detects who owns the JobSet CRD before touching Kubeflow Trainer. A verified external JobSet controller is reused and Trainer installs without its bundled one; a retained CRD with no controller gets the bundled controller without a schema refresh; anything ambiguous stops before any Helm mutation and prints the manual Trainer install path with --skip-phases=deps. setup reset and automatic recovery retain the shared JobSet CRD in every ownership mode, so a reset no longer destroys JobSets that belong to other workloads. A failed Trainer uninstall now stops the reset and exits non-zero instead of continuing into CRD cleanup, and --skip-phases=helm skips the Helm phase on both init and reset rather than being silently ignored.

Supply chain and release integrity. The attest self-test workflow could be dispatched at a release tag and mint the release signing identity for an arbitrary image. From this tag on, attest-selftest.yml holds its ref to main and attest.yml refuses allow_untagged on a release tag (#341). Tags v0.2.0 through v0.3.0 still carry the older workflow; the residual control is that Attest Self-Test stays disabled at the repository level, as documented in SECURITY.md. Two admission-policy samples under config/samples/policy/ require the release signature and SLSA provenance on the controller image before a Pod is admitted, one for Kyverno ImageValidatingPolicy and one for Sigstore policy-controller ClusterImagePolicy (#334). THIRD_PARTY_NOTICES.md, a signed release asset, now matches the modules compiled into nvcrectl (#363).

Docs and examples

  • Runnable Certification and WorkloadRun manifests under examples/, with the commands to run them (#355)
  • CHANGELOG.md in Keep a Changelog format, covering every stable release (#355, #363)

Breaking changes

  • nvcrectl setup init and setup reset reject unknown --skip-phases values. init accepts deps and helm; reset accepts cr, helm, and deps. An invocation that passes any other name now exits with an error naming the accepted phases instead of silently ignoring it (#346). Remove the unknown names from your scripts.

Thanks to @asivanadi0, @kaynetu, @natherz97, and @ndipebot.

Install nvcrectl

The installer is signed. Download it with its bundle, verify, then run — piping it into a shell executes it before anything has checked it.

REL=https://github.com/NVIDIA/cluster-readiness-engine/releases/download/v0.4.0

curl -fsSLO "${REL}/installer"
curl -fsSLO "${REL}/installer.sigstore.json"

cosign verify-blob-attestation \
  --bundle installer.sigstore.json \
  --type https://slsa.dev/provenance/v1 \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.4.0" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  installer

bash installer -v v0.4.0
Without cosign 
curl -fsSL https://github.com/NVIDIA/cluster-readiness-engine/releases/download/v0.4.0/installer | bash -s -- -v v0.4.0

This runs the installer before anything verifies it. TLS proves you reached GitHub; it proves nothing about the asset if the release itself was tampered with. Use it only where installing cosign first is genuinely impractical.

# newest stable release; `releases/latest` never resolves to a pre-release
curl -fsSL https://github.com/NVIDIA/cluster-readiness-engine/releases/latest/download/installer | bash

Helm Chart

This release's chart is published at digest sha256:f55f49639834922ce4721b3905b65ce5706c0f0276c2d1c5babc77103c0cf01f. The signature covers that digest; the version tag is mutable, so verify and install the same digest rather than verifying a digest and installing a tag.

CHART=ghcr.io/nvidia/cluster-readiness-engine
DIGEST=sha256:f55f49639834922ce4721b3905b65ce5706c0f0276c2d1c5babc77103c0cf01f

# 1. Verify the chart. Requires cosign; the package is public, so no login is needed.
cosign verify \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.4.0" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "${CHART}@${DIGEST}"

# 2. Pull that exact digest, then install the file you just verified.
helm pull "oci://${CHART}@${DIGEST}"
helm install nvcre ./cluster-readiness-engine@${DIGEST/:/-}.tgz \
  --namespace nvcre --create-namespace

Installing with --version v0.4.0 instead resolves the tag at install time, which is not provably the digest you verified.

helm install --verify does not check this signature. That flag looks for a PGP .prov file, a mechanism unrelated to Sigstore, and we publish no .prov files.

Controller image

A different artifact from the chart, verified separately. The release built and signed these exact digests:

Subject Digest
index (multi-platform) sha256:9edcb0dcac225bd66daafb8af6e0038ab5fad950222ca5fa34ccb2c81dfd74a6
linux/amd64 sha256:2997ef1d8564185b09f85a06994ecc833c2aab0add20bd72c6fc2789d95e73e8
linux/arm64 sha256:480a4914690f217ad81c6b0fc9e85e7ba65df4af2db390c71576367d8f6366d0

The index carries the signature and the SLSA provenance; each platform digest carries a signature and its own CycloneDX SBOM. Pin the index digest rather than the tag:

IMAGE=ghcr.io/nvidia/cluster-readiness-engine/manager
INDEX=sha256:9edcb0dcac225bd66daafb8af6e0038ab5fad950222ca5fa34ccb2c81dfd74a6

cosign verify \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.4.0" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "${IMAGE}@${INDEX}"

nvcrectl CLI

Download the binary for your platform from the assets below and place it on your $PATH.

Platform Binary
Linux amd64 nvcrectl-linux-amd64
Linux arm64 nvcrectl-linux-arm64
macOS amd64 nvcrectl-darwin-amd64
macOS arm64 (Apple Silicon) nvcrectl-darwin-arm64

Verify your download

Every asset ships with a Sigstore bundle, <asset>.sigstore.json, carrying a SLSA Build Provenance v1 statement. Verify the artifact itself, not a checksum served from the same place as the artifact:

REL=https://github.com/NVIDIA/cluster-readiness-engine/releases/download/v0.4.0
ASSET=nvcrectl-linux-amd64   # or installer, or any published asset

# The bundle is a separate asset; fetch both.
curl -fsSLO "${REL}/${ASSET}"
curl -fsSLO "${REL}/${ASSET}.sigstore.json"

cosign verify-blob-attestation \
  --bundle "${ASSET}.sigstore.json" \
  --type https://slsa.dev/provenance/v1 \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.4.0" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "${ASSET}"

Each nvcrectl-* binary also ships a CycloneDX SBOM, <binary>.cyclonedx.json. There are two bundles involving it, one segment apart in name, and they prove different things:

Bundle Proves
<binary>.cyclonedx.json.sigstore.json the SBOM file is the one we published and was not altered
<binary>.cyclonedx.sigstore.json the SBOM describes that binary — it is bound to the binary as its subject

Before feeding an SBOM to a scanner, check both. The binding proves an SBOM describes the binary; it says nothing about the bytes of the file on your disk, which is a separate download.

cosign verify-blob-attestation \
  --bundle "${ASSET}.cyclonedx.sigstore.json" \
  --type cyclonedx \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.4.0" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "${ASSET}"

# 2. the SBOM file you downloaded is the one that was published
cosign verify-blob-attestation \
  --bundle "${ASSET}.cyclonedx.json.sigstore.json" \
  --type https://slsa.dev/provenance/v1 \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.4.0" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "${ASSET}.cyclonedx.json"

Only the first would let a tampered local .cyclonedx.json reach your scanner: the binding attestation carries its own copy of the SBOM and never looks at your file.

checksums.txt is still published and still works, but it is not the verification story: it is served from the same origin as the assets and is itself unsigned, so it detects corruption rather than tampering.

sha256sum --check --ignore-missing checksums.txt

Third-Party Notices

Third-party Go module licenses are listed in THIRD_PARTY_NOTICES.md and attached as a release asset.

Support

What's Changed

Documentation

Dependency Updates

  • build(deps): bump the docker-images group with 2 updates by @dependabot[bot] in #360
  • build(deps): bump golang.org/x/text from 0.41.0 to 0.42.0 in the go-dependencies group by @dependabot[bot] in #361
  • build(deps): bump the actions group with 4 updates by @dependabot[bot] in #362

Other Changes

  • chore(fern): register v0.3.0 by @github-actions[bot] in #345
  • docs: propose JobSet ownership and cleanup design by @kaynetu in #336
  • docs: design phase transition events in ADR-080 by @kaynetu in #338
  • docs: add workload object labels ADR by @kaynetu in #337
  • fix(setup): respect JobSet ownership across install, reset, and recovery by @kaynetu in #346
  • fix(ci): gate attest-selftest.yml to refs/heads/main (#340) by @asivanadi0 in #341
  • ci: supply-chain hardening, benchmark reporting, and expanded project docs by @ndipebot in #355
  • feat: add ADR for cordoned node selection by @natherz97 in #348
  • fix: address feedback for ADR 081 by @natherz97 in #356
  • docs: admission-policy samples for Kyverno and Sigstore policy-controller (#272) by @asivanadi0 in #334

New Contributors

Full Changelog: v0.3.0...v0.4.0

Contributors

ndipebot, dependabot, and 3 other contributors
Assets 27
Loading