Repository navigation
Releases
Releases · NVIDIA/ccluster
v0.1.0
Added
CCLComponentSetreportsstatus.componentHealthStatuses: each released component's
health(Healthy,Progressing,Degraded,Suspended,Missing,Unknown,
SucceededorFailed),reason,messageandlastTransitionTime, as its chart's
health function judges it from Argo CD's resource tree, read through argocd-server.
job_deployment,dynamo_deploymentandcompute_deploymentjudge their Job,
DynamoGraphDeployment and load balancer; every other chart starts from Argo CD's
aggregate health and adds its Pods, such as one in CrashLoopBackOff or waiting for a GPU.
The reasons the operator sets itself are Go constants,ComponentHealthReason<Reason>,
in the API package; a chart's own reasons are exported from its chart package, such as
jobdeployment.ReasonComplete.status.componentSyncStatuseskeeps an entry for a component removed from the spec until
its Application is gone, withremovalset toDeletePendingorDeleting.
status.componentHealthStatuseslists it too, judged by the default health function.
Progressingreports the reasonComponentRemovingwhile those Applications are deleted.- The cp operator takes
--componentset-max-concurrent-reconciles(default 4), the number
ofCCLComponentSets it reconciles at once, for the rollout and for component health
each. Before, it reconciled them one at a time.
CCLInfraPlanes are still reconciled one at a time. catalogs/cluster_base/kai_schedulerinstalls the KAI scheduler (upstream chart
v0.15.3), which everydeploymentschart names as itsschedulerNameby default and
whoseQueueCRDkai_gpu_queuesneeds. Install it before both. It bin-packs GPUs,
sets requests and limits for the scheduler, binder and operator, and creates an
unlimiteddefault-queuefor pods without akai.scheduler/queuelabel. The admission
webhook does not set thenvidiaRuntimeClass on GPU pods, because the catalog runs
gpu-operator with its container toolkit off, and a node's containerd may then have no
nvidiaruntime handler. A checksum ConfigMap resyncs the Application when values
that reach the hook-appliedkai-configConfig CR change.cluster_base.Components()
installs it after gpu-operator.catalogs/cluster_base/istiotakesistio-cni.tolerations(IstioCNI.Tolerations), which
replaces the istio-cni-node DaemonSet's tolerations; an empty list removes them all. Unset,
the DaemonSet keeps the upstream wildcard tolerations.- The ip operator verifies
metrics_server:Verifiedstays False until the
v1beta1.metrics.k8s.ioAPIService is Available, which HorizontalPodAutoscalers and
KEDA's cpu and memory triggers need. The check accepts any metrics server, so a managed
cluster's built-in one passes and the chart should be left out there. The ip
operator's ClusterRole now reads APIServices. - The ip operator verifies
kube_prometheus_stack:Verifiedstays False until the
Serviceprometheus/prometheus-op-kube-prometh-prometheus, the address the inference
and dynamo ScaledObjects and prometheus-adapter hardcode, has a ready endpoint. A
Prometheus under another name or namespace fails. The ip operator's ClusterRole now
reads EndpointSlices. ccluster list infraplanes|componentsets -n NSand
ccluster get infraplane|componentset NAME -n NSshow what is registered and deployed in
a namespace, as a table, a summary, or with-o yaml|jsonthe objects.list -Alists
across every namespace.Client.ListInfraPlanesandClient.ListComponentSetsin Go, and
CCLClient.list_infraplanesandCCLClient.list_component_setsin Python, list one
namespace.ccluster serverserves a read-only dashboard of the control plane saved by
ccluster initathttp://127.0.0.1:8484and opens it in the browser, naming the
control plane by its kubeconfig context. It counts the available infra planes and the
finished component sets, and tables both kinds: planes with their conditions, and sets
by status (finished, rolling out, failed, terminating) with their components in
rollout order by health, followed by any removed component whose chart is still
being uninstalled. It refreshes every five seconds and keeps the last data with a
notice when a refresh fails. Without-nit reads across the cluster and
offers every namespace holding either kind; without cluster-wide read access it says
so and suggests-n. With-nit reads only that namespace. It reads with the
kubeconfig context's credentials, listens only on 127.0.0.1, ::1 or localhost, and
answers only requests addressed to one of those names. The UI is built with React and
NVIDIA's Kaizen UI design system in the style of NVIDIA's platform UI, with a light/dark toggle
that starts from the system setting, and is embedded in the binary, so the release
tarballs andgo installinclude it.
Changed
- Breaking:
catalogs/cluster_base/istioupgrades Istio from 1.24.6 to 1.32.0, which
supports Kubernetes 1.33 to 1.37. The chart pulls its subcharts from
https://blob.istio.io/istio-release/charts, because Istio stopped publishing to
istio-release.storage.googleapis.comat 1.31. Sidecars are now injected as native
sidecars, the Istio default since 1.27, soistio-proxyruns as an init container. Istio
does not support an in-place upgrade across this many minor versions. - Breaking:
cluster_base/final_stackrenames its Let's Encrypt HTTP-01 ClusterIssuer
fromletsencrypt-prodtoacme-ca, the issuer the Ingresses ofinference_deployment
anddynamo_deploymentrequest. On a plane built fromcluster_basethey now get a
trusted certificate instead of nginx's fake one. Its ACME account key Secret is now
acme-ca-account-key, so the issuer registers a new account, under the email
ccluster-dev@nvidia.com. Anything else that requestedletsencrypt-prodmust request
acme-ca. Issuance needs a publicly resolvable domain and port 80 on the ingress
LoadBalancer reachable from the internet. - Breaking:
cluster_base/istiodefaultsistio-cnito the standard CNI paths,
cniConfDir: /etc/cni/net.dandcniBinDir: /opt/cni/bin, which suit EKS, kind and
kubeadm. The previous k3s defaults leftistio-cni-nodeunready, and the component
Degraded, on every other cluster. On k3s setcniConfDirto
/var/lib/rancher/k3s/agent/etc/cni/net.dandcniBinDirto
/var/lib/rancher/k3s/data/cni(istio.CNIConfDirK3s,istio.CNIBinDirK3s); on GKE
COS setcniBinDirto/home/kubernetes/bin(istio.CNIBinDirGKE). - Breaking:
status.componentSyncsis renamedstatus.componentSyncStatusesand loses
health; health is instatus.componentHealthStatuses. - A component is released only once the one before it is
HealthyorSucceededas its
chart judges it, not as Argo CD's aggregate health says. A runningjob_deploymentJob
now lets the next component follow, and a crash-looping Pod or a pending
DynamoGraphDeployment holds it. When the resource tree cannot be read, the aggregate
decides, as before. - Updates follow list order, like the first rollout: a component's new spec is written only
once every component before it has synced its own current spec and isHealthyor
Succeeded. An edit that changes several components reaches them one at a time, and a
component that does not finish holds back updates to every component after it. - The Application of a component removed from the spec is deleted only after every
remaining component has finished, one at a time, highestccluster.nvidia.com/wavelabel
first. Before, it was deleted in the same pass that wrote the new spec.Progressing
staysTrueuntil the last one is gone. - Deleting a
CCLComponentSetwhile removals still wait deletes the removed components'
Applications first, then the remaining components' in reverse order. - The rollout judges component health in spec order and stops at the first component that
is not finished, so a pass reads no resource tree past it. All the reads of one pass share
one 10-second deadline. - The cp operator no longer reconciles a
CCLComponentSetwhen Argo CD's periodic refresh
of one of its Applications changes nothing but the refresh time. Before, every refresh
made the rollout and the component health controller judge the set again. A rollout pass
that could not read a component's resource tree now tries again after 30 seconds, which
the next refresh used to do. - The cp operator skips the
CCLComponentSetstatus write when the status is unchanged,
such as after an Application event that changes nothing the status reports. This saves
one API request per such reconcile. ccluster-cp-operatorreads each of its own flags from aCCLUSTER_*
environment variable named after it, such asCCLUSTER_BASE_DOMAINfor--base-domain,
with a flag on the command line taking precedence. Theccluster-cpchart sets them in
manager.envOverrides, a map, so an install sets the base domain with
--set manager.envOverrides.CCLUSTER_BASE_DOMAIN=<domain>without restating the
chart's other settings, and a release keeps its own pinned catalog revision.
manager.argsnow holds only--leader-elect.ccluster register, in Go and Python, waits until the control plane has reached the
registered cluster, the CCLInfraPlane'sClusterConfigAppliedcondition beingTrue, and
prints the cluster's domain. If that takes over two minutes it fails with the condition's
reason and message, such asClusterUnreachablefor a--serverthe control plane cannot
reach, instead of reporting success. The quickstart'sregister.pyand Goregister
wait forClusterConfigAppliedrather thanAvailable.ccluster-pycatalogValuesmodels keep keys they do not model, at any depth, and
catalogs.entryandcatalogs.componentpass them to the chart unchanged...
v0.0.9
Added
- A Go build of the
cclustercommand line, released as
ccluster_X.Y.Z_<os>_<arch>.tar.gzfor Linux and macOS on amd64 and arm64 and
installable withgo install github.com/NVIDIA/ccluster/cmd/ccluster@vX.Y.Z. It has the
Python command'sinit,registerandunregister, their long flags and the same
config file; it does not accept the short forms-cpand-ip. The Python command stays
inccluster-py; install one or the other, since both are calledccluster.
ccluster completion bash|zsh|fishprints a shell completion script; the README shows
how to load it. github.com/NVIDIA/ccluster/pkg/cclclient, a Go client designed like the Python
CCLClient: it registers and unregisters infra planes and applies, polls and deletes
component sets in one namespace.ComponentSetFinishedchecks a set already in hand,
such as one from an informer, for convergence.catalogs.Componentplaces the entry a
chart package'sNewreturns onto an infra plane.- A generated Go client for the API in
github.com/NVIDIA/ccluster/pkg/client: a typed
clientset with server-side apply and a fake that tracks field ownership, apply
configurations withExtractfunctions, listers and a shared informer factory for
CCLInfraPlane,CCLComponentSetandCCLClusterConfig. - The API's Go package exports the condition types and reasons the operators report,
such asCCLComponentSetConditionProgressingandCCLComponentSetReasonWaveSyncFailed,
so Go code can match conditions without repeating their strings. CCLInfraPlanegainsstatus.domain, the domain the cp operator derives for the plane,
written on every status update whether or not the cluster is reachable, and aDomain
column.CCLClusterConfiggainsstatus.domain: thespec.domainthe ip operator last
verified the cluster against, from the same pass asobservedGeneration.- The cp operator writes the
CCLClusterConfigsingleton into each infra-plane cluster,
carrying the plane's derived domain, using the plane's credentials Secret (the one Argo
CD uses). A newClusterConfigAppliedcondition reportsApplied,CRDMissing,
Conflict,AccessDenied,ClusterUnreachable,UnsupportedCredentialsor
InvalidCredentials, andUnknown/NotAvailablewhileAvailableisFalse. A plane
writes and deletes only theCCLClusterConfigit created: any other, such as one applied
by hand, is left alone and reported asConflict, retried every minute, until it is
deleted. Deleting theCCLInfraPlanedeletes theCCLClusterConfigit created. A plane
registering the cp cluster itself is written with the operator's own ServiceAccount,
which the cp role now grantscclclusterconfigs. New cp flag--infraplane-resync-period
(default 5m) sets how often a writtenCCLClusterConfigis rewritten to restore drift. - The cp operator watches each infra-plane cluster's
CCLClusterConfigand copies its
status, verbatim, into the newCCLInfraPlane.status.clusterConfig, so the ip operator's
Verifiedcondition is visible on the control plane;kubectl get cclinfraplaneshows it
in aVerifiedcolumn. A change on the infra plane, including a deleted
CCLClusterConfig, reconciles theCCLInfraPlaneat once, and
--infraplane-resync-periodis a safety net. Accluster_remote_watchesmetric counts
the infra-plane clusters being watched. - The ip operator reports the cluster's LoadBalancers in a new
CCLClusterConfig.status.loadBalancers: every Service of typeLoadBalancerlabelled
ccluster.nvidia.com/loadbalancer: <lb>, listed by name with itsipandhostname.
It watches those Services, so a newly assigned address shows at once. Two Services with
the same name are left out and reported asVerified=False/LoadBalancerConflict. The cp
operator copies the list intoCCLInfraPlane.status.clusterConfig. The label's value is
the chart's choice and becomes the plane's subdomain,*.<lb>.<domain>, so it has to be
a DNS label and unique on the cluster. The ip operator's ClusterRole gains read access to
Services. - The cp operator writes an external-dns
DNSEndpointperCCLInfraPlaneinto its own
namespace (or--dns-endpoint-namespace), pointing*.<lb>.<domain>at each of the
plane's LoadBalancers labelledccluster.nvidia.com/loadbalancer: <lb>, such as
*.nginx.<domain>at its ingress: an A or AAAA record for an IPv4 or IPv6 address, in
canonical form and with an IPv4-mapped IPv6 address as IPv4, a CNAME for a hostname. A
LoadBalancer without an address has no record, and the DNSEndpoint is withdrawn while
none has one and when the plane is deleted. The records go into the most specific DNS
zone external-dns manages that the domain falls under, which the user creates and
delegates. A newDNSReadycondition andDNScolumn report on it,
DNSEndpointCRDMissingwhile external-dns is not installed,InvalidLoadBalancerfor a
LoadBalancer that gets no record because its label value cannot be a DNS label or its
address is not valid, such as an IP with a zone or a hostname that is not a lowercase DNS
name, andDNSEndpointErrorwhen the DNSEndpoint cannot be written. The operator's
ClusterRole gainsexternaldns.k8s.iodnsendpoints. - The cp chart gains an optional external-dns dependency (
external-dns.enabled, off by
default) that publishes those DNSEndpoints and nothing else: it reads only the release
namespace and only thecrdsource. Its provider, zones and credentials are set through
its values.make deploy-cp DNS=trueenables it. - The cp chart declares
ccluster-cp, a read-only Argo CD account (apiKeyonly,
applications, geton every Application), for the operator to read Applications'
resource trees, live manifests and events from argocd-server. When the chart installs
Argo CD, a post-install hook creates the account's token and stores it in Secret
ccluster-cp-argocd-token; with an existing Argo CD, the user creates it. The cp
operator reads argocd-server's API through its Service in the Argo CD namespace, which
must be in the operator's cluster. New flags--argocd-server-service(default
argocd-server) and--argocd-token-secretname the Service and the token Secret; the
cp chart's default arguments set the Service for the bundled Argo CD. inference_deployment,compute_deployment,job_deploymentanddynamo_deployment
takeimagePullSecretNamesandsecretEnvVars, which name Secrets in the release
namespace. Secret values no longer have to pass through the CCLComponentSet or the Argo
CD Application, and a pull Secret kept current by another tool is used on the next pull.
imagePullSecretNamesare added to every pod'simagePullSecretsafter the chart's
own, each once. EachsecretEnvVarsentry,{secretName, key}, is set afterenvVars
as asecretKeyRef. The charts never create or read these Secrets. With both values
unset, the rendered objects are unchanged. Indynamo_deployment,secretEnvVarsis
filtered asenvVarsis:DYN_ROUTER_MODEsets the Frontend's router mode, and
DYN_EVENT_PLANE,DYN_REQUEST_PLANE,PYTHONHASHSEED,DYN_SELF_HOST_METADATA,
DYN_ROUTER_QUEUE_THRESHOLD,DYN_DISABLE_FRONTEND_ADMIN_API,
DYN_DISABLE_FRONTEND_NVEXTandDYN_VLLM_ENABLE_INFERENCE_V1_GENERATEare dropped.
Validaterejects asecretEnvVarsvariable that is also inenvVarsor that the
chart can set itself, andHF_TOKENset by bothhfTokenandsecretEnvVars.- Python:
CCLClient.check_infraplanedry-runs the CCLInfraPlaneapply_infraplane
writes, so the API server checks it without anything being written. - Every merge to
mainpublishesccluster-pyto PyPI as a pre-release of the next
version,X.Y.Z.dev<N>withNthe commit count, next to theX.Y.Z-dev.<sha>charts
of the same commit, so a development chart has a matching client.pipinstalls it only
when pinned exactly, given--pre, or no release satisfies the requested range.
Changed
- Breaking: The cp operator requires
--base-domainand refuses to start without a
valid one. Each CCLInfraPlane's domain is derived as<name>.<namespace>.<base-domain>.
The cp operator writes that domain to the plane'sCCLClusterConfigand publishes
*.<lb>.<name>.<namespace>.<base-domain>for each labelled LoadBalancer. Two planes can
never claim equal or overlapping domains (#159), and one DNS zone for the base domain
covers every plane. Changing the base domain moves every plane's domain, records and
SPIFFE trust domain, so set it once per control plane. Add--base-domainto the cp
chart'smanager.args.make deploy-cpandmake run-cppassBASE_DOMAIN, which
defaults toexample.com. - Breaking: A CCLInfraPlane's name must be a DNS label of at most 63 characters, since
the domain is derived from it. - Breaking: A
CCLClusterConfigapplied by hand to a cluster aCCLInfraPlane
registers makes that plane reportConflict. Delete it so the cp operator writes its
own. - Breaking:
catalogs/cluster_base/ingress_nginxnow creates the controller Service,
as aLoadBalancerlabelledccluster.nvidia.com/loadbalancer: nginx, where it used to
create none. An infra plane gets a new cloud load balancer on its next sync; a Service
provisioned for the controller outside ccluster is no longer needed. - Breaking: The
final_stackSPIRE OIDC discovery Ingress moves from
oidc-discovery.<domain>tooidc-discovery.nginx.<domain>, under the ingress's
wildcard record. The old URL stops resolving to it, so anything configured with it, such
as an AWS IAM OIDC provider for SPIRE JWTs, must be updated to the new URL. - Breaking: The
inference_deploymentanddynamo_deploymentIngress host moves from
<name>.<namespace>.<domain>to<name>.nginx.<domain>, under the ingress's wildcard
rec...
v0.0.8
Changed
- The operator and
compute-sshimages are built on NVIDIA's distroless base,
nvcr.io/nvidia/distroless/static:v4.1.4, instead of upstream
gcr.io/distroless/static:nonroot. They still run as UID 65532.
Their OCI labels describe ccluster rather than the base image:
org.opencontainers.image.revisionis the commit the image was built from, and
versionandcreatedare empty, because a release retags rather than rebuilds. - The
final_stackletsencrypt-prodClusterIssuer no longer sets an ACME account
email, so clusters no longer register their Let's Encrypt account under a fixed
third-party address.
v0.0.7
Added
ccluster-pyinstalls acclustercommand built on click.ccluster init -cp CONTEXT
saves the control plane's kubeconfig context after checking it serves the ccluster API.
ccluster register NAME -n NS -ip CONTEXT [--server URL]then creates a cluster-admin
ServiceAccount token on the infra-plane cluster and registers it with that control plane as
the CCLInfraPlane NAME.
ccluster unregister NAME -n NSdeletes the CCLInfraPlane and its credentials Secret,
leaving the infra plane's ServiceAccount and token in place.- The compute-ssh launcher that
compute_deploymentcopies into the user container is
built fromcatalogs/deployments/compute_deployment/compute-sshand published as
ghcr.io/nvidia/ccluster-compute-ssh, tagged with the commit hash on every merge to
main and withX.Y.Zon release. inference_deployment,compute_deployment,job_deploymentanddynamo_deployment
takepodLabelsandpodAnnotations, added verbatim to every pod the chart creates,
including the Dynamo Frontend and workers. A key the chart sets itself keeps the chart's
value. They are for the caller's own metadata and must never carry tenant input, which
belongs inuserAnnotations. Rendered output is unchanged when they are unset.spire_crdshas the first real check,spirecrds.Verify: it checks that the three SPIRE CRDs are
Established, through the newcommon.CRDsEstablishedhelper in thecatalogs/common
package, which holds helpers shared by catalog chart packages. The ip operator can now
read CustomResourceDefinitions.
Changed
- Breaking: the cp operator's
--argocd-namespacedefaults to the operator's own
namespace, where the cp chart installs Argo CD, instead ofargocd. A plain
helm installof the chart with its bundled Argo CD now works in any namespace. With an
existing Argo CD in another namespace, set--argocd-namespaceinmanager.args, or
ARGOCD_NAMESPACEformake deploy-cp. compute_deployment'ssshInjectorImageis optional and defaults to
ghcr.io/nvidia/ccluster-compute-sshat the release version. A caller that sets it
keeps its own image; the image must provide/compute-ssh.
Fixed
- The
istiocluster_base component now reachesSynced, so the components after it in a
CCLComponentSet are released. The cp operator tells Argo CD to ignore, and not revert,
thefailurePolicythat istiod sets on its two validating webhooks once it is serving. job_deploymentno longer sets theReplace=truesync option on its Job. An automated
sync that followed the one creating the Job tried to replace it, which Kubernetes always
rejects for an existing Job, so the Application's sync stayed failed and the
CCLComponentSet stopped atWaveSyncFailedafter the Job had succeeded.
v0.0.6
Added
CCLComponentSetstatus gainscomponentSyncs, which reports Argo CD's health and sync
verdict for each released component and whether that verdict was reached on the spec the
operator wrote.- The first ccluster-ip-operator controller: it runs every
catalogs/cluster_base
component'sVerifieragainst theCCLClusterConfigand reports the outcome as its
Verifiedcondition, naming each failing chart with the error its Verifier returned.
Eachcluster_basechart is registered in the operator's verifier registry
(internal/controller/ip) with a check its chart package defines; every chart starts
with one that always passes, and real checks follow. The ip ClusterRole is now generated frominternal/controller/ip. dynamo_deploymentgainskvTransfer, which selects the NIXL backend of the
disaggregated pools, UCX or LIBFABRIC. LIBFABRIC requireskvTransfer.efa, which gives
every worker one EFA device through DRA.dynamo_deploymentandinference_deploymentgainvolumeMounts. Dynamo mounts the
claims into the Frontend and every worker.
Changed
inference_deploymentrendershpa.cooldownPeriod: 0as 1800, as the platform chart
does.inference_deploymentsidecars emit only inbounddownstream_rqstats.- The Applications the operator writes carry their Helm values as
source.helm.values, a
YAML document, instead ofsource.helm.valuesObject. Argo CD renders the two
identically. Existing Applications are rewritten on the next reconcile; Argo CD compares
the rewritten source, which renders the same manifests, so the rewrite alone starts no
sync.
Removed
- Breaking:
supplementalGroupIdsis removed from thecompute_deployment,
inference_deploymentandjob_deploymentvalues. It added the host GIDs that AMD GPU
device nodes need, and no supported accelerator uses it.inference_deploymentnow
renders no podsecurityContext. With the list empty it rendered
securityContext: null, which server-side apply rejects, so Argo CD never created the
Rollout. - Breaking:
compute_deploymentdropssshUsername,sshPasswordand
sshGithubUsername, and the Secret that held the password.sshPublicKeyis now
required and is the only way to log in, as it already was for every compute deployment
the platform creates.
Fixed
- A CCLComponentSet no longer releases the next component on a
Healthy/Syncedverdict
left over from the previous spec: it waits until Argo CD'sstatus.sync.comparedTo
names the spec the operator wrote and no operation, including a PostSync hook, is
pending or running. A failed sync of that spec, including a failed hook, is reported as
WaveSyncFailed. - The
inference_deploymentKEDA query counts only the deployment's own pods. It summed
the in-flight requests of every inference deployment in the namespace on the same port. dynamo_deploymentandinference_deploymentrender anhpa.averageValueoutside
1-2^53 as 2^53 instead of passing it to KEDA.job_deploymentno longer setsttlSecondsAfterFinished: 7200on its Job. After the
TTL controller deleted a finished Job, self-heal created it again and it ran again. The
finished Job now stays until the CCLComponentSet is deleted; its pods stay unless pod
garbage collection deletes them first.
v0.0.5
Added
CCLClusterConfig, a cluster-scoped singleton describing the infra-plane cluster the
ip operator runs in (its domain). The ip chart ships this CRD; the cp chart does not.- Release artifacts carry their license payload. The operator images ship
LICENSE,
THIRD_PARTY_NOTICES.mdand the license and NOTICE files of every linked Go module
under/licenses; the packaged Helm charts includeLICENSEand
THIRD_PARTY_NOTICES.md; theccluster-pywheel and sdist includeLICENSE.
THIRD_PARTY_NOTICES.mdnow also lists the Go module licenses and the argo-cd chart
the cp chart bundles.
Removed
- Breaking: The
cluster_base/teleport_kube_agentcomponent, its Go package and its
ccluster.catalogs.cluster_base.teleport_kube_agentPython model. It wrapped the
AGPL-3.0 upstream chart, the catalog's only non-permissive dependency, and needed a
caller-run Teleport cluster; deploy the upstream chart directly instead.
cluster_base.Componentsno longer includes it.
v0.0.4
Added
CCLClient.is_component_set_finished(*, name)reports whether a set is applied and
reported converged, with one read. It requires successful validation and Application
writes, completion conditions for the current generation, and a ComponentSet that is
not being deleted.CCLClient.get_component_set(*, name)andCCLClient.get_infraplane(*, name)read one
object and return it as its model, raisingNotFoundErrorif it does not exist.component_set_current_conditions(component_set)returns the conditions the operator
computed from the object's current generation, keyed by type, so a caller can read the
reason and message behind a rollout that has not finished.- The operator and catalog
Chart.yamlfiles carryhome,sourcesandmaintainers,
and everycluster_basechart has a description.THIRD_PARTY_NOTICES.mdrecords
teleport-kube-agent as AGPL-3.0, fetched from upstream by Argo CD rather than
redistributed.
Changed
- Breaking:
CCLClient(namespace, api_client=None)becomes
CCLClient(namespace, *, api_client).api_clientis required and there is no
in-cluster default, so the caller decides which connection its clients share. Clients
built on the sameapi_clientreuse its API discovery results, so a client per
namespace costs nothing more. - Breaking: Constructing a
CCLClientreads and writes no object: it no longer
creates its namespace.CCLClient.ensure_namespace()does, and is the only method that
does, so a mistyped namespace fails against the API server rather than quietly gaining a
working infra plane. - Breaking:
CCLClient.create_infraplaneandCCLClient.create_component_setare
renamedapply_infraplaneandapply_component_set. Both server-side apply, so they
create the object or update it in place; the old names said only the first. - Breaking:
CCLClientmethods take the object's name by keyword only, for example
delete_component_set(name="local-base").
Fixed
- Application and AppProject reconciliation no longer deletes the Argo CD status and any
pending operation on the objects it manages: the managed spec is replaced with a patch
instead of a whole-object update. Explicit null Helm values survive a spec rewrite. - A CCLComponentSet waits for a referenced CCLInfraPlane whose status is still the
previous generation's, rather than trusting anAvailablecondition computed for an
older spec. The status reports that separately from a plane that is not Available. - A write to an Application or AppProject built on a stale read is retried in place
instead of failing the reconcile. Neither object has a status subresource, so Argo CD's
own status writes move the resource version the operator's patch carries, and a read
from the informer cache can miss an object that already exists. Neither lost race
surfaces as a transientApplicationFailedcondition on the CCLComponentSet any more. - Pyright and Pylance no longer report
apiVersionas a missing argument when a
CCLInfraPlaneorCCLComponentSetis constructed:CustomResource.api_versionpasses
its default toFieldby keyword, the only form type checkers read as a default.
v0.0.3
Added
catalogs/deployments/,catalogs/resources/andcatalogs/namespace_base/: the
tenant charts ported from the legacy Python catalog, one version of each and without
the cserve variants.inference_deployment(frominference_deployment_v3),
compute_deployment,job_deploymentanddynamo_deploymentare the workloads;
kai_gpu_queuesandpersistent_volumeare the GPU quota queues and tenant volumes
they reference throughkaiQueueandvolumeMounts;namespace_baseis the
namespace baseline they assume. Each pairs a Helm chart with a Go package (Values,
Validate,New) and generated Pydantic models underccluster.catalogs. The charts
take camelCase values, scope everything to the release namespace instead of a user id,
and read plain values in place of the base64 fields the Argo CD--setpath needed.catalogs/components: the registry of every chart in the catalog. Adding a chart is
now a directory with acomponent.go, one line incomponents.goand
make python-generate; a test fails when the list and the tree disagree.CCLClient.create_component_set(name, components)and
CCLClient.delete_component_set(name)inccluster-py.create_component_set
server-side applies aCCLComponentSetin the client's namespace from a list of
ComponentSpecs built withccluster.catalogs.component, so calling it again updates
the set in place;delete_component_setreturns as soon as the delete is accepted and
leaves the operator to uninstall the set's charts in reverse, one at a time.
python/examples/create_component_set.pydrives both from the command line.
Changed
CCLClient.register_clusterandunregister_clusterare renamed
create_infraplaneanddelete_infraplane, after theCCLInfraPlanethey create and
remove and matchingcreate_component_setanddelete_component_set.
python/examples/register_cluster.pyis nowcreate_infraplane.py.ccluster-pymodels a CRD'sspecandstatusonly:CCLInfraPlaneSpec,
CCLInfraPlaneStatus,CCLComponentSetSpec,CCLComponentSetStatusand their nested
types, inccluster.v1alpha1.<kind>, generated straight from the CRD schema by
hack/pyschemaand datamodel-code-generator, constraints and docstrings included.
ccluster.v1alpha1.baseis a hand-written genericCustomResource[Spec, Status]
template with a smallObjectMeta;ccluster.v1alpha1itself is generated from the
CRD kinds with theKindenum and one concreteCustomResourceper kind
(CCLComponentSet,CCLInfraPlane), whichCCLClient.create_infraplaneand
create_component_setbuild and return.hack/pyschemanow reflects chartValues
only and shapes no Python; the chart packages are written by the generator with
CHARTappended, and the catalog and version directories are namespace packages.
v0.0.2
Added
ccluster.client.CCLClientinccluster-py:register_cluster(name, server=None, bearer_token=None, ca_cert=None)creates an Argo CD-shaped credentials Secret and the
CCLInfraPlanethat points at it, andunregister_cluster(name)removes both. The
client writes into one namespace, creating it if needed, and connects with the
in-cluster configuration unless given anApiClient. With noserver,bearer_token
orca_cert,register_clusterregisters the cluster the operator itself runs in,
credential-free, using Argo CD's special-cased in-cluster server address.
python/examples/register_cluster.pydrives it from the command line.ccluster-py
now depends on thekubernetespackage.
Changed
- Charts published from
mainare versionedX.Y.Z-dev.<sha>, whereX.Y.Zis the
VERSIONfile with the patch bumped (0.0.2-dev.<sha>afterv0.0.1), instead of
0.0.0-g<sha>. Dev builds now sort between the release they follow and the next one. make build-installer-*(and somake build/make helm-generate) no longer rewrites
config/{cp,ip}/kustomization.yaml. The overlays commit the image name
ghcr.io/nvidia/ccluster-{cp,ip}-operatorand set no tag; the deployed image is
manager.imagein the chart'svalues.yaml, stamped atmake helm-package, so the
kustomize image was dead output that only dirtied a tracked file.- The chart values toggles follow the plugin's new naming:
crd.enabled,
metrics.enabled,certManager.enabled,prometheus.enabled,rbac.helpers.enabled
(wasrbacHelpers.enable) and, for consistency,argo-cd.enabled. New keys with the
plugin's defaults:manager.enabled,manager.terminationGracePeriodSeconds,
metrics.secure,rbac.namespaced,serviceAccount.*,networkPolicy.enabled. The
unusedmanager.envandmanager.envOverrideskeys are gone. The rendered manifests are
otherwise unchanged, except that an emptymanager.image.tagnow falls back to
Chart.appVersion.
Fixed
manager.imagePullSecretsin theccluster-cpandccluster-ipcharts is now rendered
into the manager Deployment's pod spec. It was declared invalues.yamlbut no template
consumed it, so a chart installed from a cluster that needs credentials for the private
ghcr.io/nvidiaimages wentImagePullBackOffwith the value silently ignored. The
charts are regenerated with kubebuilder v4.15.0, whosehelm/v2-alphaplugin emits the
block; the Makefile now pins and downloads that version instead of using whatever is on
PATH.
v0.0.1
First release. Everything below is new; the entries that described intermediate states
of the unreleased tree (the CCLDeployment to CCLComponentSet rename, the move from
kustomize to Helm deploy targets, and fixes to code that never shipped) are folded into
the description of what ships.
Added
CCLInfraPlane(ccluster.nvidia.com/v1alpha1), which registers an infra-plane cluster
with Argo CD. The control-plane operator creates one Argo CDAppProjectper namespace,
restricted to the catalog repository, and mirrors the Secret named by
spec.clusterCredentialsRefinto the Argo CD namespace as a project-scoped cluster
Secret named<namespace>-<name>. The project is removed by the last CCLInfraPlane in
the namespace.CCLComponentSet(ccluster.nvidia.com/v1alpha1), which rolls out an ordered bundle of
catalog charts onto registered clusters. Each component becomes one Argo CD
Applicationin the CCLComponentSet's own namespace under that namespace's project; a
component's Application is not created until every component before it reports Healthy
and Synced, and teardown runs in reverse, one at a time. Progress is reported on
status.conditionsasValidated,ComponentsAppliedandProgressing. The
controller starts and reportsApplicationCRDMissingwhen Argo CD is not installed,
and picks up the Application watch on restart once it is.catalogs/cluster_base, a pure-Helm catalog of 18 platform components (Istio,
cert-manager, SPIRE, KEDA, GPU Operator, kube-prometheus-stack, Longhorn, external-dns,
Fluent Bit, Argo Rollouts and more), which is whatCCLComponentSetcomponentchart
paths resolve against. The repository and revision are operator configuration
(--catalog-repo-url,--catalog-revision), settable at deploy time with
make deploy-cp CATALOG_REPO_URL=... CATALOG_REVISION=....- Go packages describing each
catalogs/cluster_basechart's Helm values as typed structs,
undergithub.com/NVIDIA/ccluster/catalogs/.... Each exports
New(namespace, values) (v1alpha1.ComponentEntry, error), building the API type
directly;catalogs/cluster_base.Components(infraPlane)returns a sample set of all
eighteen in rollout order to copy and edit.teleport_kube_agentvalidates its required
values rather than installing an agent that cannot connect. These are a convenience for
Go callers, not a validation layer: the CRD'svaluesfield remains untyped. ccluster-py, a generated Pydantic v2 rendering of the CRDs and of every catalog
component's values, published to PyPI. Regenerated bymake python-generate; CI fails
on drift from the Go types.- Helm charts
ccluster-cpandccluster-ipfor the two operators, generated from the
kustomize overlays by the kubebuilderhelm/v2-alphaplugin and published to
oci://ghcr.io/nvidia/charts.make deploy-cp/deploy-ipinstall them with
helm upgrade --install;make undeploy-*uninstalls but keeps the CRDs
(helm.sh/resource-policy: keep), so custom resources survive. - Argo CD as a pinned dependency of the control-plane chart, installed alongside the
operator bymake deploy-cpand never by the operator itself. Skip it with
make deploy-cp ARGOCD=false; bump it withmake helm-deps-update-cp. The bundled
application controller runs withcontroller.diff.server.sideto match the
ServerSideApplysync option on every generated Application, so charts with very large
CRDs converge instead of reporting OutOfSync forever. make helm-adopt-crds-cp, which hands CRDs applied bymake installover to the Helm
release without deleting anything.- Release pipeline. Every merge to
mainpublishes both operator images (:<sha>) and
both charts (0.0.0-g<sha>) to GHCR. A release is a reviewed PR titled
release: new ccluster version vX.Y.Zthat updates this file and the rootVERSION
file, followed by a GitHub Release on its merge commit, which retags the images as
X.Y.Z, re-packages the charts, publishesccluster-py==X.Y.Zto PyPI and attaches
the assets. The workflow refuses a tag that disagrees withVERSION, and PRs carrying
thereleaselabel run the full e2e tier in CI. - Agent skills under
.agents/skills/(commit-pr,release,pr-review) that write
down the commit, release and review procedures for any coding agent;.claude/skills
symlinks to the same directory. - Two-tier end-to-end suite.
make test-e2edeploys withARGOCD=falseinto a
disposable kind cluster and covers RBAC, watch wiring and the status paths reachable
without Argo CD;make test-e2e-fullupgrades the same release to real Argo CD and
proves ordered release, reverse teardown,selfHealdrift correction, a values change
rolling forward, and the AppProject as a tenancy boundary. DESIGN.md, describing the control-plane design and the alternatives rejected along
the way, notably Argo CDApplicationSetgenerators and theRollingSyncstrategy.- Legacy
catalogpackage: Helm charts paired with typed Pydantic value schemas for the
cluster base components and the workload chartsinference_deployment_v3,
cserve_deployment_v3,compute_deployment,job_deploymentandnamespace_base.
Being retired in favour ofcatalogs/andccluster-py. - Community-health files (Code of Conduct, Contributing, Security, Support, Maintainers).