Helm install race: cluster-autoscaler / kccm / kcsi block on admin-kubeconfig Secret that CAPI has not produced yet

[lexfrei]

Environment

• Cozystack: v1.2.2 (verified unchanged on v1.3.0-rc.1 and master)
• Chart: packages/apps/kubernetes
• Kubernetes: v1.35.0+k3s3

Symptom

helm install for any apps.cozystack.io/Kubernetes application fails with context deadline exceeded after the 5-minute chart wait. The underlying error is a mount-time failure on three Deployments inside the chart:

FailedMount: MountVolume.SetUp failed for volume "kubeconfig":
  secret "<release>-admin-kubeconfig" not found

The affected workloads are:

• <release>-cluster-autoscaler
• <release>-kccm (cloud-controller-manager)
• <release>-kcsi-controller (CSI controller)

HelmRelease / HelmController output

creating 35 resource(s)
beginning wait for 35 resources with timeout of 5m0s
Deployment is not ready: <ns>/<release>-cluster-autoscaler. 0 out of 1 expected pods are ready
(148 duplicate lines omitted)
Error received when checking status of resource <release>-cluster-autoscaler.
Error: 'client rate limiter Wait returned an error: context deadline exceeded'
wait for resources failed after 5m0s: context deadline exceeded

Root cause

The chart renders 35 resources in a single Helm release and relies on the default Helm wait to serialize their readiness. Three workloads in that set mount <release>-admin-kubeconfig as a non-optional Secret volume:

• packages/apps/kubernetes/templates/cluster-autoscaler/deployment.yaml → secretName: {{ .Release.Name }}-admin-kubeconfig
• Same pattern for kccm/manager.yaml and csi/controller.yaml

That Secret is not produced by the chart — it is produced asynchronously by the CAPI Kubeadm control-plane provider after the Cluster CR (also created in this same release) has finished provisioning the workload-cluster control plane. On clusters where control-plane provisioning exceeds the 5-minute --wait budget (typical, especially on first boot / slow node images), the Deployment pods stay ContainerCreating for the whole window, Helm reports them not-ready, and the install times out.

CAPI operators who followed along:

Cluster kubernetes-demo-k8s   PHASE: Provisioning   AGE: 3m15s

The Secret appears only once the control plane reaches Ready, but by then the Helm operation has already failed and the HelmRelease is marked install-failed.

Reproduction

1. Cozystack v1.2.2 (also reproduced with the chart from v1.3.0-rc.1).
2. Create a Tenant (any name).
3. Create an apps.cozystack.io/Kubernetes Application inside that tenant with default values.
4. Wait ~5 minutes. The HelmRelease transitions to install-failed.

Suggested fixes (pick one)

Ordered from most surgical to most invasive:

1. Mark the admin-kubeconfig secret volume optional on the three Deployments. The workloads would start, crash-loop briefly while the Secret is absent, and recover once CAPI produces it. Single-attribute change per Deployment (optional: true on the secret volume source). Avoids the helm-wait timeout entirely but trades it for a short CrashLoopBackOff window.

2. Move cluster-autoscaler / kccm / kcsi-controller into a separate HelmRelease that dependsOn the workload-cluster control-plane Secret being present. This is the FluxCD-idiomatic shape and mirrors how the chart already splits kubernetes-demo-k8s-cilium, kubernetes-demo-k8s-coredns, etc. as child HelmReleases that dependsOn the parent (visible in the HelmRelease list during reproduction).

3. Add an init-container per Deployment that polls for the Secret with a short sleep loop. Inelegant but fully contained.

4. Raise the Helm wait timeout in the cozystack-operator past the 95th-percentile CAPI provisioning time (15m). This only papers over the race — a slow node can still miss the window — but at least tightens the loop.

Why this matters

First-time-user experience: creating a Kubernetes cluster from the dashboard produces an install-failed HelmRelease, which the UI surfaces as Failed. The cluster eventually does come up (CAPI keeps going even after the HelmRelease gives up), but the retry path is non-obvious — operators typically flux suspend && resume the HelmRelease or delete and recreate the Application.

Related

No matching existing issues found in the repo for admin-kubeconfig, FailedMount, cluster-autoscaler timeout. The last commit touching templates/cluster-autoscaler/ is f6d4541 (March 2025, resource limits). The chart structure has been in this shape since the v0.0.1 prep in February 2024.

[lexfrei]

Closing — my reproduction was confounded. I observed Cluster stuck in Provisioning and <release>-admin-kubeconfig missing after 5 minutes, and jumped to "chart race between CAPI provisioning and helm wait".

Digging one layer deeper on the same cluster, the Kamaji KamajiControlPlane status reports:

Message: cannot create or update TenantControlPlane: admission webhook
  "vtenantcontrolplane.kb.io" denied the request:
  tenant-root DataStore does not exist

kubectl get datastore returns nothing on this platform install, so the control plane was never going to come up in any timeout — the admin-kubeconfig Secret would never be produced.

The race I described may well still exist on a correctly-configured cozystack install where control-plane provisioning genuinely races the 5m Helm wait (especially for the first tenant where Kamaji has to pull images). But I have not yet reproduced it on such a cluster. If it reappears with a clean DataStore setup, I will re-open with a proper reproduction.

Apologies for the noise.

[lexfrei]

Reopening. On reflection the DataStore gap on my platform install and the chart-level race are independent problems, and the chart race stands on static analysis alone — I do not need a live reproduction to show it.

Static-analysis case

The timing contract

templates/fluxcd.yaml and every other child HelmRelease in packages/apps/kubernetes/templates/helmreleases/ already acknowledge that the parent's admin-kubeconfig Secret is asynchronous: each child sets

kubeConfig:
  secretRef:
    name: {{ .Release.Name }}-admin-kubeconfig
    key: super-admin.svc
dependsOn:
  - name: {{ .Release.Name }}
timeout: 10m

Flux HelmController polls the referenced Secret and only starts the child install once it exists. This is correct.

The inconsistency: three workloads (cluster-autoscaler, kccm, csi-controller) mount the same Secret directly as a Deployment volume in the parent HelmRelease:

# templates/cluster-autoscaler/deployment.yaml
volumes:
  - secret:
      secretName: {{ .Release.Name }}-admin-kubeconfig
    name: kubeconfig

No optional: true, no dependsOn, no init-container wait. Verified against current master:

• packages/apps/kubernetes/templates/cluster-autoscaler/deployment.yaml
• packages/apps/kubernetes/templates/kccm/manager.yaml
• packages/apps/kubernetes/templates/csi/deploy.yaml (the only optional: true in that file is on a configMapKeyRef, not on the Secret volume)

The wait budget

The parent HelmRelease installed by cozystack-operator does not set install.timeout, so the default applies (5m). The parent chart contains the Cluster/KamajiControlPlane/Machine* CRs whose reconciliation (image pulls, etcd bootstrap, apiserver Ready, CAPI control-plane secret generation) routinely exceeds 5m on a cold node, and is effectively unbounded under load.

Therefore

Whenever CAPI control-plane provisioning exceeds the 5m helm-wait budget, those three Deployments stay ContainerCreating with FailedMount: secret "...-admin-kubeconfig" not found, the parent HelmRelease fails with context deadline exceeded, and install.remediation triggers a Helm uninstall remediation — which happens to also clean up the Cluster CR (CAPI was not done), and the whole process restarts. This is visible in any cluster where provisioning-time statistically brushes against the wait timeout, not only on a broken one.

Platform-level confound was separate

My dev9 install has no DataStore — Kamaji's vtenantcontrolplane admission webhook rejects every TenantControlPlane and the admin-kubeconfig Secret would never be produced no matter how large the helm wait is. That is a different problem (platform install bug). It made my repro misleading, but it does not refute the chart-level race.

Fix options still applicable

In priority order:

1. Mark the kubeconfig Secret volume optional: true on the three Deployments. Pods would then start and CrashLoopBackOff until the Secret appears, which helmwait tolerates.
2. Move cluster-autoscaler/kccm/csi-controller to a child HelmRelease that dependsOn the parent and references the Secret via spec.kubeConfig.secretRef — the same pattern already used for cilium, coredns, csi (sub-release), etc.
3. As a workaround, set install.timeout: 15m on the parent HelmRelease rendered by cozystack-operator so the wait covers realistic control-plane bootstrap time.

(1) is smallest-diff, (2) is most consistent with the rest of the chart, (3) is band-aid.

Happy to ship a PR for option (1) or (2) if a maintainer confirms the preferred direction.

[tym83]

Reproduced on another Cozystack install (v1.2+, kubernetes chart be8bda8d49ff, tenant chart 6856691e01be, K8s v1.33.2). Same FailedMount + Helm 5m timeout loop on cluster-autoscaler/kccm/kcsi.

Adding one observation that narrows the root cause from "CAPI hasn't produced kubeconfig yet" to "DataStore CR for the tenant is never produced" in our case:

```console
$ kubectl get datastore -A
NAME DRIVER READY AGE
tenant-client123 etcd true 94d
tenant-freedom etcd true 76d
tenant-internal etcd true 346d
tenant-root etcd true 147d

expected here: one for our new tenant namespace — absent

$ kubectl get tenantcontrolplane -A | grep kubernetes-c1

briefly appears with Provisioning status, vanishes when HelmRelease retry triggers uninstall

```

Other `Kubernetes` managed services in the same cluster (`tenant-client123/kubernetes-alice`, `tenant-root/kubernetes-*`) are Ready — their DataStore CRs exist and are `READY=true`. Cluster-wide Kamaji setup is healthy; the failure is on the new-tenant path.

Topology

Parent tenant nested under `tenant-root` (itself a top-level): `tenant-root` → `tenant-whmcs` → `tenant-whmcs-oojyzknp` (new). `spec.etcd: true` on the child Tenant, `spec.inherit.ingress: true`. Child's etcd StatefulSet (3 replicas) comes up Ready, but the DataStore template in the `tenant` chart doesn't appear to render — or if it renders, the resulting CR is not admitted.

Hypothesis complement to the OP

OP suggested options 1-4 (mark kubeconfig volume optional / split HRs / init-container / bump timeout) all treat the symptom. If DataStore CR is truly missing (not just late), a fifth path surfaces: ensure the `tenant` chart's DataStore template is unconditional when `spec.etcd: true`, and that the `kubernetes-` HelmRelease `dependsOn` it. That removes the race entirely rather than tightening tolerances.

Happy to share full `kubectl describe` on `tenant-whmcs-oojyzknp` Tenant CR / deep-nested tenant spec / events timeline if it helps reproduce the DataStore-missing variant.