Bridge Jobs for deploy/undeploy lifecycle flow control

[lockwobr]

Summary

Introduce a convention-based pattern for deploy-time readiness and undeploy-time CR cleanup: each component can ship readiness.yaml and/or cleanup.yaml alongside its existing values.yaml and manifests/*.yaml, rendered through the same template pipeline. AICR generates manifests with the right annotations for each supported deployer; deploy tools execute them via their own native primitives.

Problem

Two related gaps currently live in generated bash:

Deploy-time readiness. deploy.sh prints Deployment complete. when installs finish, but for GPU bundles that's not "ready for workloads." Recent CUJ2 run: 7 min install → 26 min actually ready.

Undeploy finalizer race. undeploy.sh.tmpl has grown ~700 lines across 7 PRs (#253, #282, #364, #416, #477, #561, #602) chasing a race: today's pre-flight scans for stuck finalizers after helm has removed the controller — too late, nothing left to release them. Hence escalation to force_clear_namespace_finalizers.

A pre-uninstall bridge Job (or argocd sync-wave ordering — see below) runs while the controller is alive, so finalizers resolve normally and the race disappears.

Related: #602, #607, #516.

Proposed Convention

Extend the existing recipes/components/<name>/ layout:

File	Purpose	Status
values.yaml, values-*.yaml	Helm values + overlays	existing
manifests/*.yaml	Raw manifests	existing
readiness.yaml	Post-install bridge Job	new
cleanup.yaml	Pre-uninstall bridge Job	new

Bundler picks up the new files by presence, renders through the existing pkg/manifest template pipeline (full access to .Values, .Release.Namespace, overlay values), and annotates for each target deployer. Components that don't need a bridge simply don't ship the file.

Example — recipes/components/gpu-operator/readiness.yaml

Waits for ClusterPolicy.status.state == ready. Reuses the upstream chart's gpu-operator SA — no new RBAC.

{{- $gpuOp := index .Values "gpu-operator" }}
{{- if ne (toString ($gpuOp.readiness.enabled | default true)) "false" }}
---
apiVersion: batch/v1
kind: Job
metadata:
  name: gpu-operator-readiness
  namespace: {{ .Release.Namespace }}
  annotations:
    "helm.sh/hook": post-install,post-upgrade
    "helm.sh/hook-weight": "20"
    "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
    "argocd.argoproj.io/hook": PostSync
    "argocd.argoproj.io/sync-wave": "10"
spec:
  activeDeadlineSeconds: {{ $gpuOp.readiness.timeoutSeconds | default 1800 }}
  template:
    spec:
      serviceAccountName: gpu-operator
      restartPolicy: Never
      containers:
        - name: wait
          image: {{ $gpuOp.readiness.image | default "bitnami/kubectl:1.30" }}
          command: [kubectl, wait, --for=jsonpath={.status.state}=ready,
                    clusterpolicy, --all,
                    --timeout={{ $gpuOp.readiness.timeout | default "30m" }}]
{{- end }}

cleanup.yaml follows the same shape with helm.sh/hook: pre-delete, deletes CR instances via kubectl delete --wait=true, and includes a last-resort force-remove fallback for stuck finalizers.

For components without a controller SA (e.g., skyhook-customizations), readiness.yaml / cleanup.yaml ships a minimal inline SA + ClusterRole + ClusterRoleBinding scoped to the CR group. Decision is per-component, not global.

Deployer coverage

Deployer	Readiness mechanism	Cleanup mechanism
helm	helm.sh/hook: post-install (helm --wait blocks on Job)	helm.sh/hook: pre-delete (helm waits for Job before removing controller)
argocdhelm	Same as helm — outer chart's helm lifecycle drives the inner hooks	Same — helm pre-delete fires during app lifecycle
argocd	argocd.argoproj.io/hook: PostSync + sync-wave	No bridge Job needed — ArgoCD's existing automated.prune: true (already set in application.yaml.tmpl) + sync-wave ordering on CR resources (higher wave than controller) + resources-finalizer.argocd.argoproj.io/foreground on the Application handles it natively: CRs prune first, controller prunes last, finalizers resolve cleanly

The argocd path uses its own native primitives — the cleanup.yaml Job would only emit for the helm and argocdhelm deployers; the argocd deployer emits sync-wave annotations on the right CR resources instead. Same outcome, two rendering paths.

What drops out of undeploy.sh.tmpl

Today	Under this pattern
extra_crds_for_release() hardcoded case	component's own cleanup.yaml
skip_preflight_for_release() hardcoded case	deleted
check_release_for_stuck_crds / check_crd_for_stuck_resources	deleted or thin post-flight observability
force_clear_namespace_finalizers at script level	bounded fallback inside each cleanup.yaml
delete_release_cluster_resources orphan loop	deleted
capture_kubectl_json + 5× ERROR blocks	deleted with pre-flight

undeploy.sh.tmpl collapses to roughly:

for release in reversed(releases); do
  helm uninstall "$release" -n "$ns" --timeout "${HELM_TIMEOUT}s"
done

deploy.sh needs no new flags — existing helm --wait already blocks on the readiness Job.

Hook injection for upstream charts

Helm hooks must live inside a chart's templates/ dir — no external override. For upstream charts AICR doesn't control:

• Wrapper/envelope chart — natural fit for [Feature]: Generic Helm-based bundle format with vendored charts #516 (every component becomes a local chart)
• Companion cleanup/readiness release — works with today's bundle format; two invocations per component in a specific order
• Chart-provided extension points (extraManifests, extraDeploy) — free lunch when upstream offers them

readiness.yaml / cleanup.yaml land in whichever chart AICR authors by the same convention.

Open Questions

1. kubectl image pinning — match cluster version, minimalist image, or AICR-shipped?
2. Hook-injection default — wrapper (needs [Feature]: Generic Helm-based bundle format with vendored charts #516) vs companion release (works today). Probably ship companion first; migrate when [Feature]: Generic Helm-based bundle format with vendored charts #516 lands.
3. Finalizer force-remove fallback — always-on (loud events) or opt-in? Leaning toward always-on to match today's force_clear_namespace_finalizers.
4. Readiness beyond jsonpath — complex cases (DaemonSet rollout + CR status combined) may need a shell step inside the Job. YAGNI until needed.
5. Sync-wave emission for argocd deployer — does AICR's argocd deployer need extension to set sync-waves on rendered CR resources, or can this come from the chart's own templates?

Acceptance Criteria

• Bundler picks up readiness.yaml and cleanup.yaml by presence
• Files render through the same pipeline as manifests/*.yaml
• Output includes hook annotations appropriate to the target deployer (helm/argocdhelm: helm hooks; argocd: sync-wave + finalizers)
• undeploy.sh.tmpl reduces to a minimal uninstall loop
• deploy.sh needs no new flags
• Tests cover: readiness-only, cleanup-only, both, neither; controller-SA reuse and inline-RBAC cases
• Install + uninstall round-trip passes against Kind/KWOK for at least one recipe covering each lifecycle type

Non-Goals

• Day-N / continuous readiness (covered in Enhance deployment-phase validation to reflect post-install workload readiness #607)
• In-cluster readiness operator
• Migration away from helm/argocd (different conversation)

[lockwobr]

Currently blocked by #631 at least the deployment side of this. Might make sense to split into 2 tickets.

[mchmarny]

I'm really on the fence on this one. On one hand, we do want to make sure the stuff comes up correctly, and the undeploy finalizer race is a real problem (see 700+ lines in undeploy.sh.tmpl).

On the other hand, deploy-time readiness and undeploy-time CR cleanup are fundamentally deployer concerns. This is exactly what Argo CD sync-waves and Flux health checks are designed to solve.

IMO AICR's role is to generate the right configuration; orchestrating its lifecycle is the deployer's job. The readiness.yaml example (wait for ClusterPolicy ready) is a textbook Argo CD health check. The cleanup.yaml pattern is deployer-specific lifecycle management each tool already has idioms for. AICR abstracting across all of them adds complexity but not a corresponding level of value.

What I'd suggest instead:

1. Simplify undeploy by trusting deployer primitives, not by adding bridge jobs
2. Add deployer-specific annotations — if components need sync-wave ordering or hook annotations, those belong in Helm values or deployer-specific overlays
3. Split the issue — per your comment, undeploy simplification is more urgent and clearly scoped

The wrapper chart direction (#516) may change my position though — if AICR controls the chart, embedding hooks becomes natural. Until then though, I'd recommend we don't build infrastructure for injecting lifecycle logic into charts we don't own.

Thoughts?

[lockwobr]

@mchmarny Fair feedback, going to split this into 2 parts. The same 2 parts was referring to above.

undeploy — agreed, walk away.

It's a user concern, not a product feature. If AICR devs need cleanup for e2e/kwok, testing workflows, tools/ is the right home (something like tools/nuke). Removes the whole wrapper-chart-dependency conversation from this issue and shrinks the scope a lot. Whatever bridge/sync-wave logic helps our own test workflows lives as dev plumbing, not as a deliverable.

readiness

I want to push back, because the deployer-primitives story weakens for the signals we actually care about.

ArgoCD's built-in health assessment covers standard K8s kinds (Deployment, DaemonSet, StatefulSet, Job, Service, PVC, etc.) plus a handful of CRDs for which upstream has registered Lua scripts — cert-manager, Argo Rollouts/Workflows, a few others. For the signals driving the 19-minute CUJ2 convergence gap — ClusterPolicy.status.state == ready, Skyhook.status.status == complete, DRA kubelet-plugin registration — ArgoCD doesn't know. It would need one of:

• Upstream Lua health script registered for the CRD (hasn't happened, probably won't soon)
• User-configured resource.customizations.health._ in ArgoCD's cluster-wide config (not something a bundle can ship; it's argocd-server config)
• A bridge that exposes readiness through a primitive argocd does understand (PostSync hook Job that waits on the CRD status, whose completion argocd recognizes natively)

So "textbook ArgoCD health check" is true for the Deployment / DaemonSet part of each chart — which helm --wait also handles on its own — but not for the component-specific CRD status signals that are the actual readiness gap here. For those, argocd is in the same position as helm: you need a bridge, or the CRD needs to expose standard K8s .status.conditions[type=Ready] (which loops back to upstream fixes we don't control).

That's why I still think the bridge is useful. No deploy tool knows about these CRDs out of the box — helm, argocd, and argocdhelm all have the same blind spot here. One Job manifest with both helm.sh/hook and argocd.argoproj.io/hook annotations works across all three. Flux would work too without any extra effort on our side — its helm-controller runs genuine helm install, so the helm hook fires natively — even though we don't ship a Flux deployer today. We're filling a gap, not papering over something the deployer already handles.

Where I agree with you on readiness:

• The wrapper-chart direction ([Feature]: Generic Helm-based bundle format with vendored charts #516) materially cleans this up — when AICR is authoring the chart, shipping a readiness hook is just chart authoring. I'm fine sequencing the actual work behind [Feature]: Generic Helm-based bundle format with vendored charts #516 rather than building hook-injection plumbing into upstream charts we don't own.
• Splitting the issue makes sense. With undeploy gone, this is now purely a deploy-time readiness issue, which tightens scope considerably.

[pwittrock]

It's great we have this pattern as an escape hatch. Let's try to avoid using it whenever possible. The reasoning is: to the extent possible AICR should strive to minimize implementing business logic, and should instead focus on the composition of bundles.

If this is a common pattern, this seems like a potential gap in GPU operator and should be addressed there. (e.g. create a values option which will surface this information through a pod). These sort of bridges are a bit brittle: they don't continuously reconcile the status, and may timeout or fail.

---

Flux can look at conditions directly so it doesn't require this bridge if the resource implements the correct conditions: reference

ArgoCD as stated can be configured to look directly at CRDs with the Lua script.

Helmfile does seem to require a bridge of this nature.

If we are going to introduce this sort of logic, it should probably be rendered as part of the deployer, not the recipe.

---

Installation and management of packages is going to require orchestration and management logic. Day 2 operations will make this even more true. Enough systems already exist that solves these. I'd recommend solving them well in a commonly used production ready system first. Other systems can be derived from this base.