Reconsider Helm-SDK-based reconciliation for APIGateway

[SachithKasthuriarachchi]

Context

The gateway-operator provisions gateways by driving the Helm Go SDK directly from the APIGateway reconciler. I'd like to open a discussion on whether embedding Helm in the reconcile loop is the right long-term model, or whether we should reconcile owned Kubernetes objects directly.

Why Helm-in-the-reconciler is harder to operate

• Level-Triggered vs. Edge-Triggered Mismatch: A Kubernetes controller is a level-triggered reconciler whose source of truth is the live cluster state. Helm is an edge-triggered, release-revision state machine whose source of truth is its own release storage (Secrets/ConfigMaps). Stacking them means a CR status can show Programmed=False while Helm thinks the release is successfully deployed, or Helm can get wedged in a pending-upgrade state while the controller infinitely retries. Operators are forced to reason about two independent, conflicting state models to debug a single gateway.
• Lack of Self-Healing on Interrupted Steps: The InstallOrUpgrade workflow routes purely on existence (any existing release → upgrade()). If a prior reconciliation attempt left the release in pending-install or pending-upgrade (due to an operator OOM, eviction, crash mid-run, or a --wait timeout), every subsequent retry will fail with Helm errors like "another operation in progress" or "has no deployed releases". Because automatic rollback/uninstall remediation isn't natively handled here, recovery requires a human to manually run helm rollback or helm uninstall against the cluster. This introduces the exact kind of out-of-band, imperative operational step that CRDs are designed to eliminate.
• Degraded Observability: To diagnose a stuck gateway, an engineer currently has to cross-reference the CR status, the operator's in-memory tracker/logs, the cluster's Helm history, and the actual pods. With direct object reconciliation, a simple kubectl get or kubectl describe on the owned objects tells the whole story. The Helm layer adds an opaque abstraction hop that isn't transparent through standard Kubernetes tooling.
• Bypassing Kubernetes Garbage Collection: Objects are owned by the Helm release secret, not by the custom resource (CR) via native OwnerReferences. If a user accidentally deletes a child Deployment, it won't be recreated until the next reconcile loop completely re-renders and re-runs the chart. Additionally, lifecycle management and garbage collection rely on Helm uninstalling cleanly rather than Kubernetes' native cascading deletion.
• Blocking the Reconciler Queue: Using Wait: true with a high Timeout (e.g., 300s) means a slow rollout completely blocks the reconciler thread and consumes worker capacity. This conflates "deployment is progressing normally" with "deployment failed." A native level-triggered controller would simply apply the changes, return, and naturally observe readiness on subsequent events.

Proposed Alternatives / Paths Forward

If we move away from the direct Helm Go SDK approach, we could consider:

1. Pure Controller-Runtime Reconciliation: Port the Helm chart templates directly into Go structs within the operator, managing them using standard controller-runtime clients with native OwnerReferences.
2. In-Memory Template Rendering + Server-Side Apply (SSA): Use Helm purely as a rendering engine in-memory to generate manifests, but apply them directly to the cluster using Kubernetes Server-Side Apply (SSA), bypassing the Helm release storage lifecycle entirely.
3. Externalizing to GitOps/Flux/Argo: Lean into existing community tools if the goal is strictly GitOps-driven Helm deployments.

[Tharsanan1]

Hi,

The concerns are valid in the current operator helm SDK based reconciliation design. The current code can be improved(helm-operator-plugins) to address the issues: Pending state recovery and Blocking reconciler but we cannot address Dual state model, Observability, Native GC with the SDK approach.

1. If we're comfortable maintaining two sources of truth — the Helm chart for standalone users and Go structs in the operator — Pure Controller-Runtime Reconciliation is the cleanest long-term model.
2. If we want the chart to remain the single source of truth, SSA rendering is the better fit. But SSA does not delete resources that disappear from a re-rendered chart. Closing this gap requires explicit inventory tracking on the operator side — either maintaining a status.inventory of applied resources and diffing on each reconcile (the pattern Flux uses), or adopting ApplySet (KEP-3659, alpha since Kubernetes 1.27). This is the one non-trivial piece that must be explicitly scoped before committing to SSA rendering.

[SachithKasthuriarachchi]

@Tharsanan1 My primary concern here is the end-user experience. A Kubernetes operator should provide a responsive, real-time feedback loop. When a user updates a gateway CR, it shouldn't freeze the reconciler worker for 5 minutes, exhaust retries, or force the user to dig into Helm internals to figure out why a release is wedged.

If we choose In-Memory Rendering + SSA, we keep the Helm chart as the sole source of truth and solve the dual-state/observability issues. Standalone users still helm install the chart, while the operator handles the apply/prune lifecycle via plain Kubernetes objects. As you pointed out, we would need to scope the pruning gap using upstream libraries like cli-utils (which Flux uses for inventory tracking) or ApplySet.

If we choose Pure Controller-Runtime (Typed-Go), we get maximum operator stability and type safety. To mitigate the "two sources of truth" concern, we could enforce parity in CI using Golden File/Diff testing (asserting that helm template outputs and the operator's Go builders produce semantically identical manifests). This prevents silent drift, though it does accept a "double-write" workflow for new features.

Ultimately, both paths successfully get the edge-triggered Helm SDK out of the loop, which will drastically improve the operator's self-healing capabilities and remove the black-box debugging experience.

[Tharsanan1]

Yeah, +1 to Pure Controller-Runtime

[SachithKasthuriarachchi]

Thanks! Looking forward to seeing this land in a future release. Speaking from a PE/SRE perspective, the Helm-induced state issues have been a major operational pain point for us, so this architectural shift is hugely appreciated.