A HelmRelease postRenderer that cannot apply merges green and silently blocks the release from upgrading

[devantler]

🤖 Generated by the Agentic Engineer

Evidence

#3577 merged with CI fully green (19 success / 7 skipped / 0 failing), including
ksail workload validate on both overlays and the merge-queue 🚀 Deploy to Prod job. It was
nonetheless unable to apply, and it silently disabled the flux-operator HelmRelease's ability to
upgrade at all. From helm-controller, 2026-09-04T10:41:27.775Z:

error while running post render on manifests: add operation does not apply:
doc is missing path: "/spec/template/spec/securityContext/fsGroupChangePolicy": missing value

Reproduced locally against the real chart: rendering flux-operator 0.50.0 with that HelmRelease's
own values shows spec.template.spec.securityContext is not rendered at all, so a JSON-6902 leaf
add beneath it cannot apply — and a JSON patch is all-or-nothing, so it took the sibling container
ops down with it. Fix in #3580.

Two properties made this invisible rather than merely broken:

1. Nothing in CI executes a HelmRelease's postRenderers. ksail workload validate and
   kubectl kustomize validate the manifests in the repository. A postRenderers block is inert
   data inside a HelmRelease CR at that point — it is only executed by helm-controller, against the
   chart's rendered output, at reconcile time. So no static check can reach it, by construction.
2. The failure then latches into a healthy-looking state. After the failed upgrade,
   helm-controller recorded observedPostRenderersDigest for the new spec, so every subsequent
   reconcile logged release in-sync with desired state. The HelmRelease reported
   Ready=True / UpgradeSucceeded — quoting the previous successful upgrade
   (lastDeployed=2026-08-30T12:20:34Z, five days earlier) — with observedGeneration equal to
   generation. Every surface a health check would read said healthy.

Audience and problem

This repository uses postRenderers as its standard mechanism for supplying security-context fields
to Helm-installed workloads that the cluster-wide mutation cannot reach (#3541, #3544, #3577). That
makes the mechanism load-bearing for security posture specifically — and it is the one mechanism
whose failures are invisible to both CI and cluster health.

The consequences compound:

• A security fix can merge, report success, and never apply.
• The affected release becomes unable to upgrade — so the next routine chart bump fails too, and
  fails for a reason unrelated to that bump.
• Posture measurement then reads the unfixed value, which invites re-deriving work already
  believed done.

Related: #2996 (a FluxInstance kustomize patch matching no target is silently discarded) is the same
class — a patch that reports success while doing nothing.

Expected behaviour

A change to a HelmRelease postRenderers block should fail in CI, on the PR that introduces it,
when the patch cannot apply to that chart's actual rendered output.

Acceptance criteria

• CI renders each changed HelmRelease's chart at its pinned version with that release's own
  values, applies its postRenderers, and fails the check when application errors.
• The check is proven by a RED/GREEN pair: fix(security): supply C-0211's two non-privilege fields to flux-operator #3577's exact patch fails it, fix(security): repair the flux-operator post-renderer so its security fields apply #3580's passes.
• Scoped to HelmReleases whose postRenderers the PR actually changes, so it stays proportionate
  to a manifest repository's CI budget.
• A postRenderer failure at reconcile time is observable without reading controller logs —
  either surfaced on the HelmRelease's own conditions or alerted on. (Ready=True while an
  upgrade is failing is the reporting defect, and it is separable from the CI gate; split it out
  if it is not cheap here.)

Size

Medium. The rendering half is mechanical — the chart, its version and its values are all in the
HelmRelease already. The judgement is in scoping which releases to render per PR, and in deciding how
much of the observability half belongs here versus its own issue.

Part of #2627