fluxcd.controlplane.io/reconcile annotation handling

[bozho]

Hi,

We're on Flux operator v0.59 and I'm testing fluxcd.controlplane.io/reconcile annotation behaviour on resources managed by ResourceSet.

If a resource manifest sets the annotation to enabled, the resource will get created and kept up to date. If I manually set the value to disabled on an in-cluster resource, the operator will stop reconciling the resource. That's all expected and fine.

However, if the resource manifest sets the annotation to disabled for an existing in-cluster resource, the resource will get deleted if it exists in the cluster. Is this the intended behaviour, or a bug?

In the official docs, Conditional resource exclusion section says:

To exclude a resource based on input values, the fluxcd.controlplane.io/reconcile annotation can be set to disabled on the resource metadata. This will prevent the resource from being reconciled by the operator.

spec:
  inputs:
    - tenant: "team1"
    - tenant: "team2"
  resources:
    - apiVersion: v1
      kind: Namespace
      metadata:
        name: << inputs.tenant >>
    - apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: flux
        namespace: << inputs.tenant >>
        annotations:
          fluxcd.controlplane.io/reconcile: << if eq inputs.tenant "team1" >>enabled<< else >>disabled<< end >>

In the above example, the ServiceAccount resource is generated only for the team1 tenant.

To me, this reads more like "the operator will not touch team2's resource, whether it exists or not", rather than "if the team2 SA exists, it will get deleted".

The current behaviour actually suits me, as it would allow me to dynamically control existence of some resources without having to resort to using resource templates, but I want to make sure it's the correct behaviour :-)

[matheuscscp]

Fair, deleting sounds a bit too destructive

[bozho]

Don't get me wrong - having an annotation that would allow us to dynamically control existence of resources without having to resort to resource templates in a RecordSet would be great, just a different one. Although, I'm not sure how much would that be "in the spirit" of k8s...

[Slymaster]

The deletion isn't a special code path. The annotation is evaluated in two different places, with different effects:

• In the rendered manifest, fluxcd.controlplane.io/reconcile: disabled drops the object from the build output (builder/resourceset.go). It is no longer in the desired set, so it falls out of the new inventory and garbage collection deletes it as stale (resourceset_controller.go), exactly as if it had been removed from resources.
• On the in-cluster object, the same annotation is an apply exclusion (resourceset_controller.go). The object stays in the desired set and in the inventory, only the apply is skipped, so it is frozen rather than deleted.

The same code is in v0.59 and current main, so in a template disabled effectively means "this object should not exist for this input".

To leave an existing object in place instead, use the garbage-collection opt-out: objects whose in-cluster annotations include fluxcd.controlplane.io/prune: disabled are skipped (garbage collection docs). The live object needs that annotation before it drops out of the set: render fluxcd.controlplane.io/prune: disabled on it while it is still enabled (or kubectl annotate it), then switch reconcile to disabled. The operator then leaves it in place and stops tracking it.

Given matheuscscp's comment, an issue asking either for the docs to state this explicitly or for template exclusion to imply no-prune would probably be welcome.

[bozho]

Mate, why are you spamming flux discussions with LLM-generated content? I can ask an LLM myself.

[matheuscscp]

Agreed. @Slymaster Please stop or you will be banned from the org.

[Slymaster]

Fair point, sorry for the noise. I used an AI assistant to write these answers and posted too many of them. I'll stop here.