Flux reconciliation triggers Password generator and ExternalSecret regeneration

[mloskot]

Background

I'm implementing an idea which I originally discussed on Slack about generating and managing API keys with ESO, which briefly boils down to this:

1. Generate Secret-s with API keys using Password generator and ExternalSecret
2. Push API keys to Azure Key Vault with PushSecret
3. Optionally, synchronise API keys back from AKV to K8s with ExternalSecret

The important requirement here is that every API key is generated only once.

N.B. My idea discussed on the Slack received the "approved stamp" from the ESO maintainers :-)

Generators and PushSecret were created for these types of use cases

Implementation

Here is the implementation of my idea which I tested using ESO 0.8.1 on AKS cluster:

apikey-generator.yaml

---
apiVersion: generators.external-secrets.io/v1alpha1
kind: Password
metadata:
  name: test-apikey-generator
  namespace: tests
spec:
  length: 32
  digits: 16
  symbols: 0
  noUpper: true
  allowRepeat: true

apikey-secret.yaml

---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: test-apikey
  namespace: tests
spec:
  refreshInterval: 0s  # generate API key only once
  target:
    name: test-apikey
  dataFrom:
  - sourceRef:
      generatorRef:
        apiVersion: generators.external-secrets.io/v1alpha1
        kind: Password
        name: test-apikey-generator

Observations

Case 1

If I manually deploy the configuration

kubectl apply -f apikey-generator.yaml
kubectl apply -f apikey-secret.yaml

then, I can see the secret value is generated as expected and it remains unchanged.

Case 2

Instead of using kubectl apply, If I do

• commit and push apikey-generator.yaml and apikey-secret.yaml
• let GitOps with Flux apply its reconciliation magic to my cluster

then, value of test-apikey secret gets frequently re-generated which is an unexpected behaviour, and undesired w.r.t. to the requirements of the API keys management.

Additionally, if I add PushSecret syncing the Azure Key Vault

---
apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
  name: test-apikey
  namespace: tests
spec:
  refreshInterval: 10s
  secretStoreRefs:
  - name: azure-backend-apikeys
    kind: ClusterSecretStore
  selector:
    secret:
      name: test-apikey
  data:
  - match:
      secretKey: password
      remoteRef:
        remoteKey: test-apikey

then, in the case of the manual deployment of the generator and the secret, there is single secret value maintained in the AKV:

but in the case of GitOps with Flux, there is stream of updates of the secret value:

Questions

How can I prevent Flux triggering re-runs of the Password generator in order to maintain stable values of the secrets with API keys ?

Is this possibly related to this issue?

• Generator double-refresh and missed refreshInterval #2184

[moolen]

If there's a resource version mismatch, it will do a full reconcile of the ExternalSecret.
The resource version is computed from the ExternalSecret labels and annotations. Flux sets labels and/or annotations and it will do a full refresh. flux sets (helm|kustomize).toolkit.fluxcd.io labels, typically for the name, namespace.

Because there are new labels and annotations it will do a full refresh. This is a feature to provide a way to re-generate / refresh on demand.

I think the root cause should be clear. If you let flux manage the ExternalSecrets from the get-go this won't happen i believe 🤞 .

related code / precedence:

external-secrets/pkg/controllers/externalsecret/externalsecret_controller.go

Lines 506 to 514 in fb944d5

	// refresh if resource version changed
	if es.Status.SyncedResourceVersion != getResourceVersion(es) {
	return true
	}
	// skip refresh if refresh interval is 0
	if es.Spec.RefreshInterval.Duration == 0 && es.Status.SyncedResourceVersion != "" {
	return false
	}

[moolen]

Ah, i see, that was a misunderstanding. To build a repro case: do you use Kind=Kustomization or Kind=HelmRelease with flux to render the ExternalSecret manifests?

[moolen]

I don't see it gets frequently changed using a Kind=Kustomization. Can you share your flux manifests?

[mloskot]

@moolen Hmm, I'm observing quite different behaviours on two separate AKS clusters. So, I need to make sense of it on my end first. Please, let's keep this thread open and I'll be back with my answer and config files soon.

[mloskot]

@moolen As I mentioned, I have two AKS clusters, A and B, both GitOps-based using Flux.
I have identified what was the difference between A and B that seems to be causing in cluster B re-generation of Password-based Secret despite setting its 'manager' ExternalSecret with refreshInterval: 0s.

I found that in cluster B I had multiple 2-3 Flux Kustomization-s with overlapping path-s, which effectively boiled down to

• path: ./clusters/b in flux-kustomization-main.yaml
• path: ./clusters/b/apps in flux-kustomization-apps.yaml
• path: ./clusters/b/apps/tests in test-api-key.yaml

where test-api-key.yaml is

---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: test-api-keys
  namespace: flux-system
spec:
  interval: 30s
  path: ./clusters/b/apps/tests
  prune: true
  sourceRef:
    kind: GitRepository
    name: my-repo

Such structure is not recommended by Flux (authors) because:

reconcilers will fight over the resources, you should make sure this doesn't happen

see:

• How to structure Flux Kustomizations for single repository? fluxcd/flux2#3801 (comment)

As soon as I cleaned the structure of B cluster ensuring there are no overlapping path-s monitored by Flux Kustomizations and re-deployed my test API keys secrets configuration, then the secret re-generation problem is gone in the cluster B, and I am observing stable value of secret

I can't explain why overlapping paths would affect Flux such that ESO re-runs Password generator referred by my API key secrets, but that is what I am observing - getting rid of overlapping paths fixes the problem.

[tculp]

If there were some way of controlling the resourceVersion manually that would be very helpful. For example an annotation like external-secrets.io/resource-version: 1 making other labels and annotations ignored. Or if there was an explicit field like updatePolicy: create that only allowed the ExternalSecret to create a secret but not modify it