Deleted resources not pruned? How does flux identify resources to be pruned?

[jlewi]

HI Folks,

We are observing situations in which we expected resources to be pruned but they aren't. We have pruning enabled and in general pruning of deleted resources appears to work. Here's the behavior we observed.

• Commit a virtual service named "gateway" in networking.istio.io_v1beta1_virtualservice_gateway.yaml
• Sometime later create a commit that removes file networking.istio.io_v1beta1_virtualservice_gateway.yaml

I was expecting the gateway resource to be deleted but it wasn't.

We've checked the following places and they all indicate reconciliation succeeded. Here's the abbreviated status

  status:
    conditions:
    - lastTransitionTime: "2021-03-04T23:43:25Z"
      message: 'Applied revision: env/dev/d3a2028c6fdb147e2c552e8dd0263e1182fa663b'
      reason: ReconciliationSucceeded
      status: "True"
      type: Ready
    lastAppliedRevision: env/dev/d3a2028c6fdb147e2c552e8dd0263e1182fa663b
    lastAttemptedRevision: env/dev/d3a2028c6fdb147e2c552e8dd0263e1182fa663b
    observedGeneration: 2

I noticed the virtual service has the flux labels but it is missing the managedFields

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  annotations:
    hydros.primer.ai/env: dev
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.istio.io/v1beta1","kind":"VirtualService","metadata":{"annotations":{"hydros.primer.ai/env":"dev"},"labels":{"kustomize.toolkit.fluxcd.io/checksum":"521cf26e423d0f457049dabda3a784a0da72d865","kustomize.toolkit.fluxcd.io/name":"primer-deploy","kustomize.toolkit.fluxcd.io/namespace":"flux-system","mlp-tenant":"primer"},"name":"gateway","namespace":"mlp-primer"},"spec":{"gateways":["istio-system/ingressgateway"],"hosts":["*"],"http":[{"match":[{"uri":{"prefix":"/api/ml-platform/v2/namespaces/mlp-primer/"}}],"rewrite":{"uri":"/"},"route":[{"destination":{"host":"gateway","port":{"number":80}}}]}]}}
  creationTimestamp: "2021-01-29T05:13:41Z"
  generation: 1
  labels:
    kustomize.toolkit.fluxcd.io/checksum: 521cf26e423d0f457049dabda3a784a0da72d865
    kustomize.toolkit.fluxcd.io/name: primer-deploy
    kustomize.toolkit.fluxcd.io/namespace: flux-system
    mlp-tenant: primer
  name: gateway
  namespace: mlp-primer
  resourceVersion: "1285821941"
  selfLink: /apis/networking.istio.io/v1beta1/namespaces/mlp-primer/virtualservices/gateway
  uid: a188ac03-a6eb-4c25-b304-6a2a68756e14
spec:
  gateways:
  - istio-system/ingressgateway
  hosts:
  - '*'
  http:
  - match:
    - uri:
        prefix: /api/ml-platform/v2/namespaces/mlp-primer/
    rewrite:
      uri: /
    route:
    - destination:
        host: gateway
        port:
          number: 80

My expectation is that since flux created this resource it would notice that it was no longer in the repo and prune it.

How does flux determine resources it owns and should be pruned? Is this based on labels? managedFields? Annotations?

Edit

• We are using v0.5.0 which looks like it is outdated (looks like there were some changes to GC after that)

  • We will try updating to v0.9.0

• It looks like the GC logic lives here and should identify resources based on the checksum label.