Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revoke kubectl managed fields ownership #527

Merged
merged 2 commits into from Jan 31, 2022

Conversation

stefanprodan
Copy link
Member

@stefanprodan stefanprodan commented Jan 9, 2022

This PR enforces Flux ownership of Kubernetes objects' fields that were applied on the cluster outside of the declared desired state. In addition, metadata annotations and labels removed from Git and are now removed from the cluster.

In order to undo changes made with kubectl apply -f and kubectl apply --server-side --force-conflicts, we have to replace kubectl-* managers with our own manager before the controller runs the server-side apply.

In addition, this PR removes the kubectl last applied configuration annotation and Flux v1 & v2 deprecated metadata.

References:

Upstream bugs:

Test this PR using ghcr.io/fluxcd/kustomize-controller:rc-e611de4e.

To use the release candidate on your cluster, add the following image patch to clusters/<cluster>/flux-system/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- gotk-components.yaml
- gotk-sync.yaml
images:
  - name: ghcr.io/fluxcd/kustomize-controller
    newName: ghcr.io/fluxcd/kustomize-controller
    newTag: rc-e611de4e

Big thanks to @somtochiama and @kingdonb for all the help validating this approach 🥇

@stefanprodan stefanprodan added the enhancement New feature or request label Jan 9, 2022
@stefanprodan stefanprodan force-pushed the ssa-override-kubectl-manager branch 10 times, most recently from 48ddfab to 1cc795f Compare January 9, 2022 19:56
kingdonb pushed a commit to kingdonb/bootstrap-repo that referenced this pull request Jan 10, 2022
We are looking for notifications spam around the time that this upgrade
goes into the cluster

(I have just re-enabled slack notifications so that I will be able to
tell one way or another)
@stefanprodan stefanprodan force-pushed the ssa-override-kubectl-manager branch 3 times, most recently from 7dcf93d to 408a889 Compare January 10, 2022 20:09
@stefanprodan stefanprodan force-pushed the ssa-override-kubectl-manager branch 2 times, most recently from 440702f to 78bd051 Compare January 17, 2022 14:35
@ericjohansson89
Copy link

ericjohansson89 commented Jan 18, 2022

This is a contribution of information to solve this issue

On resource of kind Kustomization with v1beta2 we are switching from

spec:
 patchesStrategicMerge:

to

spec:
 patches:

The kustomize controller however does not seem to have the correct permissions due to managedFields to remove the patchesStrategicMerge part meaning we get a spec that looks like

spec:
 patches:
 patchesStrategicMerge:

Flux information. RC candidate setup:

► checking prerequisites
✔ Kubernetes 1.20.7-eks-d88609 >=1.19.0-0
► checking controllers
✔ helm-controller: deployment ready
► fluxcd/helm-controller:v0.12.0
✔ image-automation-controller: deployment ready
► fluxcd/image-automation-controller:v0.15.0
✔ image-reflector-controller: deployment ready
► fluxcd/image-reflector-controller:v0.12.0
✔ kustomize-controller: deployment ready
► fluxcd/kustomize-controller:rc-78bd051e
✔ notification-controller: deployment ready
► fluxcd/notification-controller:v0.17.1
✔ source-controller: deployment ready
► fluxcd/source-controller:v0.16.0
✔ all checks passed

pre RC candidate setup:

► checking prerequisites
✔ Kubernetes 1.20.7-eks-d88609 >=1.19.0-0
► checking controllers
✗ helm-controller: deployment not ready
► fluxcd/helm-controller:v0.12.0
✔ image-automation-controller: deployment ready
► fluxcd/image-automation-controller:v0.15.0
✔ image-reflector-controller: deployment ready
► fluxcd/image-reflector-controller:v0.12.0
✔ kustomize-controller: deployment ready
► fluxcd/kustomize-controller:v0.15.5
✔ notification-controller: deployment ready
► fluxcd/notification-controller:v0.17.1
✔ source-controller: deployment ready
► fluxcd/source-controller:v0.16.0
✔ all checks passed

This is how a kustomization object looks before merging a switch from patchesStrategicMerge to patches look like

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  annotations:
    kustomize.toolkit.fluxcd.io/checksum: b164335250c2cdc202b558b3db8978769ed7a024
    reconcile.fluxcd.io/requestedAt: "2021-10-08T14:04:54.480554103+02:00"
  creationTimestamp: "2021-09-24T07:48:44Z"
  finalizers:
  - finalizers.fluxcd.io
  generation: 2
  labels:
    kustomize.toolkit.fluxcd.io/name: flux-self-management
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: flux-install
  namespace: flux-system
  resourceVersion: "602528344"
  uid: d44a6ef6-906c-4e70-aa64-13364c78414e
spec:
  force: false
  interval: 5m0s
  patchesStrategicMerge:
  - apiVersion: v1
    kind: Namespace
    metadata:
      annotations:
        redacted: redacted
      name: flux-system
  path: ./infrastructure/flux
  prune: true
  sourceRef:
    kind: GitRepository
    name: fleet-repo
  validation: client
status:
  conditions:
  - lastTransitionTime: "2022-01-18T07:54:30Z"
    message: 'Applied revision: dev/ad4a8d05c1abd1a4d77118d2a56f100f82b046df'
    reason: ReconciliationSucceeded
    status: "True"
    type: Ready
  inventory:
    entries:
    - id: _alerts.notification.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _buckets.source.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _gitrepositories.source.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _helmcharts.source.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _helmreleases.helm.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _helmrepositories.source.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _imagepolicies.image.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _imagerepositories.image.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _imageupdateautomations.image.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _kustomizations.kustomize.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _providers.notification.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _receivers.notification.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _flux-system__Namespace
      v: v1
    - id: flux-system_helm-controller__ServiceAccount
      v: v1
    - id: flux-system_image-automation-controller__ServiceAccount
      v: v1
    - id: flux-system_image-reflector-controller__ServiceAccount
      v: v1
    - id: flux-system_kustomize-controller__ServiceAccount
      v: v1
    - id: flux-system_notification-controller__ServiceAccount
      v: v1
    - id: flux-system_source-controller__ServiceAccount
      v: v1
    - id: _crd-controller_rbac.authorization.k8s.io_ClusterRole
      v: v1
    - id: _flux-edit_rbac.authorization.k8s.io_ClusterRole
      v: v1
    - id: _cluster-reconciler_rbac.authorization.k8s.io_ClusterRoleBinding
      v: v1
    - id: _crd-controller_rbac.authorization.k8s.io_ClusterRoleBinding
      v: v1
    - id: flux-system_notification-controller__Service
      v: v1
    - id: flux-system_source-controller__Service
      v: v1
    - id: flux-system_webhook-receiver__Service
      v: v1
    - id: flux-system_helm-controller_apps_Deployment
      v: v1
    - id: flux-system_image-automation-controller_apps_Deployment
      v: v1
    - id: flux-system_image-reflector-controller_apps_Deployment
      v: v1
    - id: flux-system_kustomize-controller_apps_Deployment
      v: v1
    - id: flux-system_notification-controller_apps_Deployment
      v: v1
    - id: flux-system_source-controller_apps_Deployment
      v: v1
    - id: flux-system_allow-egress_networking.k8s.io_NetworkPolicy
      v: v1
    - id: flux-system_allow-scraping_networking.k8s.io_NetworkPolicy
      v: v1
    - id: flux-system_allow-webhooks_networking.k8s.io_NetworkPolicy
      v: v1
  lastAppliedRevision: dev/ad4a8d05c1abd1a4d77118d2a56f100f82b046df
  lastAttemptedRevision: dev/ad4a8d05c1abd1a4d77118d2a56f100f82b046df
  lastHandledReconcileAt: "2021-10-08T14:04:54.480554103+02:00"
  observedGeneration: 2

After applying the RC and making the changes these are the two variants we have been seing on the kustomization

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  annotations:
    kustomize.toolkit.fluxcd.io/checksum: ea438a2e6864f3268ec968890839aba7c669aaa0
    reconcile.fluxcd.io/requestedAt: "2022-01-11T08:14:23.902964+01:00"
  creationTimestamp: "2021-09-20T09:22:40Z"
  finalizers:
  - finalizers.fluxcd.io
  generation: 10
  labels:
    kustomize.toolkit.fluxcd.io/name: flux-self-management
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  managedFields:
  - apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          f:kustomize.toolkit.fluxcd.io/name: {}
          f:kustomize.toolkit.fluxcd.io/namespace: {}
      f:spec:
        f:interval: {}
        f:patchesStrategicMerge: {}
        f:path: {}
        f:prune: {}
        f:sourceRef:
          f:kind: {}
          f:name: {}
    manager: kustomize-controller
    operation: Apply
    time: "2022-01-17T15:45:04Z"
  - apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
          f:kustomize.toolkit.fluxcd.io/checksum: {}
          f:reconcile.fluxcd.io/requestedAt: {}
        f:finalizers:
          .: {}
          v:"finalizers.fluxcd.io": {}
        f:labels:
          .: {}
          f:kustomize.toolkit.fluxcd.io/name: {}
          f:kustomize.toolkit.fluxcd.io/namespace: {}
      f:spec:
        .: {}
        f:force: {}
        f:interval: {}
        f:patchesStrategicMerge: {}
        f:path: {}
        f:prune: {}
        f:sourceRef:
          .: {}
          f:kind: {}
          f:name: {}
        f:validation: {}
      f:status:
        .: {}
        f:inventory:
          .: {}
          f:entries: {}
        f:lastHandledReconcileAt: {}
    manager: before-first-apply
    operation: Update
    time: "2022-01-17T15:44:59Z"
  - apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        f:conditions: {}
        f:lastAppliedRevision: {}
        f:lastAttemptedRevision: {}
        f:observedGeneration: {}
    manager: kustomize-controller
    operation: Update
    time: "2022-01-17T15:45:04Z"
  name: flux-install
  namespace: flux-system
  resourceVersion: "488748412"
  uid: 6619e059-068d-4b84-a63e-3b54f77802ae
spec:
  force: false
  interval: 5m0s
  patchesStrategicMerge:
  - apiVersion: v1
    kind: Namespace
    metadata:
      annotations:
        redacted: redacted
      name: flux-system
  path: ./infrastructure/flux
  prune: true
  sourceRef:
    kind: GitRepository
    name: fleet-repo
  validation: client
status:
  conditions:
  - lastTransitionTime: "2022-01-17T15:45:08Z"
    message: 'Applied revision: dev/36eef2ff32851ee0332d5ce656cc337eec5c43e5'
    reason: ReconciliationSucceeded
    status: "True"
    type: Ready
  inventory:
    entries:
    - id: _alerts.notification.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _buckets.source.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _gitrepositories.source.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _helmcharts.source.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _helmreleases.helm.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _helmrepositories.source.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _imagepolicies.image.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _imagerepositories.image.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _imageupdateautomations.image.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _kustomizations.kustomize.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _providers.notification.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _receivers.notification.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _flux-system__Namespace
      v: v1
    - id: flux-system_helm-controller__ServiceAccount
      v: v1
    - id: flux-system_image-automation-controller__ServiceAccount
      v: v1
    - id: flux-system_image-reflector-controller__ServiceAccount
      v: v1
    - id: flux-system_kustomize-controller__ServiceAccount
      v: v1
    - id: flux-system_notification-controller__ServiceAccount
      v: v1
    - id: flux-system_source-controller__ServiceAccount
      v: v1
    - id: _crd-controller_rbac.authorization.k8s.io_ClusterRole
      v: v1
    - id: _flux-edit_rbac.authorization.k8s.io_ClusterRole
      v: v1
    - id: _cluster-reconciler_rbac.authorization.k8s.io_ClusterRoleBinding
      v: v1
    - id: _crd-controller_rbac.authorization.k8s.io_ClusterRoleBinding
      v: v1
    - id: flux-system_notification-controller__Service
      v: v1
    - id: flux-system_source-controller__Service
      v: v1
    - id: flux-system_webhook-receiver__Service
      v: v1
    - id: flux-system_helm-controller_apps_Deployment
      v: v1
    - id: flux-system_image-automation-controller_apps_Deployment
      v: v1
    - id: flux-system_image-reflector-controller_apps_Deployment
      v: v1
    - id: flux-system_kustomize-controller_apps_Deployment
      v: v1
    - id: flux-system_notification-controller_apps_Deployment
      v: v1
    - id: flux-system_source-controller_apps_Deployment
      v: v1
    - id: flux-system_allow-egress_networking.k8s.io_NetworkPolicy
      v: v1
    - id: flux-system_allow-scraping_networking.k8s.io_NetworkPolicy
      v: v1
    - id: flux-system_allow-webhooks_networking.k8s.io_NetworkPolicy
      v: v1
  lastAppliedRevision: dev/36eef2ff32851ee0332d5ce656cc337eec5c43e5
  lastAttemptedRevision: dev/36eef2ff32851ee0332d5ce656cc337eec5c43e5
  lastHandledReconcileAt: "2022-01-11T08:14:23.902964+01:00"
  observedGeneration: 10
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  annotations:
    reconcile.fluxcd.io/requestedAt: "2022-01-11T08:14:23.902964+01:00"
  creationTimestamp: "2021-09-20T09:22:40Z"
  finalizers:
  - finalizers.fluxcd.io
  generation: 11
  labels:
    kustomize.toolkit.fluxcd.io/name: flux-self-management
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  managedFields:
  - apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          f:kustomize.toolkit.fluxcd.io/name: {}
          f:kustomize.toolkit.fluxcd.io/namespace: {}
      f:spec:
        f:interval: {}
        f:patches: {}
        f:path: {}
        f:prune: {}
        f:sourceRef:
          f:kind: {}
          f:name: {}
    manager: kustomize-controller
    operation: Apply
    time: "2022-01-17T15:49:14Z"
  - apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        f:conditions: {}
        f:lastAppliedRevision: {}
        f:lastAttemptedRevision: {}
        f:observedGeneration: {}
    manager: kustomize-controller
    operation: Update
    time: "2022-01-17T15:45:04Z"
  name: flux-install
  namespace: flux-system
  resourceVersion: "488753002"
  uid: 6619e059-068d-4b84-a63e-3b54f77802ae
spec:
  force: false
  interval: 5m0s
  patches:
  - patch: |-
      apiVersion: v1
      kind: Namespace
      metadata:
        name: flux-system
        annotations:
          redacted: redacted
    target:
      kind: Namespace
      name: flux-system
  patchesStrategicMerge:
  - apiVersion: v1
    kind: Namespace
    metadata:
      annotations:
        redacted: redacted
      name: flux-system
  path: ./infrastructure/flux
  prune: true
  sourceRef:
    kind: GitRepository
    name: fleet-repo
  validation: client
status:
  conditions:
  - lastTransitionTime: "2022-01-17T15:49:39Z"
    message: 'Applied revision: TECH-596/20db32e2eb78fc06d1b32d79e47aba5cb1c0a44e'
    reason: ReconciliationSucceeded
    status: "True"
    type: Ready
  inventory:
    entries:
    - id: _alerts.notification.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _buckets.source.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _gitrepositories.source.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _helmcharts.source.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _helmreleases.helm.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _helmrepositories.source.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _imagepolicies.image.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _imagerepositories.image.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _imageupdateautomations.image.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _kustomizations.kustomize.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _providers.notification.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _receivers.notification.toolkit.fluxcd.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _flux-system__Namespace
      v: v1
    - id: flux-system_helm-controller__ServiceAccount
      v: v1
    - id: flux-system_image-automation-controller__ServiceAccount
      v: v1
    - id: flux-system_image-reflector-controller__ServiceAccount
      v: v1
    - id: flux-system_kustomize-controller__ServiceAccount
      v: v1
    - id: flux-system_notification-controller__ServiceAccount
      v: v1
    - id: flux-system_source-controller__ServiceAccount
      v: v1
    - id: _crd-controller_rbac.authorization.k8s.io_ClusterRole
      v: v1
    - id: _flux-edit_rbac.authorization.k8s.io_ClusterRole
      v: v1
    - id: _cluster-reconciler_rbac.authorization.k8s.io_ClusterRoleBinding
      v: v1
    - id: _crd-controller_rbac.authorization.k8s.io_ClusterRoleBinding
      v: v1
    - id: flux-system_notification-controller__Service
      v: v1
    - id: flux-system_source-controller__Service
      v: v1
    - id: flux-system_webhook-receiver__Service
      v: v1
    - id: flux-system_helm-controller_apps_Deployment
      v: v1
    - id: flux-system_image-automation-controller_apps_Deployment
      v: v1
    - id: flux-system_image-reflector-controller_apps_Deployment
      v: v1
    - id: flux-system_kustomize-controller_apps_Deployment
      v: v1
    - id: flux-system_notification-controller_apps_Deployment
      v: v1
    - id: flux-system_source-controller_apps_Deployment
      v: v1
    - id: flux-system_allow-egress_networking.k8s.io_NetworkPolicy
      v: v1
    - id: flux-system_allow-scraping_networking.k8s.io_NetworkPolicy
      v: v1
    - id: flux-system_allow-webhooks_networking.k8s.io_NetworkPolicy
      v: v1
  lastAppliedRevision: TECH-596/20db32e2eb78fc06d1b32d79e47aba5cb1c0a44e
  lastAttemptedRevision: TECH-596/20db32e2eb78fc06d1b32d79e47aba5cb1c0a44e
  lastHandledReconcileAt: "2022-01-11T08:14:23.902964+01:00"
  observedGeneration: 11

We also had a successful run on the controller switching out the the two variants of patches by

  • install the new RC
  • revert git commit from patches format back to patchesStrategicMerge
  • re-add the commit to switch from patchesStrategicMerge to patches.

Please let us know if we can be of any more assistance

@ericjohansson89
Copy link

So we managed to work around the issue with stale patchesStrategicMerge by doing the following steps
in the Kustomization spec do the following

spec:
  force: false
  interval: 5m0s
  patches:
  - patch: |-
      apiVersion: v1
      kind: Namespace
      metadata:
        name: flux-system
        annotations:
          redacted: redacted
    target:
      kind: Namespace
      name: flux-system
  patchesStrategicMerge: <-- set to []
  - apiVersion: v1
    kind: Namespace
    metadata:
      annotations:
        redacted: redacted
      name: flux-system
  path: ./infrastructure/flux
  prune: true
  sourceRef:
    kind: GitRepository
    name: fleet-repo

So the way to proceed is to

  • Upgrade to a version of the kustomize-controller which contains this fix @stefanprodan does it work on earlier versions as well? in a commit and let the controller update.
  • Update your Kustomizations to contain patches instead of patchesStrategicMege while setting the patchesStrategicMege: [] in a git commit.
  • Push the commit and let the the kustomize controller reconcile your Kustomizations
  • In a new commit remove the section of patchesStrategicMerge from your Kustomization objects and it should now only contain patches section in your yaml once it is pushed and reconciled in the cluster.

@stefanprodan stefanprodan force-pushed the ssa-override-kubectl-manager branch 6 times, most recently from 0d8a732 to ad94bd6 Compare January 21, 2022 14:24
@stefanprodan
Copy link
Member Author

@ericjohansson89 with the latest version of this PR (ghcr.io/fluxcd/kustomize-controller:rc-ad94bd61) the workaround is no longer needed, the controller will now properly remove the fields.

@stefanprodan stefanprodan marked this pull request as ready for review January 21, 2022 15:37
@kingdonb
Copy link
Member

I can confirm, rc-ad94bd61 after testing extensively resolves many different forms of the issue

After testing several RCs this one looks like the winner 🎉

@kingdonb
Copy link
Member

kingdonb commented Jan 21, 2022

Edit: cross stuff out I described a problem but it was not related to the PR, sorry for the noise
tl;dr: if you are upgrading from Flux v0.17.2 or below, make sure you remember to upgrade all your Flux Kustomizations to v1beta2 or else you will experience the same notification spam issue I did. I was testing the upgrade to SSA which required starting with a Flux version that came before SSA. When I finished upgrading properly, including upgrading all of my v1beta1 Flux Kustomizations to v1beta2, the notifications noise from flux-system went away.

🎉 user error, no problem found here

I have not noticed any other strange behavior with any other Kustomization, and I have about 25 different Kustomizations running on this cluster, so while that's not conclusive, it looks like all the issues I was able to reproduce with prior versions, were fixed by this RC build.

That includes:

  • annotations which could not be individually or collectively removed by Flux from any resources, (with or without removing the whole annotations map)
  • map entries from other places in arbitrary resources which could not be removed by Flux
  • array entries from arbitrary resources, or from deployment podspec containers array which could not be removed by Flux

If you're still experiencing difficulty removing any of the above, or any that I missed, after installing this Kustomize Controller RC, please let us know too.

@kingdonb
Copy link
Member

I have performed more exhaustive tests against e611de4 with a matrix of Kubernetes minor versions in our support matrix and I believe this resolves the issue from all angles, 👍 LGTM

Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
@stefanprodan stefanprodan merged commit 8e74f2c into main Jan 31, 2022
@stefanprodan stefanprodan deleted the ssa-override-kubectl-manager branch January 31, 2022 10:01
@stefanprodan stefanprodan added the area/server-side-apply SSA related issues and pull requests label Feb 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/server-side-apply SSA related issues and pull requests enhancement New feature or request
Projects
None yet
5 participants