A way to cause resources to be deleted and recreated

[mikebryant]

Summary

There should be a way to specify that resources should be deleted and recreated on a change to all/part of the spec

Motivation

Some resources can't be changed in place, e.g.

        Group:       iam.cnrm.cloud.google.com
        Hook Phase:  Failed
        Kind:        IAMPolicyMember
        Message:     admission webhook "deny-immutable-field-updates.cnrm.cloud.google.com" denied the request: the IAMPolicyMember's spec is immutable
        Name:       key-access
        Namespace:   default
        Status:      SyncFailed
        Sync Phase:  Sync
        Version:     v1beta1

Proposal

Add an option to the argocd.argoproj.io/sync-options annotation for this behaviour?

[alexmt]

We've recently introduced Replace=true sync option: https://argo-cd.readthedocs.io/en/latest/user-guide/sync-options/#replace-resource-instead-of-applying-changes . The kubectl replace works this way: if update fails it deletes and recreates deployment. The feature is available in v2.0-rc1

[sgielen]

@alexmt What I'm still missing in the documentation you linked: Can the Replace=true option be added into the argocd.argoproj.io/sync-options annotation for a single resource as well?

[sgielen]

I just performed the following test:

Add this Pod to ArgoCD:

apiVersion: v1
kind: Pod
metadata:
  namespace: default
  name: argocd-pod-test
spec:
  containers:
  - command: ['sh', '-c', 'date; sleep 5; date; sleep 3600']
    image: ubuntu:bionic
    name: default

Wait until it is running, then change the manifest to this:

apiVersion: v1
kind: Pod
metadata:
  namespace: default
  name: argocd-pod-test
  annotations:
    argocd.argoproj.io/sync-options: Replace=true
spec:
  containers:
  - command: ['sh', '-c', 'date; sleep 5; date; sleep 3600']
    image: ubuntu:xenial # <-- changed
    name: default
    env: # <-- changed
    - name: TEST_ENV
      value: TEST_VALUE

Unfortunately, the change fails to apply because Kubernetes responds with Forbidden: pod updates may not change fields other than ``spec.containers[*].image``., so ArgoCD attempted to update the Pod instead of Replacing it according to annotations...

[alexmt]

Sorry for delay in response @sgielen . You added annotation to the Pod spec stored in Git, correct? Which Argo CD version are you using?

[mmerrill3]

Hi @alexmt I tried this to, by just updating a deployment (in github) with version 2.0.2 of argocd. I added the annotation for Replace=true to my deployment, and added a new label to the selector, but that still doesn't work. argo give the error

error when replacing "/dev/shm/105286521": Deployment.apps "gatekeeper-controller-manager" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app":"gatekeeper", "chart":"gatekeeper", "control-plane":"controller-manager", "gatekeeper.sh/operation":"webhook", "gatekeeper.sh/system":"yes", "heritage":"tiller", "mike":"1", "release":"docker-desktop-apps"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

Desired manifest from the argcd UI:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    argocd.argoproj.io/sync-options: Replace=true
  labels:
    app: gatekeeper
    app.kubernetes.io/instance: docker-desktop-apps
    chart: gatekeeper
    control-plane: controller-manager
    gatekeeper.sh/operation: webhook
    gatekeeper.sh/system: 'yes'
    heritage: tiller
    release: docker-desktop-apps
  name: gatekeeper-controller-manager
  namespace: gatekeeper
spec:
  replicas: 3
  selector:
    matchLabels:
      app: gatekeeper
      chart: gatekeeper
      control-plane: controller-manager
      gatekeeper.sh/operation: webhook
      gatekeeper.sh/system: 'yes'
      heritage: tiller
      mike: '1'
      release: docker-desktop-apps
  template:
    metadata:
      annotations:
        container.seccomp.security.alpha.kubernetes.io/manager: runtime/default
      labels:
        app: gatekeeper
        chart: gatekeeper
        control-plane: controller-manager
        gatekeeper.sh/operation: webhook
        gatekeeper.sh/system: 'yes'
        heritage: tiller
        mike: '1'
        release: docker-desktop-apps
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: gatekeeper.sh/operation
                      operator: In
                      values:
                        - webhook
                topologyKey: kubernetes.io/hostname
              weight: 100
      automountServiceAccountToken: true
      containers:
        - args:
            - '--port=8443'
            - '--logtostderr'
            - '--log-denies=false'
            - '--emit-admission-events=false'
            - '--log-level=INFO'
            - '--exempt-namespace=gatekeeper'
            - '--operation=webhook'
            - '--enable-mutation=false'
          command:
            - /manager
          env:
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
          image: 'openpolicyagent/gatekeeper:v3.5.0-rc.1'
          imagePullPolicy: IfNotPresent
          livenessProbe:
            httpGet:
              path: /healthz
              port: 9090
          name: manager
          ports:
            - containerPort: 8443
              name: webhook-server
              protocol: TCP
            - containerPort: 8888
              name: metrics
              protocol: TCP
            - containerPort: 9090
              name: healthz
              protocol: TCP
          readinessProbe:
            httpGet:
              path: /readyz
              port: 9090
          resources:
            limits:
              cpu: 1000m
              memory: 512Mi
            requests:
              cpu: 100m
              memory: 256Mi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - all
            readOnlyRootFilesystem: true
            runAsGroup: 999
            runAsNonRoot: true
            runAsUser: 1000
          volumeMounts:
            - mountPath: /certs
              name: cert
              readOnly: true
      hostNetwork: false
      imagePullSecrets: []
      nodeSelector:
        kubernetes.io/os: linux
      priorityClassName: system-cluster-critical
      serviceAccountName: gatekeeper-admin
      terminationGracePeriodSeconds: 60
      tolerations: []
      volumes:
        - name: cert
          secret:
            defaultMode: 420
            secretName: gatekeeper-webhook-server-cert

[mmerrill3]

I don't think this option is setup for individual resources, but rather only at the application CRD level. I don't see this annotation getting read, or used, anywhere.

[no-response]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

[mikebryant]

/reopen

[renaudguerin]

Like I'm @mikebryant I have some immutable GCP ConfigConnector resources (DNSRecordSet in dns.cnrm.cloud.google.com/v1beta1) where even "Replace" doesn't seem to be good enough :

error when replacing "/dev/shm/438125077": admission webhook "deny-immutable-field-updates.cnrm.cloud.google.com" denied the request: annotation cnrm.cloud.google.com/state-into-spec is immutable

As @mmerrill3 mentioned, a first problem is that "Replace" might not be set up for individual resources, but I'm getting the sync error above when triggering a manual sync with Replace for this resource in the UI.

Is there a need for a "delete + apply" sync option for when "replace" won't do, or am I missing something ?

[renaudguerin]

Hi @alexmt, should this be labelled as a bug rather than an enhancement ?

The stable documentation suggests this should be working at the resource level, but it isn't for me (ArgoCD v2.1.1+aab9542) :

error when replacing "/dev/shm/515177248": admission webhook "deny-immutable-field-updates.cnrm.cloud.google.com" denied the request: annotation cnrm.cloud.google.com/state-into-spec is immutable

Resource :

apiVersion: dns.cnrm.cloud.google.com/v1beta1
kind: DNSRecordSet
metadata:
  name: api
  annotations:
    argocd.argoproj.io/sync-options: Replace=true

[zmx]

DO NOT work for Job

apiVersion: batch/v1
kind: Job
metadata:
  name: styx
  annotations:
    argocd.argoproj.io/sync-wave: "0"
    argocd.argoproj.io/sync-options: Replace=true

{
    "Version": "v2.1.3+d855831",
    "BuildDate": "2021-09-29T21:51:21Z",
    "GitCommit": "d855831540e51d8a90b1006d2eb9f49ab1b088af",
    "GitTreeState": "clean",
    "GoVersion": "go1.16.5",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KsonnetVersion": "v0.13.1",
    "KustomizeVersion": "v4.2.0 2021-06-30T22:49:26Z",
    "HelmVersion": "v3.6.0+g7f2df64",
    "KubectlVersion": "v0.21.0",
    "JsonnetVersion": "v0.17.0"
}

[sgielen]

@alexmt A gentle nudge regarding this issue, your question has been answered and it appears multiple people are running into this :-)

[DanielHajduk2-TomTom]

Just run into the same issue. Cannot change spec of job despite using "Replace=true" syncOption...
What I expected it to do is to remove resource first and then create it again, completely skipping Kubernetes Spec validation issues.

Edit. - For anyone struggling with this issue in case of kubernetes jobs -> calculate a checksum of immutable fields and simply append substr of it to name of the job.

[renaudguerin]

Hi @alexmt, is there anything else blocking this ?
Thanks!

[w4rgrum]

Seems from the UI using force allows the sync of the Job to succeed (delete+recreate).
Is there a way to annotate resources to use "Force"?

The annotation below could be nice to have:

  annotations:
    argocd.argoproj.io/sync-options: Force=true

[renaudguerin]

Happy new year ! Any update on this please ?

[sathieu]

To fix the Job spec being immutable, I added the following annotations in the Helm chart:

apiVersion: batch/v1
kind: Job
metadata:
  name: generate-ca-bundle
  annotations:
    "helm.sh/hook": post-install,post-upgrade
    "helm.sh/hook-delete-policy": before-hook-creation

Which maps to:

apiVersion: batch/v1
kind: Job
metadata:
  name: generate-ca-bundle
  annotations:
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation

Refs:

• https://argo-cd.readthedocs.io/en/stable/user-guide/helm/
• https://argo-cd.readthedocs.io/en/stable/user-guide/resource_hooks/
• inspired by kube-prometheus-stack
• pushed to kubitus-installer

[lamebear]

I needed to have the Job specified in a sync wave, so what I did was create a Job that runs in a PreSync hook that deletes the Job that wouldn't update. The only other change I needed to do was to create a Role and RoleBinding allowing the SA to delete the Job. It's a bit of a hack, but it works.

apiVersion: batch/v1
kind: Job
metadata:
  annotations:    
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
  name: vault-config-remove-job
  namespace: vault
spec:
  template:
    spec:
      serviceAccountName: vault-config
      containers:
      - name: delete-job
        image: bitnami/kubectl:latest
        command: 
          - kubectl
          - delete
          - jobs
          - -n
          - vault
          - vault-config
          - --ignore-not-found=true
      restartPolicy: Never
  backoffLimit: 4

[sathieu]

@lamebear I don't understand why you need to delete the configmap (there is no immutable fields in a configmap)...

[dntosas]

Seems from the UI using force allows the sync of the Job to succeed (delete+recreate). Is there a way to annotate resources to use "Force"?

The annotation below could be nice to have:

  annotations:
    argocd.argoproj.io/sync-options: Force=true

+1

[waylonm]

Hitting the same on ArgoCD v2.2.8 via gitops operator. Testing SyncOption Replace=True on a Job resource without any hooks, both at Argocd Application level and specific resource level via annotation does not work and sync fails due to immutability.

[WorldIsM]

Hi, any updates on this issue? I have similar problem!

[sathieu]

@aldycool Have you tried using a Deployment with spec.strategy.type==Recreate or a stafulset?

[aldycool]

@sathieu Yes you're right, I've tested your approach (Deployment with Recreate) and it works fine, thanks.

I actually knew about this from the kubernetes docs, but this statement threw me off:

This will only guarantee Pod termination previous to creation for upgrades. If you upgrade a Deployment, all Pods of the old revision will be terminated immediately.

I thought that in order to kill the pods, the Deployment manifest must somehow be different than the existing one (to be considered an upgrade), but evidently it works evem though the Deployment is exactly the same as it was.

[ktzsolt]

@aldycool the problem never really was with Deployments but with immutable types like job or cronjob to update their image spec to newer images for example myjobimage:1.0.0 to myjobimage:2.0.0

Any news about my proposed solution on this comment? #5882 (comment)

We need "Force" support in the manifest annotation, not just on the UI:

  annotations:
    argocd.argoproj.io/sync-options: Replace=true, Force=true

[lamebear]

Is there any update on this issue? This issue has been open for almost 2 years at this point.

For those finding this issue now, there are currently three work arounds each with drawbacks:

1. Proposed by @sathieu, place the job in a post install/upgrade hook with a hook delete policy of delete-before-creation. (comment)
   This has the downside of not being able to specify the job in a sync wave.

2. Proposed by myself, use a job in a pre install/upgrade hook to delete the job. (comment)
   This allows the job to be placed into a sync-wave, but requires an additional hook as well as RBAC config to allow the pre-hook job to delete the required job.

3. Proposed by @w4rgrum, update the source helm chart to generate a new name for the job every time it renders and enable pruning in sync-options. (comment)
   This also allows the job to be placed into a sync-wave, but the job name is now no-longer constant and this may not be an option for third-party charts that do not allow overriding the name of the job in this way.

There is a solution proposed by @ktzsolt to allow setting Force=true in the sync-options that would behave as checking the Force box in the UI would. This would allow the syncing of jobs without one of these work-arounds and their associated drawbacks.

[lamebear]

@lamebear I don't understand why you need to delete the configmap (there is no immutable fields in a configmap)...

@sathieu Sorry I didn't see your comment when you wrote it, but my code is not deleting a ConfigMap, it is deleting a Job named vault-config.

[ndlanier]

@aldycool the problem never really was with Deployments but with immutable types like job or cronjob to update their image spec to newer images for example myjobimage:1.0.0 to myjobimage:2.0.0

Any news about my proposed solution on this comment? #5882 (comment)

We need "Force" support in the manifest annotation, not just on the UI:

  annotations:
    argocd.argoproj.io/sync-options: Replace=true, Force=true

Adding my two cents to this issue, I believe this is a much needed feature. I am trying to get secrets to update in a similar fashion. The secrets are created from an application set and when the values are updated they are not shown out of sync so I have to delete the secrets and re-create them whenever they get updated. Have a replace and force annotation to add to these secrets would be ideal.

[brunocascio]

@aldycool the problem never really was with Deployments but with immutable types like job or cronjob to update their image spec to newer images for example myjobimage:1.0.0 to myjobimage:2.0.0
Any news about my proposed solution on this comment? #5882 (comment)
We need "Force" support in the manifest annotation, not just on the UI:

  annotations:
    argocd.argoproj.io/sync-options: Replace=true, Force=true

Adding my two cents to this issue, I believe this is a much needed feature. I am trying to get secrets to update in a similar fashion. The secrets are created from an application set and when the values are updated they are not shown out of sync so I have to delete the secrets and re-create them whenever they get updated. Have a replace and force annotation to add to these secrets would be ideal.

Adding also my 2c here, I need to re-run jobs after a configmap get changed, but the only way of using Replace+Force via UI.
If we can add feature would be awesome...

[divyangjp]

I found a workaround which works for the job only resource type.

• Using helm chart with a job and a dummy configmap
• Job with argocd.argoproj.io/hook: PostSync and argocd.argoproj.io/hook-delete-policy: HookSucceeded annotations
• Job with metadata generateName
• dummy configmap has a data field which populates using job's docker image tag (in our case, new job is dependent on new docker image tag)
• Application with Replace=true

With this setup, everytime new job docker container image is updated, the configmap gets updated as well which triggers PostSync hook.

Example Configmap

apiVersion: v1
kind: ConfigMap
metadata:
  name: docker-tag
data:
  job_docker_tag: {{ .Values.image.tag }}

Partial example of Job resource template

apiVersion: batch/v1
kind: Job
metadata:
  generateName: {{ .Chart.Name }}-
  namespace: {{ .Values.namespace }}
  annotations:
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:

[n9]

@alexmt I have drafted a PR for Force=true argoproj/gitops-engine#526. It needs some refactoring to reduce a cognitive complexity. But is this solution, ok for you?

[ArjonBu]

Any updates on this?

[crenshaw-dev]

Working with @n9 on the prerequisite gitops-engine PR.

[yukccy]

Working with @n9 on the prerequisite gitops-engine PR.

Is there any updates for the Force=true option? Really need a force pod recreation for each ConfigMap change.

[janosmiko]

@yukccy you can achieve that with https://github.com/stakater/Reloader