Experimental AppSet progressive sync capability

[wmgroot]

Background

We are currently implementing complex automated git tagging and directory structures in order to work around the lack of ability to limit the rollout options for updating Application resources managed by an ApplicationSet. This leads to increased complexity in our repositories and increases the chance for human error or edge cases in the tooling we're writing.

This PR is an attempt to move toward supporting more fine grained options for orchestrating when Applications are updated in response to a change in an ApplicationSet spec. The basic approach boils down to allowing a user to provide a list of Application label selectors in the order they would like the Applications to be updated. Each set of Applications must become 'Ready' in the K8s API before the AppSet reconciler moves on to updating the next set of Applications.

I'm open to alternate ideas, but I believe it makes sense to include the update strategy logic within the ApplicationSet spec, the same way a Deployment Rollout resource is responsible for the strategy used to cycle its pods.

Prior Art & Issues

argoproj/argo-cd#7437
argoproj#61
argoproj/argo-cd#1283

https://github.com/maruina/argocd-progressive-rollout-controller (abandoned)
https://github.com/Skyscanner/applicationset-progressive-sync (pre-alpha)

TODO

• Tests
• Refactor against new ApplicationSet controller merged into the ArgoCD repo
• Consensus on an ApplicationSet spec (current config is hardcoded for demo purposes)
• Refactor to reduce calls to retrieve lists of Applications
• Refactor to reduce annotation juggling (possibly by moving state into AppSet status)
• Find an improved way to detect that an Application has a Sync/Progressing state pending

Results

Tested with a simple nginx pod configured with a 2m readiness probe delay to simulate an application taking time to become ready. A single commit updating an annotation on the pod spec rolled out each Application in series, waiting for the prior Application to become 'Ready'.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: hello-world-nginx
spec:
  generators:
  // ... matched some clusters
  template:
    metadata:
      name: 'hello-world-nginx-{{name}}'
      labels:
        rolloutWave: "{{metadata.labels.cluster/rolloutWave}}"
    spec:
      project: hello-world-nginx
      source:
        // simple helm chart with a single pod nginx Deployment
        helm:
          releaseName: hello-world-nginx
          values: |-
            clusterName: {{name}}
            deployment:
              template:
                annotations:
                  test: "26"  # updated this annotation to produce a pod rollout across multiple Applications
      destination:
        name: '{{name}}'
        namespace: hello-world-nginx--demo
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

$ kubectl get pod -n hello-world-nginx--demo --context cluster1
NAME                                 READY   STATUS    RESTARTS   AGE
hello-world-nginx-74f689ff9d-99jxj   1/1     Running   0          5m17s

$ kubectl get pod -n hello-world-nginx--demo --context cluster2
NAME                                 READY   STATUS    RESTARTS   AGE
hello-world-nginx-74f689ff9d-wgsrk   1/1     Running   0          3m3s