Add Progressive Rollout Strategy for ApplicationSet resources

[cnmcavoy]

Summary

Enhance the ArgoCD ApplicationSet resource to embed a rollout strategy for a progressive application resource update after the ApplicationSet spec or Application templates are modified.

Motivation

As cluster operators, we would like to make changes to ApplicationSets which may target multiple environments, pre-defined staging areas, or other configurations, and have these changes rolled out in a declarative, defined manner rather than all at once as ApplicationSets currently behave. A progressive ApplicationSet rollout would prevent mistakes in configuration from having a larger blast radius than intended and give cluster operators a chance to verify and have confidence in their changes.

Proposal

As an initial proposal, two rollout strategies are suggested:

• AllAtOnce (Replace?)
  • The current application set rollout strategy behavior (default)
  • No strategy spec
• RollingUpdate
  • The rolling update strategy deterministically chooses applications to update one by one, proceeding each step only if the previous application syncs completed successfully, and respecting the maxUpdate value.
  • Steps for the rolling update are defined by a list of matchExpression label selectors. Each step must finish updating before the next step advances.
  • If steps are left undefined the application update order is deterministic.

ApplicationSet CRD Updates

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: guestbook
spec:
  generators:
  - list:
      elements:
      - cluster: engineering-dev
        url: https://1.2.3.4
      - cluster: engineering-prod
        url: https://2.4.6.8
      - cluster: finance-preprod
        url: https://9.8.7.6/
  strategy:
    type: RollingUpdate
    rollingUpdate:
      steps:
        - matchExpressions:
            - key: env
              operator: In
              values: 
                - dev
          maxUpdate: 0 # if undefined or 0, all applications matched are updated together
        - matchExpressions:
            - key: env
              operator: In
              values: 
                - qa
        - matchExpressions:
            - key: env
              operator: In
              values:
                - us_east_2
                - eu_west_1
                - ap_south_1
          maxUpdate: 1 # maxUpdate supports both integer and percentage string values
  template:
    metadata:
      name: '{{cluster}}-guestbook'
    spec:
      source:
        repoURL: https://github.com/infra-team/cluster-deployments.git
        targetRevision: HEAD
        path: guestbook/{{cluster}}
      destination:
        server: '{{url}}'
        namespace: guestbook

In the above example, when the guestbook ApplicationSet is created or modified, the Application resources are each updated in the order defined in the rollout strategy. In this case, all Applications whose desired resource state matches the label expression env: dev are updated to match the template, as the maxUpdate is set to zero. The rolling update strategy progresses after the first set of Applications has successfully completed. Progress towards the next step starts only after the current step has completely finished, regardless of the maxUpdate value. The maxUpdate field only throttles the total number of matching Applications updating in the current step. After the first step completes, the ApplicationSet updates Application resources matching the second step's matchExpression together, as the maxUpdate was undefined. Finally, after the third step, the Application resources matching are updated, one by one, as the maxUpdate for the final step is 1.

An Application rollout is considered “complete” when the Application resource has been:

• Synced successfully.
• Moved into a “Progressing” state.
• Moved out of a “Progressing” state and into a “Healthy” state.

ApplicationSet creation with rollout strategy

Application resource creation from an ApplicationSet with a defined strategy looks much like the update process. When a brand new ApplicationSet is first created with a rollout strategy specified, the desired Application resource metadata labels are used to determine when each Application resource is created. Each Application created will be created in the order defined by the steps, if any, and advance to the next step only when a step completes successfully. The same applies if an ApplicationSet is modified to target a different set of destination clusters or namespaces, Applications are created or updated in the order defined by their desired state and the defined step order in the strategy.

ApplicationSet rollout failure

In the event that an ApplicationSet spec or template is modified and a target Application resource fails to “complete” a sync in any of the steps, the ApplicationSet rollout is stalled. The ApplicationSet resource will ensure the status field for “ApplicationSetUpToDate” is False. If the maxUpdate allows it, the ApplicationSet will continue updating Applications in the current step, but otherwise, no further changes will be propagated to Application resources by the ApplicationSet, and no steps will advance until each Application can successfully complete a sync. If the ApplicationSet is modified while still in the midst of an ApplicationSet rollout, stalled or otherwise, then the existing rollout is abandoned, the application resources are left in their present state, and the new rollout begins.

Pausing Application Changes

To implement the “paused” functionality of Applications that are not yet ready to be updated, we have a few options.

• Disable auto-sync.
  • Potentially conflicts with user provided auto-sync settings.
  • Provides the benefit of being able to see the full diff of the ApplicationSet change.
• “Pause” the Application.
  • Not Yet Implemented: Provide a way to pause syncing an application #4808
• Prevent any updates at all to the live Applications via the rolling update strategy defined.
  • Example implementation: Experimental AppSet progressive sync capability wmgroot/applicationset#1

Proof of Concept Implementation

We believe that the proof of concept implementation mentioned in option #3 above also demonstrates the strategy concept is firstmost, possible with minimal changes, and secondly, identifies the most likely code locations to implement the feature. The proof of concept utilizes hard-coded labels and annotations as a method of proving the idea can work, not as a guide for the implementation. A summary of the changes are included below:

• Retrieve a complete list of all current Applications.
• Build a map of which Applications belong in each rollout step.
  • This map includes the decision for whether or not each Application rollout step has been completed.
  • The logic to determine which Applications have already been updated depends on state tracked in the ApplicationSet status field, or in annotations set on each Application resource.
• Remove Applications from the list of desired Application updates.
  • Determine the rollout step for each Application.
  • If the previous rollout step is not yet complete, remove the current Application from the list of Applications to update, preventing the live Application from changing.

A full implementation can be built after the enhancement proposal has been approved and accepted by the community.
Co-written w/@wmgroot.

[romuduck]

Hi both, isn't it similar to
https://github.com/Skyscanner/applicationset-progressive-sync

++

[wmgroot]

Yes, this is a formal enhancement proposal to officially implement a solution solving the same problems that skyscanner/applicationset-progressive-sync is looking to solve.

[wmgroot]

Contribution guidelines recommend that we open an Enhancement Proposal before implementing any PRs. This EP has now been open for a month with no feedback or guidance from the project maintainers.

Should we continue to hold off on writing any formal PRs for this capability, or can we start working on this feature?

[romuduck]

same question
And we could help contributing too to speed it all up if needed

[snehagupta-coder]

To speedup I think its possible to put a topic in Agenda here and participate in weekly meeting:
https://docs.google.com/document/d/1xkoFkVviB70YBzSEa4bDnu-rUZ1sIFtwKKG1Uw8XsY8/edit#

[romuduck]

hello @wmgroot , @cnmcavoy
Do you think we could consider adding validation steps in the middle, with or without pause, with API/prometheus metrics/... as per argo-rollout? (maybe just an API check for MVP)
Also, something Im not sure to understand: if a new commit should update 3 apps progressively (sequentially), and 2nd app rollout failed. What will we see in the face of all 3 apps?

• 1st app: new commit, synced OK
• 2nd app: new commit, synced NOK / unhealthy
• 3rd app: new commit or old commit? synced OK or NOK?

[romuduck]

Also the more I think about it, the more I think what we need is to kind of extend argo-rollout from an application POD update via canary/blue-green to a set of applications. After all, these are all meant to minimize risk of deploying everything at once.
Btw, even at app level, id love to see argo-rollout perform "analysis" for any app change (not just PODs).

[crenshaw-dev]

@wmgroot apologies for the slow response. It's a well-written proposal! Generally for larger proposals we look for a PR adding a Markdown file modeled on this template. Using a PR makes it easier to have threads discussing different areas. It also gives us an easy way to "accept" a proposal (by merging it).

[wmgroot]

@romuduck The proposed implementation would be able to lean on the existing argo-rollouts feature set, since the proposal would respect the rollout process of the argo-rollouts controller before a progressing Application resource becomes "ready".

If your example, the 3rd Application would remain unchanged until the 2nd Application became "ready". The applicationset controller would not update the live Application resource on the cluster, which would mean that the 3rd Application's spec would differ from what is potentially tracked in Git for the owning ApplicationSet resource, assuming a GitOps process is used.

[romuduck]

@wmgroot what i meant to say is that argo-rollout consists in:

• a progressive delivery mechanism at replicaset/pod level from which part of the logic (sequencing, pausing) could be used as is
• an analysis layer which would be great if we could use all the time (after an application change rollout, not just deployment.yaml update, as well as during each step of an applicationset progressive rollout).

Hence there s probably some common code...
But I'm not familiar with argo-rollout code :-)

[sidewinder12s]

We're also very interested in this from a cluster operations perspective.

[crenshaw-dev]

If anyone wants to push this, most of the work would be copy/paste into the proposal template, filling in some empty sections, and then opening a PR to collect comments.

[wmgroot]

I've been working on the actual PR with the changes to implement this functionality, I'll go ahead and open a proposal PR using the formal proposal template.

[crenshaw-dev]

Nice, thanks!

[wmgroot]

Formal proposal PR has been added here: #9979

[crenshaw-dev]

I'll be reviewing and pushing to get this as an alpha feature into 2.6.

[adabuleanu]

Hello. Great job on this feature. I have two questions:

• Is there any way to "link" this feature on multiple ApplicationSets? Like for example I have two appsets (mostly because the generated apps must have a different definition and templating inside appsets is not flexible enough to generate the apps from one appset -> reference ticket #11213 ) and I want to have order of installation between all of the apps
• Does this feature only apply at install time? What will happen on deletion? In my experience order really matters when you try to cleanup apps (e.g. you have a aws-lb-controller app and if this app is deleted all apps that have an ingress/service resource pointing to the ALB will be stuck)