Infinite loop when multiple stage targeting the same clusters

[maruina]

We got an infinite loop with the following ProgressiveSync

spec:
[...]
  stages:
  - maxParallel: 1
    maxTargets: 1
    name: one cluster as canary in eu-west-1
    targets:
      clusters:
        selector:
          matchLabels:
            region: eu-west-1
  - maxParallel: 3
    maxTargets: 3
    name: one cluster as canary in every other region
    targets:
      clusters:
        selector:
          matchExpression:
          - key: region
            operator: NotIn
            values:
            - eu-west-1
    name: rollout to remaining clusters
  - maxParallel: 25%
    maxTargets: 100%
    targets:
      clusters:
        selector: {}

After some investigation, we found the following issues:

• The scheduler relies on an annotation to determine when an Application has been synced (see https://github.com/Skyscanner/applicationset-progressive-sync/blob/main/internal/scheduler/scheduler.go#L24). While we check for the annotation, we never set it. Because of that, we never hit this code path: https://github.com/Skyscanner/applicationset-progressive-sync/blob/main/internal/scheduler/scheduler.go#L59
• We always re-write the stage status, but looking at https://github.com/Skyscanner/applicationset-progressive-sync/blob/main/api/v1alpha1/progressivesync_types.go#L107:28 we always overwrite the StartedAt and FinishedAt time
• We didn't catch this scenario in our test because our test at https://github.com/Skyscanner/applicationset-progressive-sync/blob/main/controllers/progressivesync_controller_test.go#L268 has a label selector so we return only one cluster per stage. We should change the second stage to have a label selector to include the first cluster as well.