Rollout does not scale preview service properly on changes

[jessesuen]

We have a v0.3.1 Rollout in the following condition:

The bottom rollout is actually the desired spec as a preview, and has 3 replicas. However, the rollout spec has 2 desired replicas and currently has the following error condition:

Rollout "partner-support-portal" has timed out progressing.

[jessesuen]

Simplified rollout object:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  annotations:
    rollout.argoproj.io/revision: '16'
  generation: 1
  labels:
    app: partner-support-portal
  name: partner-support-portal
spec:
  minReadySeconds: 30
  paused: false
  replicas: 2
  selector:
    matchLabels:
      app: partner-support-portal
  strategy:
    blueGreen:
      activeService: partner-support-portal-active
      previewService: partner-support-portal-preview
  template:
    metadata:
      labels:
        app: partner-support-portal
    spec:
      containers:
        - image: someimage:sometag
          name: partner-support-portal
status:
  HPAReplicas: 2
  availableReplicas: 5
  blueGreen:
    activeSelector: 5f59c4c4dc
    previewSelector: 7c67b5fd8d
  canary: {}
  conditions:
    - lastTransitionTime: '2019-05-20T07:22:41Z'
      lastUpdateTime: '2019-05-20T07:22:41Z'
      message: Rollout has minimum availability
      reason: AvailableReason
      status: 'True'
      type: Available
    - lastTransitionTime: '2019-05-20T11:02:00Z'
      lastUpdateTime: '2019-05-20T11:02:00Z'
      message: Rollout "partner-support-portal" has timed out progressing.
      reason: ProgressDeadlineExceeded
      status: 'False'
      type: Progressing
  currentPodHash: 6756d4c4c7
  observedGeneration: 5745f7fc9c
  readyReplicas: 5
  replicas: 5
  selector: 'app=partner-support-portal,rollouts-pod-template-hash=5f59c4c4dc'

Simplified replicaset

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  annotations:
    rollout.argoproj.io/desired-replicas: '3'
    rollout.argoproj.io/revision: '16'
    rollout.argoproj.io/revision-history: '14'
  generation: 1
  labels:
    app: partner-support-portal
  name: partner-support-portal-7c67b5fd8d
  ownerReferences:
    - apiVersion: argoproj.io/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: Rollout
      name: partner-support-portal
spec:
  minReadySeconds: 30
  replicas: 3
  selector:
    matchLabels:
      app: partner-support-portal
      rollouts-pod-template-hash: 7c67b5fd8d
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: partner-support-portal
        rollouts-pod-template-hash: 7c67b5fd8d
    spec:
      containers:
        - image: somerepo:sometag
          name: partner-support-portal
      restartPolicy: Always
status:
  availableReplicas: 3
  fullyLabeledReplicas: 3
  observedGeneration: 1
  readyReplicas: 3
  replicas: 3

[jessesuen]

It may have something to do with the change in replicas (scaling down from 3 to 2). We are hitting the following:

func (c *Controller) isScalingEvent(rollout *v1alpha1.Rollout, rsList []*appsv1.ReplicaSet) (bool, error) {
	if rollout.Spec.Strategy.CanaryStrategy != nil {
		return false, nil
	}
	newRS, previousRSs, err := c.getAllReplicaSetsAndSyncRevision(rollout, rsList, false)
	if err != nil {
		return false, err
	}

	allRSs := append(previousRSs, newRS)

	for _, rs := range controller.FilterActiveReplicaSets(allRSs) {
		desired, ok := annotations.GetDesiredReplicasAnnotation(rs)
		if !ok {
			continue
		}
		if desired != defaults.GetRolloutReplicasOrDefault(rollout) {
			return true, nil               // we are hitting this logic
		}
	}
	return false, nil
}

[jessesuen]

Tracing through the code, it appears we don't have code to handle the scaling of the preview service during a scaling event. Have a potential fix.

[jessesuen]

The bug is that we detect a scaling event on a preview replicaset, and so we enter the scaleBlueGreen() block of code, but the current logic does nothing to correct the replicas of the preview replicaset, thereby getting stuck in a perpetual loop and never getting to the actual rollout logic.