VirtualService not being cleaned when running a new deployment before the old one finishes

[lionelresnik]

Describe the bug
When using Headers traffic with Istio and applying a new rollout before the old deployment hasn't finished, then the VirtualService will not be cleaned from the http--> match--> headers, thus causing the new deployment to never finish, as the VirtualService being filled(till end of file) with the headers again and again.

To Reproduce

create the following rollout:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: 
  name: spotdeployment-headers-demo3
  namespace: lio-headers-test-bug
spec:
  replicas: 2
  selector:
    matchLabels:
      app: rollout-test-lio
  strategy:
    canary:
      canaryMetadata:
        annotations:
          oceancd.spot.io/replica-version: latest
      canaryService: canary-service-headers-d
      stableMetadata:
        annotations:
          oceancd.spot.io/replica-version: stable
      stableService: stable-service-headers-d
      steps:
        - setWeight: 26
        - setHeaderRoute:
            match:
              - headerName: User-Agent
                headerValue:
                  regex: (.*)lio(.*)
            name: set-header-1
        - pause: {}
        - setWeight: 50
        - setHeaderRoute:
            match:
              - headerName: beta-tester
                headerValue:
                  exact: enabled
            name: set-header-2
        - pause:
            duration: 2m
        - setWeight: 100
      trafficRouting:
        istio:
          virtualServices:
            - name: rollout-vsvc
              routes:
                - primary
        managedRoutes:
          - name: set-header-1
          - name: set-header-2
  template:
    metadata:
      annotations:
        oceancd.spot.io/managed-by: spotdeployment-headers-demo3
      labels:
        app: rollout-test-lio
    spec:
      containers:
        - image: argoproj/rollouts-demo:purple
          name: rollout-test-lio
          ports:
            - containerPort: 8080
              name: http
              protocol: TCP
          resources: {}

As you can see the Strategy contains:
setWeight
setHeaderRoute
pause
setWeight
setHeaderRoute
pause
setWeight

strategy:
   canary:
     canaryMetadata:
       annotations:
         oceancd.spot.io/replica-version: latest
     canaryService: canary-service-headers-d
     stableMetadata:
       annotations:
         oceancd.spot.io/replica-version: stable
     stableService: stable-service-headers-d
     steps:
       - setWeight: 26
       - setHeaderRoute:
           match:
             - headerName: User-Agent
               headerValue:
                 regex: (.*)lio(.*)
           name: set-header-1
       - pause: {}
       - setWeight: 50
       - setHeaderRoute:
           match:
             - headerName: beta-tester
               headerValue:
                 exact: enabled
           name: set-header-2
       - pause:
           duration: 2m
       - setWeight: 100
     trafficRouting:
       istio:
         virtualServices:
           - name: rollout-vsvc
             routes:
               - primary
       managedRoutes:
         - name: set-header-1
         - name: set-header-2

if you deploy a new rollout(same but with different image) in the second pause, let it run, and promote the rollout till the second
pause will cause to reproduce the issue.

Expected behavior
Once a new rollout is applied the VirtualService should be cleaned from old headers.
If im correct you should invoke
reconciler.RemoveManagedRoutes() as well when creating new.

Screenshots

Version

Latest (argo-rollouts:v1.5.0) tried with older versions as well.

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

[github-actions]

This issue is stale because it has been open 60 days with no activity.