Deployment with recreate strategy does not remove old replica set

[JorritSalverda]

I want to replace an application by running a new deployment. The image tag however does not change, although the image itself might have (that's the reason for the image pull policy always). For that reason I've added an extra label that changes on every deployment.

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: gocd-agent-${AGENT_NAME}
spec:
  replicas: ${AGENT_REPLICAS}
  strategy:
    type: Recreate
  revisionHistoryLimit: 25
  selector:
    matchLabels:
      app: gocd-agent-${AGENT_NAME}
      release: v${VERSION}
  template:
    metadata:
      labels:
        app: gocd-agent-${AGENT_NAME}
        release: v${VERSION}
    spec:
      containers:
      - name: gocd-agent
        image: ${AGENT_IMAGE}
        imagePullPolicy: Always
        volumeMounts:
        - name: ssh-keys
          mountPath: /var/go/.ssh
        - name: gcloud-keys
          mountPath: /var/go/.gcloud
        - name: docker-sock
          mountPath: /var/run/docker.sock
        - name: docker-bin
          mountPath: /usr/bin/docker
        env:
        - name: "GO_SERVER"
          value: "gocd-server"
        - name: "AGENT_KEY"
          value: "something"
        - name: "AGENT_RESOURCES"
          value: "${AGENT_RESOURCES}"
        - name: "DOCKER_GID_ON_HOST"
          value: "107"
        resources:
          limits:
            cpu: 1000m
            memory: 1024Mi
          requests:
            cpu: 10m
            memory: 512Mi
      volumes:
      - name: ssh-keys
        secret:
          secretName: ssh-keys
      - name: gcloud-keys
        secret:
          secretName: gcloud-keys
      - name: docker-sock
        hostPath:
          path: /var/run/docker.sock
      - name: docker-bin
        hostPath:
          path: /usr/bin/docker

When applying this the deployment is accepted and a new replica set is created. However the old replica set stays alive as well.

I apply it with the following command in order to replace the variables. Take note of how the version is set to the current date and time.

AGENT_NAME=$AGENT_NAME AGENT_REPLICAS=$AGENT_REPLICAS AGENT_IMAGE=$AGENT_IMAGE AGENT_RESOURCES=$AGENT_RESOURCES VERSION=$(date +"%Y%m%d%H%M") sed -r 's/^(.*)(\$\{[A-Z_]+\})/echo "\1\2"/e' ./manifests/gocd-agent-deployment.yaml | kubectl apply -f - --record

The deployment description shows labels and selectors to have a mismatch.

Name:                   gocd-agent-jdk8
Namespace:              apex
CreationTimestamp:      Thu, 14 Apr 2016 21:07:45 +0200
Labels:                 app=gocd-agent-jdk8,release=v201604142107
Selector:               app=gocd-agent-jdk8,release=v201604151548
Replicas:               1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           Recreate
MinReadySeconds:        0
OldReplicaSets:         <none>
NewReplicaSet:          gocd-agent-jdk8-70801407 (1/1 replicas created)
Events:
  FirstSeen     LastSeen        Count   From                            SubobjectPath   Type            Reason                  Message
  ---------     --------        -----   ----                            -------------   --------        ------                  -------
  12m           12m             1       {deployment-controller }                        Normal          ScalingReplicaSet       Scaled up replica set gocd-agent-jdk8-70801407 to 1

And the rollout history acts like the second deployment never happened.

REVISION        CHANGE-CAUSE
1               kubectl apply -f - --record

Am I understanding the strategy Recreate incorrectly? Or is this a bug?

[zhouhaibing089]

I am guessing it is because you are using different selectors?

[JorritSalverda]

Is there another way to force deployment when the image tag remains the same? I started out without the selector and just the app label, but that didn't do anything at all.

[zhouhaibing089]

If you are keeping everything as it is and just change the strategy to be Recreate, then you may hit this issue: #24198 do you?

[JorritSalverda]

No, I hit that before but worked my way around by deleting deployment, rs and starting a fresh deployment.

This issue however happens since that moment.

[janetkuo]

@JorritSalverda all old RSes will be kept by default unless you specify the deployment's .spec.revisionHistoryLimit.

[janetkuo]

If you find this surprising please complain at #23597.

[JeanMertz]

I think @JorritSalverda specifically means that old RS' are kept running, instead of downscaled once a new version is deployed.

This seems to be "working as intended", as you change the selector, and thus the old pods no longer match to this RS, and can't be downscaled.

From what I understand, a RS is "updated" when its "hash" differs, ie: when anything in the config is changed. So, instead of updating the .spec.selector labels, you could change a .metadata.labels label, as those labels don't influence which pods are attached to the RS.

[JorritSalverda]

Thanks, by removing the version from .spec.selector but keeping it in .metadata.labels it makes everything work as expected. The old replicaset has it's replica number set to 0 after running the deployment, so it's no longer actively keeping pods alive and indeed the old pod is gone. The rollout history is now correct as well.

I already set the .spec.revisionHistoryLimit to 25 so it doesn't slow down or overload the cluster after a while as we're doing dozens deployment a day. I can see why keeping a number of them around is useful for rolling back. In GCE you can do the same thing with instance groups by keeping the old instance templates around. Although the default of infinite might be a bit confusing I don't mind now that I know how it works.

[CelsoSantos]

I too am facing this issue. My old/previous RSs are being kept alive/running.

My issue in particular is this.
I have a deployment YAML file where I do the following:

• I have two labels in template.metadata.labels: JenkinsBuildID:"0" and app:api
• Then, I have spec.selectors with matchLabels and to match solely by the app tag.
• Sample:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      name: api
      labels:
        JenkinsBuildID: "0"
        app: api
    spec:
      containers:
      - name: api
        image: eu.gcr.io/<myproject>/staging/api
        env:
        - name: NODE_ENV
          value: production
        - name: DATABASE_SERVICE_HOST
          value: ip.ip.ip.ip
        ports:
        - containerPort: 80
          protocol: TCP
        imagePullPolicy: Always

I use jenkins to build my project and a docker image and push it to GCR. This docker image only has the 'latest' tag.

After that is done, I instruct Jenkins to change the label JenkinsBuildID of the deployment that is already in my cluster:

export PATH=$PATH:/usr/local/share/google/google-cloud-sdk/bin/
gcloud container clusters get-credentials bb-staging
kubectl get deployment api -o yaml | sed -e "s/JenkinsBuildID: .*/JenkinsBuildID: \"$BUILD_NUMBER\"/g" | kubectl replace -f -

This does update the deployment and create a new replica set. However, the previous one will remain running and be "orphaned" as it doesn't even show on the "Old Replica Sets" section of the Kubernetes UI.

Am I doing anything wrong here??

Best Regards,
Celso Santos

[fcvarela]

Thought this might help anyone looking at how to remove inactive replicasets (spec.replicas set to zero by the deployment). You may want to add something to filter for replicasets actually managed by a deployment as this might end up destroying manually created replicasets:

    kubectl get --all-namespaces rs -o json|jq -r '.items[] | select(.spec.replicas | contains(0)) | "kubectl delete rs --namespace=\(.metadata.namespace) \(.metadata.name)"'

This will not delete anything, it will simply output a list of kubectl commands required to perform the cleanup.

[0xmichalis]

You don't need to run any script in order to cleanup old replica sets - just set .spec.revisionHistoryLimit in the Deployment to the number of old replica sets you want to retain.

http://kubernetes.io/docs/user-guide/deployments/#clean-up-policy

[fcvarela]

Yep, I'm doing that but wrote that script to clean up the ones that had already been created before.

…

On Thu, 8 Dec 2016 at 16:09, Michail Kargakis ***@***.***> wrote: You don't need to run any script in order to cleanup old replica sets - just set .spec.revisionHistoryLimit in the Deployment to the number of old replica sets you want to retain. http://kubernetes.io/docs/user-guide/deployments/#clean-up-policy — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#24330 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAMuep2OFhU-ucfG78BGPR9qGYuxknd_ks5rGCujgaJpZM4IIWvA> .

[webwurst]

I came across this today, just a short heads-up: in @fcvarela command the select expression probably should look like this select(.spec.replicas | contains(0)).

[hridyeshpant]

Hi
i am trying to blue green deployment and not finding any official document . I want to delete all old replica set once new version successfully deployed. So i need to used version in .spec.selector section so that new replicaset and deployment can be run while old one still taking traffic. but the problem is i need to clean all old deployment and replica set once new version deployed successfully.
Does any one have implemented blue green deployment while cleaning the old stuff?

[willzhang]

@JorritSalverda all old RSes will be kept by default unless you specify the deployment's .spec.revisionHistoryLimit.

it helpd me ,thanks you very much!!

[bulletinmybeard]

You don't need to run any script in order to cleanup old replica sets - just set .spec.revisionHistoryLimit in the Deployment to the number of old replica sets you want to retain.

This did the trick for me, thanks!!!

spec:
    revisionHistoryLimit: 0
    ....

[Oluwatobiadeoye]

@JorritSalverda

I think i have similar problem to yours. After a new deployment, the pods attached to the old replica are being deleted but the replica itself is not, I end up with a replica showing 0/0. Did you also have this issue?