Incorrect status of kubernetes deployment after updating deployment few times

[pagarwal-tibco]

What happened?

When we try to update deployment resource many times where it fails intermittently, status would show incorrect information. Please check steps below for more information.

What did you expect to happen?

In case of failure, deployment status should show correct information. Please check steps below for more information.

How can we reproduce it (as minimally and precisely as possible)?

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: echo
  name: echo
  namespace: default
spec:
  progressDeadlineSeconds: 180
  replicas: 1
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  selector:
    matchLabels:
      app.kubernetes.io/name: echo
  template:
    metadata:
      labels:
        app.kubernetes.io/name: echo
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      containers:
        - image: mendhak/http-https-echo
          imagePullPolicy: IfNotPresent
          name: echo
          ports:
            - containerPort: 80
              protocol: TCP
          resources: {}
      terminationGracePeriodSeconds: 10

Steps:

1. Deploy above kubernetes deployment resource
2. Change the image name to 'mendhak/http-https-echo1' and update the deployment. This should result in status of deployment as 'ProgressDeadlineExceeded'
3. Now again correct the image name as 'mendhak/http-https-echo' and update the deployment. Now deployment status will indicate success
4. Now again change the image name as 'mendhak/http-https-echo1' and update the deployment. Now deployment status will still incorrectly indicate success.
5. If issue could not be reproduced then repeat steps 2-4

At step #4, deployment status will be incorrect as follows,
message: ReplicaSet "echo-547fb74949" has successfully progressed. reason: NewReplicaSetAvailable status: "True" type: Progressing

Here reason should have been 'ProgressDeadlineExceeded' and message should contain correct replica-set name. "echo-547fb74949" replica-set has old revision than the one indicated by 'deployment.kubernetes.io/revision' deployment annotation.

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.0", GitCommit:"af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38", GitTreeState:"clean", BuildDate:"2020-12-08T17:59:43Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.7", GitCommit:"132a687512d7fb058d0f5890f07d4121b3f0a2e2", GitTreeState:"clean", BuildDate:"2021-05-27T23:27:49Z", GoVersion:"go1.15.12", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider

EKS and KinD

OS version

No response

Install tools

No response

Container runtime (CRI) and and version (if applicable)

No response

Related plugins (CNI, CSI, ...) and versions (if applicable)

No response

[k8s-ci-robot]

@pagarwal-tibco: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[neolit123]

/sig apps

[atiratree]

I am sorry, but I could not reproduce this one (albeit with different image and lower progressDeadlineSeconds timeout). Can you please give a more verbose output what has happened? Full deployment and all ReplicaSets YAMLs? Also events would be helpful.

It would be helpful if you could write a script to reproduce this and tell us how often it happens in your environment.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.