What happened?

Currently I am working on a project where we implement a Kubernetes operator and we decided that for some flows we will need to wait for some deployments to complete the rollout before advancing to some other steps from our processes.
So, for an old deployment we only update an image in the pod template and then we wait for the updated deployment to finish the rollout.

The way the status of the Deployment is built/implemented is error prone and not reliable if we only take into account the status.conditions (as suggested in the documentation)

What did you expect to happen?

After updating the spec of a deployment, when the status.ObservedGeneration is changed to the latest metadata.Generation, the status.conditions should also be updated.

How can we reproduce it (as minimally and precisely as possible)?

We found out that if:

• we start with a Deployment having a status.condition of type Progressing with the reason ProgressDeadlineExceeded
• update the deployment spec (the only change would be an image from a container from the pod template)
• on a timer (every second), start retrieving the deployment using the API and check the status
  Then, under some circumstances (so, not always; it's a concurrency/timing issue), the first change of the status.observedGeneration toward the new metadata.generation value, comes with no change(!) in the status.conditions (the old conditions are still there without any single change(!)).
  According to the Kubernetes documentation (and also the way the Operator SDK checks this rollout status), the interpretation of these would lead to say that the rollout failed, which is not accurate, because the actual rollout will only start a few seconds later...

This is an example from a log describing the issue (please observe that the two blocks are logged at a few milliseconds distance and the status.conditions are the same even the status.ObservedGeneration changed):

# 10:53:18.814 > Metadata.Generation: 2 | Status >> ObservedGeneration: 1, replicas: 1 , readyReplicas: , availableReplicas: , updatedReplicas: 1, unAvailableReplicas: 1
  Conditions:
    @10-Apr-24 10:53:02 | 10-Apr-24 10:53:02 | Available | False | MinimumReplicasUnavailable | Deployment does not have minimum availability.
    @10-Apr-24 10:53:18 | 10-Apr-24 10:53:18 | Progressing | False | ProgressDeadlineExceeded | ReplicaSet "my-deployment-5b8dc498c" has timed out progressing.
 => completed: False, failed: False

# 10:53:18.818 > Metadata.Generation: 2 | Status >> ObservedGeneration: 2, replicas: 1 , readyReplicas: , availableReplicas: , updatedReplicas: , unAvailableReplicas: 1
  Conditions:
    @10-Apr-24 10:53:02 | 10-Apr-24 10:53:02 | Available | False | MinimumReplicasUnavailable | Deployment does not have minimum availability.
    @10-Apr-24 10:53:18 | 10-Apr-24 10:53:18 | Progressing | False | ProgressDeadlineExceeded | ReplicaSet "my-deployment-5b8dc498c" has timed out progressing.
 => completed: False, failed: True

Anything else we need to know?

Please advise on how to reliably interpret the rollout status of a Deployment.

Kubernetes version

$ kubectl version
Client Version: v1.28.8+k3s1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.8+k3s1

Cloud provider

OS version

# On Linux:
$ cat /etc/os-release
PRETTY_NAME="K3s v1.28.8+k3s1"


$ uname -a
 Linux 3432b46b9978 5.15.0-101-generic #111-Ubuntu SMP Tue Mar 5 20:16:58 UTC 2024 x86_64 GNU/Linux



</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>