Deployment.ready may not accurately reflect deployment status

[jkemp101]

The check self.obj["status"]["updatedReplicas"] == self.replicas does not necessarily indicate the deployment is successful. It probably works ok if you are doing a create() but not if you are doing an update(). For instance if you have 2 replicas and update the deployment with a surge of 2 you will see an updatedReplicas of 2 but you will also see unavailableReplicas at 2. Following is an example of updating a deployment in this scenario. The True at the end of the line is the ready value.

2016-10-04 13:33:07,436 - root - INFO - {u'observedGeneration': 15, u'updatedReplicas': 2, u'unavailableReplicas': 2, u'availableReplicas': 1, u'replicas': 3} 5 True
2016-10-04 13:33:09,535 - root - INFO - {u'observedGeneration': 15, u'updatedReplicas': 2, u'unavailableReplicas': 1, u'availableReplicas': 1, u'replicas': 2} 5 True
2016-10-04 13:33:11,632 - root - INFO - {u'observedGeneration': 15, u'updatedReplicas': 2, u'unavailableReplicas': 1, u'availableReplicas': 1, u'replicas': 2} 5 True
2016-10-04 13:33:13,730 - root - INFO - {u'observedGeneration': 15, u'updatedReplicas': 2, u'unavailableReplicas': 1, u'availableReplicas': 1, u'replicas': 2} 5 True
2016-10-04 13:33:15,824 - root - INFO - {u'observedGeneration': 15, u'updatedReplicas': 2, u'unavailableReplicas': 1, u'availableReplicas': 1, u'replicas': 2} 5 True
2016-10-04 13:33:17,922 - root - INFO - {u'observedGeneration': 15, u'updatedReplicas': 2, u'unavailableReplicas': 1, u'availableReplicas': 1, u'replicas': 2} 5 True
2016-10-04 13:33:20,017 - root - INFO - {u'observedGeneration': 15, u'updatedReplicas': 2, u'unavailableReplicas': 1, u'availableReplicas': 1, u'replicas': 2} 5 True
2016-10-04 13:33:22,115 - root - INFO - {u'observedGeneration': 15, u'updatedReplicas': 2, u'unavailableReplicas': 1, u'availableReplicas': 1, u'replicas': 2} 5 True
2016-10-04 13:33:24,212 - root - INFO - {u'observedGeneration': 15, u'updatedReplicas': 2, u'availableReplicas': 2, u'replicas': 2} 5 True

I'm doing this right now but it is ugly. Would love for a cleaner check.

    start_time = datetime.datetime.now()
    while (datetime.datetime.now() - start_time).total_seconds() < timeout:
        try:
            deployment.reload()
            current_revision = deployment.annotations['deployment.kubernetes.io/revision']
            logging.info("%s %s", deployment.obj['status'], current_revision)

            if int(current_revision) < int(our_revision):
                logging.info("Waiting for our deployment to start")
                time.sleep(2)
                continue

            if 'availableReplicas' in deployment.obj['status'] and 'updatedReplicas' in deployment.obj['status'] and \
                deployment.obj['status']['updatedReplicas'] == deployment.obj['status']['availableReplicas'] and \
                deployment.obj['status']['replicas'] == deployment.obj['status']['availableReplicas']:

                #Final check, just in case
                if 'unavailableReplicas' in deployment.obj['status'] and \
                    deployment.obj['status']['unavailableReplicas'] != 0:
                    logging.info("Unavailable != 0")
                    time.sleep(2)
                    continue

                logging.info("Done waiting for deployment")
                if current_revision != our_revision:
                    logging.info("Looks like our deployment got bumped ours: %s current: %s", \
                                 our_revision, current_revision)
                return current_revision

        except Exception:  #pylint: disable=w0703
            logging.error("Error:", exc_info=True)