pulumi fails for some deployment with "Minimum number of live Pods was not attained"

[ncsibra]

Deployment with "apiVersion: extensions/v1beta1" sometimes fails to initialize during an update, when creating the first time it works fine, but fails on the first update.
A similar issue was fixed once: #497
I think currently it's caused by logic in kubernetes lib, which replace the "extensions/v1beta1" kinds to "apps/v1" because I did not experience the same issue with deployments which have, "apps/v1" kind by default.
Versions:

➜  pulumi (115-latency) ✗ pulumi version
v1.0.0
➜  pulumi (115-latency) ✗ yarn list --pattern "@pulumi"                                    
yarn list v1.17.3
warning package.json: No license field
warning prmrgt-infra: No license field
├─ @pulumi/aws@1.1.0
├─ @pulumi/cloudflare@0.17.14
├─ @pulumi/kubernetes@1.0.0
├─ @pulumi/pulumi@1.0.0
├─ @pulumi/query@0.3.0
└─ @pulumi/random@1.0.0
Done in 0.15s.
➜  pulumi (115-latency) ✗ pulumi plugin ls
NAME        KIND      VERSION       SIZE    INSTALLED  LAST USED
aws         resource  1.1.0         218 MB  n/a        1 day ago
aws         resource  1.0.0-beta.3  212 MB  n/a        1 week ago
aws         resource  0.18.26       209 MB  n/a        1 month ago
aws         resource  0.18.25       209 MB  n/a        1 month ago
aws         resource  0.18.24       209 MB  n/a        1 month ago
aws         resource  0.18.10       204 MB  n/a        1 month ago
aws         resource  0.17.1        157 MB  n/a        1 month ago
cloudflare  resource  0.17.14       41 MB   n/a        1 day ago
cloudflare  resource  0.17.11       41 MB   n/a        1 week ago
cloudflare  resource  0.17.8        41 MB   n/a        1 month ago
cloudflare  resource  0.17.3        40 MB   n/a        1 month ago
kubernetes  resource  1.0.0         53 MB   n/a        1 day ago
kubernetes  resource  1.0.0-beta.2  53 MB   n/a        1 week ago
kubernetes  resource  0.25.6        51 MB   n/a        1 month ago
kubernetes  resource  0.25.5        51 MB   n/a        1 month ago
kubernetes  resource  0.25.4        51 MB   n/a        1 month ago
kubernetes  resource  0.25.3        51 MB   n/a        1 month ago
kubernetes  resource  0.25.2        51 MB   n/a        1 month ago
kubernetes  resource  0.24.0        53 MB   n/a        1 month ago
random      resource  1.0.0         39 MB   n/a        1 day ago
random      resource  1.0.0-beta.2  39 MB   n/a        1 week ago
random      resource  0.5.6         36 MB   n/a        1 month ago
random      resource  0.5.3         36 MB   n/a        1 month ago
random      resource  0.5.1         28 MB   n/a        1 month ago

Example state:

            {
                "urn": "urn:pulumi:sbx::k8s-prometheus-deployment::kubernetes:helm.sh/v2:Chart$kubernetes:extensions/v1beta1:Deployment::monitoring/sbx-k8s-kubelet-stats-exporter",
                "custom": true,
                "id": "monitoring/sbx-k8s-kubelet-stats-exporter",
                "type": "kubernetes:extensions/v1beta1:Deployment",
                "inputs": {
                    "apiVersion": "extensions/v1beta1",
                    "kind": "Deployment",
                    "metadata": {
                        "labels": {
                            "app": "sbx-k8s-kubelet-stats-exporter",
                            "app.kubernetes.io/managed-by": "pulumi"
                        },
                        "name": "sbx-k8s-kubelet-stats-exporter",
                        "namespace": "monitoring"
                    },
                    "spec": {
                        "template": {
                            "metadata": {
                                "labels": {
                                    "app": "sbx-k8s-kubelet-stats-exporter"
                                }
                            },
                            "spec": {
                                "containers": [
                                    {
                                        "command": [
                                            "/bin/bash",
                                            "-ec",
                                            "while :; do echo '.'; sleep 5 ; done"
                                        ],
                                        "image": "tutum/curl",
                                        "name": "sbx-k8s-kubelet-stats-exporter"
                                    }
                                ],
                                "nodeSelector": {
                                    "dliver.com/stack": "general"
                                },
                                "serviceAccountName": "sbx-k8s-kubelet-stats-exporter"
                            }
                        }
                    }
                },
                "outputs": {
                    "__inputs": {
                        "apiVersion": "extensions/v1beta1",
                        "kind": "Deployment",
                        "metadata": {
                            "labels": {
                                "app": "sbx-k8s-kubelet-stats-exporter",
                                "app.kubernetes.io/managed-by": "pulumi"
                            },
                            "name": "sbx-k8s-kubelet-stats-exporter",
                            "namespace": "monitoring"
                        },
                        "spec": {
                            "template": {
                                "metadata": {
                                    "labels": {
                                        "app": "sbx-k8s-kubelet-stats-exporter"
                                    }
                                },
                                "spec": {
                                    "containers": [
                                        {
                                            "command": [
                                                "/bin/bash",
                                                "-ec",
                                                "while :; do echo '.'; sleep 5 ; done"
                                            ],
                                            "image": "tutum/curl",
                                            "name": "sbx-k8s-kubelet-stats-exporter"
                                        }
                                    ],
                                    "nodeSelector": {
                                        "dliver.com/stack": "general"
                                    },
                                    "serviceAccountName": "sbx-k8s-kubelet-stats-exporter"
                                }
                            }
                        }
                    },
                    "apiVersion": "apps/v1",
                    "kind": "Deployment",
                    "metadata": {
                        "annotations": {
                            "deployment.kubernetes.io/revision": "5",
                            "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"extensions/v1beta1\",\"kind\":\"Deployment\",\"metadata\":{\"labels\":{\"app\":\"sbx-k8s-kubelet-stats-exporter\",\"app.kubernetes.io/managed-by\":\"pulumi\"},\"name\":\"sbx-k8s-kubelet-stats-exporter\",\"namespace\":\"monitoring\"},\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"app\":\"sbx-k8s-kubelet-stats-exporter\"}},\"spec\":{\"containers\":[{\"command\":[\"/bin/bash\",\"-ec\",\"while :; do echo '.'; sleep 5 ; done\"],\"image\":\"tutum/curl\",\"name\":\"sbx-k8s-kubelet-stats-exporter\"}],\"nodeSelector\":{\"dliver.com/stack\":\"general\"},\"serviceAccountName\":\"sbx-k8s-kubelet-stats-exporter\"}}}}\n"
                        },
                        "creationTimestamp": "2019-08-15T07:22:03Z",
                        "generation": 6,
                        "labels": {
                            "app": "sbx-k8s-kubelet-stats-exporter",
                            "app.kubernetes.io/managed-by": "pulumi"
                        },
                        "name": "sbx-k8s-kubelet-stats-exporter",
                        "namespace": "monitoring",
                        "resourceVersion": "26583209",
                        "selfLink": "/apis/apps/v1/namespaces/monitoring/deployments/sbx-k8s-kubelet-stats-exporter",
                        "uid": "6157b935-bf2d-11e9-bd14-0a1041233094"
                    },
                    "spec": {
                        "progressDeadlineSeconds": 2147483647,
                        "replicas": 1,
                        "revisionHistoryLimit": 2147483647,
                        "selector": {
                            "matchLabels": {
                                "app": "sbx-k8s-kubelet-stats-exporter"
                            }
                        },
                        "strategy": {
                            "rollingUpdate": {
                                "maxSurge": 1,
                                "maxUnavailable": 1
                            },
                            "type": "RollingUpdate"
                        },
                        "template": {
                            "metadata": {
                                "annotations": {
                                    "kubectl.kubernetes.io/restartedAt": "2019-09-03T08:58:34+02:00"
                                },
                                "labels": {
                                    "app": "sbx-k8s-kubelet-stats-exporter"
                                }
                            },
                            "spec": {
                                "containers": [
                                    {
                                        "command": [
                                            "/bin/bash",
                                            "-ec",
                                            "while :; do echo '.'; sleep 5 ; done"
                                        ],
                                        "image": "tutum/curl",
                                        "imagePullPolicy": "Always",
                                        "name": "sbx-k8s-kubelet-stats-exporter",
                                        "resources": {},
                                        "terminationMessagePath": "/dev/termination-log",
                                        "terminationMessagePolicy": "File"
                                    }
                                ],
                                "dnsPolicy": "ClusterFirst",
                                "nodeSelector": {
                                    "dliver.com/stack": "general"
                                },
                                "restartPolicy": "Always",
                                "schedulerName": "default-scheduler",
                                "securityContext": {},
                                "serviceAccount": "sbx-k8s-kubelet-stats-exporter",
                                "serviceAccountName": "sbx-k8s-kubelet-stats-exporter",
                                "terminationGracePeriodSeconds": 30
                            }
                        }
                    },
                    "status": {
                        "availableReplicas": 1,
                        "conditions": [
                            {
                                "lastTransitionTime": "2019-08-15T07:22:03Z",
                                "lastUpdateTime": "2019-08-15T07:22:03Z",
                                "message": "Deployment has minimum availability.",
                                "reason": "MinimumReplicasAvailable",
                                "status": "True",
                                "type": "Available"
                            }
                        ],
                        "observedGeneration": 6,
                        "readyReplicas": 1,
                        "replicas": 1,
                        "updatedReplicas": 1
                    }
                },
                "parent": "urn:pulumi:sbx::k8s-prometheus-deployment::kubernetes:helm.sh/v2:Chart::sbx-k8s-kubelet-stats-exporter",
                "initErrors": [
                    "Resource 'sbx-k8s-kubelet-stats-exporter' was created but failed to initialize",
                    "Minimum number of live Pods was not attained"
                ],
                "provider": "urn:pulumi:sbx::k8s-prometheus-deployment::pulumi:providers:kubernetes::sbx-k8s-provider::c95b8d5c-8751-499a-823d-a21078947ab4",
                "propertyDependencies": {
                    "apiVersion": null,
                    "kind": null,
                    "metadata": null,
                    "spec": null
                }
            }

kubectl describe deploy sbx-k8s-kubelet-stats-exporter -n monitoring output:

Name:                   sbx-k8s-kubelet-stats-exporter
Namespace:              monitoring
CreationTimestamp:      Thu, 15 Aug 2019 09:22:03 +0200
Labels:                 app=sbx-k8s-kubelet-stats-exporter
                        app.kubernetes.io/managed-by=pulumi
Annotations:            deployment.kubernetes.io/revision: 5
                        kubectl.kubernetes.io/last-applied-configuration:
                          {"apiVersion":"extensions/v1beta1","kind":"Deployment","metadata":{"labels":{"app":"sbx-k8s-kubelet-stats-exporter","app.kubernetes.io/man...
Selector:               app=sbx-k8s-kubelet-stats-exporter
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  1 max unavailable, 1 max surge
Pod Template:
  Labels:           app=sbx-k8s-kubelet-stats-exporter
  Annotations:      kubectl.kubernetes.io/restartedAt: 2019-09-03T08:58:34+02:00
  Service Account:  sbx-k8s-kubelet-stats-exporter
  Containers:
   sbx-k8s-kubelet-stats-exporter:
    Image:      tutum/curl
    Port:       <none>
    Host Port:  <none>
    Command:
      /bin/bash
      -ec
      while :; do echo '.'; sleep 5 ; done
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   sbx-k8s-kubelet-stats-exporter-7c896bd8cc (1/1 replicas created)
Events:          <none>

[lblackstone]

I believe this is the same bug @RichardWLaub was hitting yesterday. The actual cause appears to be .metadata.annotations["deployment.kubernetes.io/revision"] not matching the value for .status.observedGeneration, which is checked here.

It seems that this condition is not true in some cases, so I'll need to revise it.

Note that this only applies to resources that are created with the extensions/v1beta1 apiVersion. The old apiVersion uses a different code path because it did not include some of the fields that we use to compute readiness on newer apiVersions.