Regression: Resources can get into a bad state if resource client init fails

[lblackstone]

Following #348, I ran into a bug where the provider failed to get clients with the following errors:

kubernetes:extensions:Deployment (nginx-ingress-default-backend):
    error: Plan apply failed: Could not make client to watch Deployment "nginx-ingress-default-backend": unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: not found

  pulumi:pulumi:Stack (gke-ingress-gke-ingress-dev):
    E0122 16:13:15.160706   20968 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request

After this failure, I tried running pulumi up again, but got an error because the resource had actually created successfully, but not registered that status with the engine:

  kubernetes:extensions:Deployment (nginx-ingress-default-backend):
    error: Plan apply failed: deployments.extensions "nginx-ingress-default-backend" already exists

Also, a pulumi refresh did not correct the state mismatch.

[lblackstone]

This bug is still present, but is much less likely to be triggered now that the client logic has been cleaned up further in #367 and #414.

Removing the P1 label, and expect to fix this as part of the ongoing await logic refactoring.

[hausdorff]

I am confused about how this could happen. Do we have a repro or a timeline/set of logs to indicate how this could have happened? Until them I'm super reticent to believe that it's actually less likely because of the client logic cleanup.

Marking as Q3 -- can always remove later if we decide it's not super high priority.

[lblackstone]

@hausdorff IIRC, the problem was that the resource create request was successful, but the watch client used by the await logic failed. We were not returning a partial failure in this case, so Pulumi didn't have a record of that resource being created. This can cause two error cases:

1. auto-named resource - orphans a copy of the resource
2. manually-named resource or cluster-scoped resource - subsequent operations fail because the resource already exists, but is not tracked by the engine

[lukehoban]

@lblackstone Do you have a repro for this?

[lblackstone]

I don't have a repro, and the original cause was buggy resource client logic. I think we should still audit the error path here, but don't think it's likely to occur in practice at this point.

[hausdorff]

I'm still skeptical that this is "really a bug" but I've put up #664 which fixes a (theoretical?) object leak that could have caused something like this.