Support user-specified readiness/await logic

[lblackstone]

Problem description

We already provide readiness/await logic for common k8s resources today, but this doesn't adequately cover CustomResources since each one is different. For k8s operators in particular, it's common to specify a state and then wait for a particular condition to be met before further action is taken.

In many cases, checking for readiness requires checking a field in a k8s resource against a known value (e.g. "Ready"). It's possible to watch the k8s Event stream for updates, so we should provide some mechanism for users to specify Events + fields of interest and use that as a dependency in the Pulumi resource graph.

Here's an example showing one possible way of implementing this support: https://gist.github.com/lblackstone/d2d1c29507685c0833612988945dedf3

Related issues:
#912
#1056

[marioapardo]

Can the example be done in the Python SDK? or is it just for JS right now?

[ghostsquad]

any news on this front? I feel like this is really important functionality that was made easy in ArgoCD, as they allowed you to write some simple Lua to perform health checking.

https://argo-cd.readthedocs.io/en/stable/operator-manual/health/#custom-health-checks

I feel like if I could figure out, even a temporary work around, to add health checks for CRDs, then I could use the Pulumi K8s Operator to deploy Pulumi k8s stack resources to k8s clusters, doing so in order based on health, and essentially implement the ArgoCD App of Apps pattern within Pulumi, and further simplify my workflows. I would need pulumi to check the health of the stack it deployed (to a different namespace/cluster).

[lblackstone]

any news on this front?

We're still thinking about how to support this generically in the provider.

Relatedly, we recently started factoring out the await logic into a separate library, which would make it possible to contribute checks for additional resource types. This will also require work on the provider side to make use of any new checks, but I expect that we will eventually support this more dynamically.

[kralikba]

Has there been any update to this?
I understand that this is a major design challenge. In my use case, specifying an expected value for a field of the CRD would also be a simple solution. Right now I have to resort to too many hacks using dummy Outputs and a non-pulumi kubernetes client. (I'm using C#)

[ghostsquad]

@kralikba what if in the interim, each resource was wrapped in a component resource, which could be the object and a [https://www.pulumi.com/registry/packages/command/api-docs/provider/](command resource), that called out to a CLI like https://github.com/cakehappens/seaworthy ?

Seaworthy is your post-apply validation that your K8s resources deployed correctly and are healthy.

This CLI is still just in a POC state.

---

I think the main problem with Pulumi allowing custom health checking, is that in some languages, like Go, you can't serialize functions, and the code that we all write, is just creating a schema of desired state, which is serializable, as just a series of interconnected objects.

I too am interested to know what kinds of problems are blocking this issue.

[jabbrwcky]

Hey, I stumbled across this issue while looking for a way to wait on cert-manager having issued a certificate before proceeding.

I see the issue with serializing logic, but would it not at least be possible to express resource readiness constraints for resources that implement the recommended conditions for the status subresource.

The cert-manager Certificate CR does implement status this way, e.g.:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  annotations:
    pulumi.com/patchForce: 'true'
    pulumi.com/timeoutSeconds: '60'
  name: my-cert
  namespace: zammad-stage-system
spec:
  # ...
status:
  conditions:
    - lastTransitionTime: '2023-01-04T13:37:56Z'
      message: Certificate is up to date and has not expired
      observedGeneration: 1
      reason: Ready
      status: 'True'
      type: Ready
  notAfter: '2023-04-04T13:37:56Z'
  notBefore: '2023-01-04T13:37:56Z'
  renewalTime: '2023-03-20T13:37:56Z'
  revision: 1

It might just be passed as part of resource options as plain data:

const tls = new k8s.apiextensions.CustomResource(`${name}-certificate`, {
    apiVersion: "cert-manager.io/v1",
    kind: "Certificate",
    spec: { /* ... */ }
},{
    readyConditions: [{
        status: 'True',
        type: 'Ready',
        // reason: '...',
    }]
})

This way no function serialization would be required for static language support?

[lblackstone]

No concrete updates to share at this time, but it might be worth thinking through a solution specific to CRs, since this scenario comes up frequently. It seems like the most common case there is checking the value of a known field, which seems doable without needing function serialization.

On a related note, we've recently done some exploratory work to fix pulumi/pulumi#6948, which has some similarities to this problem.

[jabbrwcky]

For comparison, the Terraform Kubernetes provider supports a wait{} block (at least for manifests), which apparently uses sth. like JsonPath to reference fields of a resource.

[lkt82]

Hi @lblackstone any updates on this :)?.

[matanbaruch]

+1

[lblackstone]

I'm not working directly in this area of Pulumi anymore, but I'd suggest that anyone interested add a 👍 to the issue. This is one of the signals that the team uses to prioritize work, so make sure to do that for any issues that are important to you!

[EronWright]

Short of having fully extensible await logic, would a useful feature be to support waiting for certain status conditions, e.g. Ready. I believe that kubectl wait and kubectl rollout status has functionality similar to this. And TF has wait conditions.