CustomResource lifecycle causes issues with importing created resources

[donovanmuller]

Problem description

• I saw an issue in the following Pulumi Program: https://gist.github.com/donovanmuller/362e3969fa84d686518174efede5d0ba

It is sometimes required to get/import/use resources created as the result of creating a CustomResource that is acted on by a controller/operator. Currently I'm facing challenges getting this to work reliably from scratch with the example above.

In the example I would like to access the Secret(containing user credentials) created as part of creating a new Elasticsearch cluster (via the ECK operator), however I am facing the following issues:

• I cannot preview the changes, due to the Secret not yet being present (it will be created after the CustomResource is applied) and therefore resorting to using --skip-preview
• The Elasticsearch custom resource status is not taken into account and therefore does not wait for the that resource to become healthy before attempting to import the Secret, which results in the import failing

I have to resort to running the program twice. First time creates the CustomResource (which then is handled by the controller/operator and creates the other resources, the Secret I need being one of them) and fails on the import. Second time succeeds on the import as it now exists.

Errors & Logs

On the first run:

$ pulumi up --skip-preview
Updating (test):
     Type                                                                             Name                                                             Status                   Info
 +   pulumi:pulumi:Stack                                                              customresource-issue-test                                        **creating failed**      1 error; 3 messag
 +   ├─ kubernetes:yaml:ConfigGroup                                                   eck-operator                                                     created
 +   │  └─ kubernetes:yaml:ConfigFile                                                 https://download.elastic.co/downloads/eck/1.0.1/all-in-one.yaml  created
 +   │     ├─ kubernetes:core:Secret                                                  elastic-system/elastic-webhook-server-cert                       created
 +   │     ├─ kubernetes:core:Namespace                                               elastic-system                                                   created
 +   │     ├─ kubernetes:core:ServiceAccount                                          elastic-system/elastic-operator                                  created
 +   │     ├─ kubernetes:admissionregistration.k8s.io:ValidatingWebhookConfiguration  elastic-webhook.k8s.elastic.co                                   created
 +   │     ├─ kubernetes:core:Service                                                 elastic-system/elastic-webhook-server                            created
 +   │     ├─ kubernetes:rbac.authorization.k8s.io:ClusterRoleBinding                 elastic-operator                                                 created
 +   │     ├─ kubernetes:rbac.authorization.k8s.io:ClusterRole                        elastic-operator                                                 created
 +   │     ├─ kubernetes:apps:StatefulSet                                             elastic-system/elastic-operator                                  created
 +   │     ├─ kubernetes:apiextensions.k8s.io:CustomResourceDefinition                kibanas.kibana.k8s.elastic.co                                    created
 +   │     ├─ kubernetes:apiextensions.k8s.io:CustomResourceDefinition                apmservers.apm.k8s.elastic.co                                    created
 +   │     └─ kubernetes:apiextensions.k8s.io:CustomResourceDefinition                elasticsearches.elasticsearch.k8s.elastic.co                     created
 +   ├─ kubernetes:elasticsearch.k8s.elastic.co:Elasticsearch                         test-elastic                                                     created
 =   └─ kubernetes:core:Secret                                                        test-es-elastic-user                                             **importing failed**     1 error

Diagnostics:
  kubernetes:core:Secret (test-es-elastic-user):
    error: resource 'default/test-es-elastic-user' does not exist

  pulumi:pulumi:Stack (customresource-issue-test):
    Found incompatible versions of @pulumi/pulumi. Differing major or minor versions are not supported.
      Version 1.13.1 referenced at node_modules/@pulumi/pulumi/package.json
      Version 1.13.1 referenced at node_modules/@pulumi/kubernetesx/node_modules/@pulumi/kubernetes/node_modules/@pulumi/pulumi/package.json

    error: update failed

Resources:
    + 15 created

Duration: 36s

error: resource 'default/test-es-elastic-user' does not exist

The second run will import successfully because the Secret has been created in the meantime:

$ pulumi up --skip-preview
Updating (test):
     Type                       Name                       Status       Info
     pulumi:pulumi:Stack        customresource-issue-test               3 messages
 =   └─ kubernetes:core:Secret  test-es-elastic-user       imported

Diagnostics:
  pulumi:pulumi:Stack (customresource-issue-test):
    Found incompatible versions of @pulumi/pulumi. Differing major or minor versions are not supported.
      Version 1.13.1 referenced at node_modules/@pulumi/pulumi/package.json
      Version 1.13.1 referenced at node_modules/@pulumi/kubernetesx/node_modules/@pulumi/kubernetes/node_modules/@pulumi/pulumi/package.json

Outputs:
  + elasticPassword: "[secret]"

Resources:
    = 1 imported
    15 unchanged

Duration: 9s

Suggestions for a fix

Perhaps there is already a solution that would cater for my use case and a solution/workaround can be suggested but in the lack thereof, it would be great:

• if there was a way to "wait" for the CustomResource to become healthy so that any dependents could continue based on the fact that the custom resource was fully created. Given that custom resources have differing ways of determining health/status, perhaps a function could be provided to determine what that would look like by checking various status properties
• if the import could retry for a specified time to wait for the resource to be created and not fail immediately
• to not have to use --skip-preview

[donovanmuller]

This appears somewhat similar to #912

Specifically where it mentions:

...
For example, a means in which I could extend the pulumi-kubernetes SDK to create a pulumi.CustomResource (this gets confusing) that:

1. Applies a SealedSecret definition with kubernetes.yaml.ConfigFile
2. Watches that kubernetes object for a success or failure
3. And returns a the type for a standard secret object.

[jnatherley]

@donovanmuller This is exactly the same issue i'm currently facing.

After reading the documentation thoroughly, this seems to be what the .get static function provides. However, i'm not quite sure how to make it wait until the resource exists or wait for the CustomResource to be ready.

I'm sure there is a way.

[donovanmuller]

My current thoughts (still need to experiment) are to wrap the CustomResource in a Dynamic Provider and basically shell out to kubectl and wait until the status of the resource reaches the desired state in the apply() callback.

[donovanmuller]

To add to this:

• if there was a way to "wait" for the CustomResource to become healthy so that any dependents could continue based on the fact that the custom resource was fully created. Given that custom resources have differing ways of determining health/status, perhaps a function could be provided to determine what that would look like by checking various status properties

perhaps the way kpt handles this when waiting for reconcile could be borrowed?
See the Recommendations for CRDs section for basic assumptions.

[lblackstone]

Thanks for the detailed writeup and resource links. pulumi/pulumi#1691 is related, and pulumi/pulumi#1691 (comment) might be a workaround for you until we're done with that work.

[timmyers]

I think I found a pretty simple workaround for this sort of issue: take a look at the following gist if interested.

https://gist.github.com/timmyers/4d2fed53a358d4c98557a5886ae2afbb