daemonset that relies on kube-dns fails because managednodegroup failed to create

[tusharshahrs]

What happened?

When creating a new daemonset called node-local-dns, which relies on the kube-dns created by the core-dns. I have noticed that if I create the daemonset on an existing cluster, everything works smoothly. However, if I attempt to create a new cluster, the provision fails with the following error message:

kubernetes:core/v1:Service (kube-dns):
     Apr 26 06:54:01.102 [INFO] [update]     error: 2 errors occurred:
     Apr 26 06:54:01.102 [INFO] [update]     	* Resource 'kube-dns' was created but failed to initialize
     Apr 26 06:54:01.102 [INFO] [update]     	* Service does not target any Pods. Selected Pods may not be ready, or field '.spec.selector' may not match labels on any Pods

Further investigation revealed that the ManagedNodeGroup failed to create, resulting in no pods running. In our code, the ManagedNodeGroup is dependent on the control_plane object, which creates the core-dns/kube-dns

The following was added:
pulumi.com/skipAwait](http://pulumi.com/skipAwait): "true" as annotation in the service

Without the annotation, error message was like below. As you can see pulumi complained about 3 services node-local-dns, kube-dns-upstream and kube-dns. node-local-dns, kube-dns-upstream are part of node-local-dns,whilekube-dns is part of core-dns

Error:

    Apr 25 20:26:46.380 [INFO] [update]   kubernetes:core/v1:Service (node-local-dns):
     Apr 25 20:26:46.380 [INFO] [update]     error: 2 errors occurred:
     Apr 25 20:26:46.380 [INFO] [update]     	* resource kube-system/node-local-dns was successfully created, but the Kubernetes API server reported that it failed to fully initialize or become live: 'node-local-dns' timed out waiting to be Ready
     Apr 25 20:26:46.380 [INFO] [update]     	* Service does not target any Pods. Selected Pods may not be ready, or field '.spec.selector' may not match labels on any Pods
     Apr 25 20:26:46.380 [INFO] [update]  
     Apr 25 20:26:46.380 [INFO] [update]   kubernetes:core/v1:Service (kube-dns):
     Apr 25 20:26:46.380 [INFO] [update]     error: 2 errors occurred:
     Apr 25 20:26:46.380 [INFO] [update]     	* Resource 'kube-dns' was created but failed to initialize
     Apr 25 20:26:46.380 [INFO] [update]     	* Service does not target any Pods. Selected Pods may not be ready, or field '.spec.selector' may not match labels on any Pods
     Apr 25 20:26:46.380 [INFO] [update]  
     Apr 25 20:26:46.380 [INFO] [update]   kubernetes:core/v1:Service (kube-dns-upstream):
     Apr 25 20:26:46.380 [INFO] [update]     error: 2 errors occurred:
     Apr 25 20:26:46.380 [INFO] [update]     	* resource kube-system/kube-dns-upstream was successfully created, but the Kubernetes API server reported that it failed to fully initialize or become live: 'kube-dns-upstream' timed out waiting to be Ready
     Apr 25 20:26:46.380 [INFO] [update]     	* Service does not target any Pods. Selected Pods may not be ready, or field '.spec.selector' may not match labels on any Pods

after added the annotation to kube-dns-upstream and node-local-dns, the error message only complains about kube-dns, which also has the annotation configured.

Error:

Apr 26 21:05:02.365 [INFO] [update] Diagnostics:
     Apr 26 21:05:02.365 [INFO] [update]   kubernetes:core/v1:Service (kube-dns):
     Apr 26 21:05:02.365 [INFO] [update]     error: 2 errors occurred:
     Apr 26 21:05:02.365 [INFO] [update]     	* Resource 'kube-dns' was created but failed to initialize
     Apr 26 21:05:02.365 [INFO] [update]     	* Service does not target any Pods. Selected Pods may not be ready, or field '.spec.selector' may not match labels on any Pods

I didn't use dependsOn, instead I retrieve the service, and use apply to configure some values of another resource with the service's cluster_ip

Code:

service = pulumi_kubernetes.core.v1.Service.get(
            resource_name="kube-dns",
            id="kube-system/kube-dns",
            opts=ResourceOptions(provider=kube_provider),
        )
......
k8s.core.v1.ConfigMap(
                    resource_name=resource_name,
                    # replace __MY__DNS__SERVER__ with the cluster_ip of kube-dns for the configmap
                    data={
                        key: service.spec.apply(
                            lambda s: value.replace(
                                "__MY__DNS__SERVER__", s.cluster_ip
                            )
                        )
                    },
                    metadata=metadata,
                    opts=resource_opts,
                )

Feels like this is the issue? https://github.com/pulumi/pulumi-kubernetes/blob/master/provider/pkg/provider/provider.go#L2009-L2053

More specifically here: https://github.com/pulumi/pulumi-kubernetes/blob/master/provider/pkg/await/await.go#L310

This only skips await logic based on inputs, and these aren't inputs when you're using .get

Expected Behavior

We expect the skipAwait annotation to resolve that. The provider is explicitly checking for an annotation value of "true".

Steps to reproduce

TBD

Output of pulumi about

tbd

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).