Await logic with Statefulsets doesn't work

[jaxxstorm]

What happened?

If you define a statefule set exactly as in the StatefulSet documentation, the service never becomes healthy:

This is because statefulesets have a required input of serviceName and we pass the service auto generated name. The service gets created first (unlike most deployment examples) and there are no healthy pods to target, causing an infinite loop:

* resource default/nginx-service-1bb29ad6 was successfully created, but the Kubernetes API server reported that it failed to fully initialize or become live: Resource operation was cancelled for "nginx-service-1bb29ad6"
    	* Service does not target any Pods. Selected Pods may not be ready, or field '.spec.selector' may not match labels on any Pods

Steps to reproduce

Create a Pulumi program like so:

import * as kubernetes from '@pulumi/kubernetes';

const nginxService = new kubernetes.core.v1.Service('nginx-service', {
  metadata: {
    labels: {
      app: 'nginx',
    },
  },
  spec: {
    ports: [ {
      port: 80,
      name: 'web',
    } ],
    clusterIP: 'None',
    selector: {
      app: 'nginx',
    },
  },
});

const wwwStatefulSet = new kubernetes.apps.v1.StatefulSet('wwwstatefulset', {
  spec: {
    selector: {
      matchLabels: {
        app: 'nginx',
      },
    },
    serviceName: nginxService.metadata.name,
    replicas: 3,
    template: {
      metadata: {
        labels: {
          app: 'nginx',
        },
      },
      spec: {
        terminationGracePeriodSeconds: 10,
        containers: [ {
          name: 'nginx',
          image: 'k8s.gcr.io/nginx-slim:0.8',
          ports: [ {
            containerPort: 80,
            name: 'web',
          } ],
          volumeMounts: [ {
            name: 'www',
            mountPath: '/usr/share/nginx/html',
          } ],
        } ],
      },
    },
    volumeClaimTemplates: [ {
      metadata: {
        name: 'www',
      },
      spec: {
        accessModes: [ 'ReadWriteOnce' ],
        storageClassName: 'my-storage-class',
        resources: {
          requests: {
            storage: '1Gi',
          },
        },
      },
    } ],
  },
});

Expected Behavior

Expected the program to provision

Actual Behavior

The pods never become healthy, and the program times out

Versions used

CLI
Version      3.30.0
Go Version   go1.17.9
Go Compiler  gc

Plugins
NAME        VERSION
aws         5.2.0
docker      3.2.0
kubernetes  3.18.2
nodejs      unknown

Host
OS       darwin
Version  12.3.1
Arch     x86_64

This project is written in nodejs (/Users/lbriggs/.asdf/shims/node v14.16.0)

Current Stack: dev

TYPE                         URN
pulumi:pulumi:Stack          urn:pulumi:dev::stateful_set::pulumi:pulumi:Stack::stateful_set-dev
pulumi:providers:kubernetes  urn:pulumi:dev::stateful_set::pulumi:providers:kubernetes::default_3_18_2
kubernetes:core/v1:Service   urn:pulumi:dev::stateful_set::kubernetes:core/v1:Service::nginx-service


Found no pending operations associated with dev

Backend
Name           pulumi.com
URL            https://app.pulumi.com/jaxxstorm
User           jaxxstorm
Organizations  jaxxstorm, lbrlabs, team-ce, demo, pulumi

NAME                VERSION
@types/node         14.18.13
@pulumi/aws         5.2.0
@pulumi/awsx        0.40.0
@pulumi/kubernetes  3.18.2
@pulumi/pulumi      3.30.0

Additional context

You can work around this issue by adding the skipAwait annotation:

const nginxService = new kubernetes.core.v1.Service('nginx-service', {
  metadata: {
    labels: {
      app: 'nginx',
    },
    annotations: {
        "pulumi.com/skipAwait": "true"
    }
  },
  spec: {
    ports: [ {
      port: 80,
      name: 'web',
    } ],
    clusterIP: 'None',
    selector: {
      app: 'nginx',
    },
  },
});

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[cniessigma]

Did you ever find a workaround to this? Running into the same issue.