Ingress await semantics are too strict

[lkt82]

Hi

After upgrading to version 3.10.1 We are getting this error on creation of a ingress object

the Kubernetes API server reported that "ingress-system/gateway-default-8i88hdw3" failed to fully initialize or become live: 'gateway-default-8i88hdw3' timed out waiting to be Ready
Ingress has at least one rule that does not target any Service. Field '.spec.rules[].http.paths[].backend.service.name' may not match any active Service

We are using a default Ingress object without any http rules other then host. This has allowed us to share TLS setting with other Ingress objects. This is suppoted by the nginx controller we use.

var defaultIngress = new Pulumi.Kubernetes.Networking.V1.Ingress(
	GetLogicalName("default"), new IngressArgs
	{
		Metadata = new ObjectMetaArgs
		{
			Namespace = _namespace.Metadata.Apply(c => c.Name),
			Annotations =
			{
				{ "nginx.ingress.kubernetes.io/ssl-redirect", "false" }
			}
		},
		Spec = new IngressSpecArgs
		{
			IngressClassName = "nginx",
			Tls ={
				new IngressTLSArgs
				{
					Hosts = PublicIpFqdn,
					SecretName = _certificate.Apply(c=> c.KubernetesSecretName)!
				}
			},
			Rules = new IngressRuleArgs
			{
				Host = PublicIpFqdn
			}
		}

	}, new()
	{
		Parent = this,
		DependsOn = _chart.GetChildren()
	});

Issue details

Steps to reproduce

1. Provision a ingress object without any http rules

Expected: The update to have no breaking changes
Actual: Did have a breaking change

[mikhailshilkov]

@viveklak It sounds related to #1795. Could you check?

[viveklak]

FWIW before #1795 networking.v1.Ingress was not being waited on at all - essentially the equivalent of setting pulumi.com/skipAwait: "true". This might have worked by accident in the past. I believe you are running into this assertion in the code:

pulumi-kubernetes/provider/pkg/await/ingress.go

Lines 365 to 369 in 920ed43

	if rule.HTTP == nil {
	iia.config.logStatus(diag.Error, fmt.Sprintf("expected value %q is unset for ingress: %s",
	".spec.rules[*].http", obj.Name))
	return apiVersion, false
	}

which was added in #928. I can't find a reference in the ingress spec that suggests rules without HTTP blocks are invalid. I will look a bit deeper. For the time-being you may be able to workaround this by adding the skipAwait annotation above.

[lkt82]

Hi @viveklak I will try to see if skipAwait works.

It's hard for me to say if this usage is invalid or not but what I can say is that we use this in production and ingress-nginx allows it.

[viveklak]

Hi @viveklak I will try to see if skipAwait works.

It's hard for me to say if this usage is invalid or not but what I can say is that we use this in production and ingress-nginx allows it.

I am a bit puzzled about this myself. I will check with @lblackstone when he is back from vacation since it struck me as odd as well. For the time being the skipAwait workaround should work - since essentially that was the behavior you were opaquely relying on previously.

[sziegler-skyhook]

I am also running into this problem using the AWS ALB Controller + K8S Ingresses.

The ALB controller uses annotations to forward traffic to a specific target group and therefore the backend service name is not an actual K8S service. (see here: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/guide/ingress/annotations/#traffic-routing)

Here is my use case:

kind: Ingress
metadata:
  name: ingress-a
  namespace: default
  annotations:
    alb.ingress.kubernetes.io/actions.forward-to-9443: '{"type":"forward","targetGroupARN":"{{ .Values.aws.primary.TargetGroup }}"}'
  spec:
  rules:
  - http:
      paths:
        - pathType: Prefix
          path: /a
          backend:
            service:
              name: forward-to-9443
              port:
                name: use-annotation
        - pathType: Prefix
          path: /b
          backend:
            service:
              name: forward-to-9443
              port:
                name: use-annotation
        - pathType: Exact
          path: /c
          backend:
            service:
              name: forward-to-9443
              port:
                name: use-annotation

This results in error: Ingress has at least one rule that does not target any Service. Field '.spec.rules[].http.paths[].backend.service.name' may not match any active Service because the service forward-to-9443 does not exist, it is an AWS ALB Annotation.

[viveklak]

I renamed the issue to use this as a tracking issue to consider relaxing the constraints currently imposed for ingress await logic. The suggested workaround for the moment is to use the pulumi.com/skipAwait: "true" annotation on the affected ingress objects. However, attempts to extract loadbalancer addresses etc. will have to resort to refreshing or polling the ingress object in such cases.

[sziegler-skyhook]

I agree on both points, skipAwait is a good workaround but the requirements should be relaxed if possible.

[lblackstone]

Yeah, I expect that we're being too strict on the await logic here. This logic was added back in 2019, so I don't recall the exact reasons for this assertion, but it's most likely just a misunderstanding of the allowed options.