Ingress .status.loadBalancer field was not updated with a hostname/IP address with traefik ingress controller configured as NodePort

[sudocovik]

Hello!

• Vote on this issue by adding a 👍 reaction
• To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already)

Issue details

After upgrading from 3.9.0 to version 3.10.1 (3.10.0 is also affected) the following messages in stdout are received when creating Ingress object:

 ~  kubernetes:networking.k8s.io/v1:Ingress new-web-access updating [diff: ~metadata]; [1/3] Finding a matching service for each Ingress path

 ~  kubernetes:networking.k8s.io/v1:Ingress new-web-access updating [diff: ~metadata]; No matching service found for ingress rule: "/" -> "new-web-w439tjcc"

 ~  kubernetes:networking.k8s.io/v1:Ingress new-web-access updating [diff: ~metadata]; [2/3] Waiting for update of .status.loadBalancer with hostname/IP

These messages appear and stay like this until timeout kicks in and generates similar error message:

kubernetes:networking.k8s.io/v1:Ingress (new-web-access):
    error: 2 errors occurred:
    	* the Kubernetes API server reported that "vfm/new-web-access-fr91qlju" failed to fully initialize or become live: 'new-web-access-fr91qlju' timed out waiting to be Ready
    	* Ingress .status.loadBalancer field was not updated with a hostname/IP address.
        for more information about this error, see https://pulumi.io/xdv72s

Happens in local environment using k3d v5.1.0 and traefik helm chart v10.6.2.
It's important to mention that the deployment, service, ingress and ingress controller are working as expected because I can access them throughout my localhost (I've destroyed and recreated the cluster to eliminate possible working state because of previous deployments).
The great mystery is why pulumi expects status.loadBalancer field when my ingress-controller is configured as NodePort, not LoadBalancer.

The workaround is to use 'pulumi.com/skipAwait': 'true' annotation on the Ingress object so this is most likely related to recent changes in await logic #1795

Steps to reproduce

1. Bootstrap local cluster using k3d (most likely works with other local cluster providers like kind)
2. Add a sample Deployment and Service object (can be default nginx image)
3. Add an Ingress resource connected to previously defined Service without host. Example:

   new k8s.networking.v1.Ingress('new-web-access', {
       metadata: {
           annotations: {
               // 'pulumi.com/skipAwait': 'true'
           }
       },
       spec: {
           rules: [{
               http: {
                   paths: [{
                       path: '/',
                       pathType: 'Prefix',
                       backend: {
                           service: {
                               name: '<<service-name>>',
                               port: {
                                   number: 80
                               }
                           }
                       }
                   }]
               }
           }]
       }
   })

4. Add traefik helm v3.Chart configured as a NodePort not LoadBalancer

   new k8s.helm.v3.Chart('ingress-controller', {
       chart: 'traefik',
       version: '10.6.2',
       fetchOpts: {
           repo: 'https://helm.traefik.io/traefik',
       },
       values: {
           service: {
               type: 'NodePort'
           },
           ports: {
               web: {
                   nodePort: 32080
               },
               websecure: {
                   expose: false
               }
           }
       }
   })

Expected: Deployment should progress and finish successfully
Actual: After 10 minute timeout the deployment fails

[viveklak]

Seems to be related to #1810 - since as mentioned there, v1 ingress has been added to await logic which was silently being skipped in the past.

The great mystery is why pulumi expects status.loadBalancer field when my ingress-controller is configured as NodePort, not LoadBalancer.

That is a bit of red-herring - the provider tries to inject a useful message for the failure but clearly this doesn't apply in all cases. The specific issue seems to be this: No matching service found for ingress rule: "/" -> "new-web-w439tjcc" Does the service pointed to actually exist?

If its created by the helm chart, you might want to create the ingress object after the helm chart is ready (add chart.ready to dependsOn block)

[viveklak]

Closing in favor of #1810

[sudocovik]

The specific issue seems to be this: No matching service found for ingress rule: "/" -> "new-web-w439tjcc" Does the service pointed to actually exist?

Yes, the service exists. It's a service for serving frontend and the frontend can be accessed through browser. Also confirmed with kubectl that service exists and only one instance is there.

If its created by the helm chart, you might want to create the ingress object after the helm chart is ready (add chart.ready to dependsOn block)

Yes, it depends on traefik helm chart. Also, in production, traefik is running for months, yet every new deployment of the frontend image has the same problem.

[ivanaliptakova]

I had the same issue in my playground project. Then I deleted pulumi stack and all resources and recreated it. Since then issue dissipated. I had a feeling that everything started after 2 pulumi up commands were running in the same time and one failed. I tried pulumi refresh which ended successfully, but the issue were still there.

[viveklak]

I had the same issue in my playground project. Then I deleted pulumi stack and all resources and recreated it. Since then issue dissipated. I had a feeling that everything started after 2 pulumi up commands were running in the same time and one failed. I tried pulumi refresh which ended successfully, but the issue were still there.

Sorry I am not sure I understand. Is the problem still there? Are you saying refreshing caused the dissipated problem to reoccur? Can you make sure you are running with v3.12.2 of the Kubernetes provider or newer? pulumi about in the project repo should confirm that.