Issue : Error creating load balancer (will retry): failed to ensure load balancer for service kube-system/abc-xyz-nginx-ingress-controller: timed out waiting for the condition

[chetanku]

I am following this doc to set up an ingress controller to an internal virtual network in my cluster.

https://docs.microsoft.com/en-us/azure/aks/ingress-internal-ip

but after the below step of installing the Nginx controller, no IP is assigned in the external IP column in my controller even after waiting for a long time.

helm install stable/nginx-ingress --namespace kube-system -f internal-ingress.yaml

I have created an internal load balancer in MC_ resource group with a static IP address. (I tried dynamic as well). I see my service principal has the network contributor permissions in the subnet as well.

 kubectl get service -l app=nginx-ingress --namespace kube-system
NAME                                               TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
abc-xyz-nginx-ingress-controller        LoadBalancer   10.0.203.34    <pending>     80:32510/TCP,443:30996/TCP   7m
abc-xyz-nginx-ingress-default-backend   ClusterIP      10.0.240.115   <none>        80/TCP                       7m

When I run this is what I see in events.

kubectl describe svc abc-xyz nginx-ingress-controller -n kube-system 

Events:
  Type     Reason                      Age               From                Message
  ----     ------                      ----              ----                -------
  Normal   EnsuringLoadBalancer        3m (x8 over 14m)  service-controller  Ensuring load balancer
  Warning  CreatingLoadBalancerFailed  3m (x8 over 14m)  service-controller  Error creating load balancer (will retry): failed to ensure load balancer for service kube-system/abc-xyz-nginx-ingress-controller: timed out waiting for the condition

When I look at pod logs I see this errors:

kubectl log abc-xyz-nginx-ingress-controller-7b4755766d-bm9dq -n kube-system

E1009 19:49:06.249065      10 streamwatcher.go:109] Unable to decode an event from the watch stream: stream error: stream ID 21; INTERNAL_ERROR
E1009 19:49:06.249098      10 streamwatcher.go:109] Unable to decode an event from the watch stream: stream error: stream ID 25; INTERNAL_ERROR
E1009 19:49:06.249512      10 streamwatcher.go:109] Unable to decode an event from the watch stream: stream error: stream ID 19; INTERNAL_ERROR
E1009 19:49:06.249602      10 streamwatcher.go:109] Unable to decode an event from the watch stream: stream error: stream ID 23; INTERNAL_ERROR
E1009 19:49:06.249684      10 streamwatcher.go:109] Unable to decode an event from the watch stream: stream error: stream ID 31; INTERNAL_ERROR
E1009 19:56:31.197139      10 streamwatcher.go:109] Unable to decode an event from the watch stream: stream error: stream ID 81; INTERNAL_ERROR
E1009 19:56:31.197145      10 streamwatcher.go:109] Unable to decode an event from the watch stream: stream error: stream ID 79; INTERNAL_ERROR
E1009 19:59:06.480846      10 streamwatcher.go:109] Unable to decode an event from the watch stream: stream error: stream ID 85; INTERNAL_ERROR
E1009 19:59:06.481064      10 streamwatcher.go:109] Unable to decode an event from the watch stream: stream error: stream ID 83; INTERNAL_ERROR
E1009 19:59:06.481194      10 streamwatcher.go:109] Unable to decode an event from the watch stream: stream error: stream ID 87; INTERNAL_ERROR

Can someone please help?

[mimckitt]

Thanks for the feedback! We are currently investigating and will update you shortly.

[chetanku]

@MicahMcKittrick-MSFT Thanks.

[chetanku]

@MicahMcKittrick-MSFT so just to add more information. the way I am doing this creating an internal load balancer manually in the MC_ resource group and I am trying to allocate Private IP address of the LB to the ingress-internal .yaml, like below. Can it be done this way?

controller:
  service:
    loadBalancerIP: Private IP address
    annotations:
      service.beta.kubernetes.io/azure-load-balancer-internal: "true"

After digging into log analytics :

azure_backoff.go:364] processHTTPRetryResponse: backoff failure, will retry, HTTP response=400, err=network.LoadBalancersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="PrivateIPAddressIsAllocated" Message="IP configuration /subscriptions//resourceGroups/MC_/providers/Microsoft.Network/loadBalancers/kubernetes-internal/frontendIPConfigurations/***** is using the private IP address PRIVATEIPADDRESS which is already allocated to resource /subscriptions//resourceGroups/MC_/providers/Microsoft.Network/loadBalancers/********." Details=[]

But if I dont create the internal load balancer manually and let helm install create it from internal-ingress.yaml , I see an IP getting assigned to the externalIP column.

So the question is, Can I assign an existing load balancer IP to the ingress controller?

[mimckitt]

@chetanku I ran through the doc and was able to get it to work

micah@Azure:~$ kubectl get service -l app=nginx-ingress --namespace kube-system
NAME                                              TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)                      AGE
littering-ostrich-nginx-ingress-controller        LoadBalancer   10.0.56.110    40.121.90.53   80:31399/TCP,443:30346/TCP   12d
littering-ostrich-nginx-ingress-default-backend   ClusterIP      10.0.59.153    <none>         80/TCP                       12d
pouring-fly-nginx-ingress-controller              LoadBalancer   10.0.219.205   10.240.0.42    80:30226/TCP,443:30764/TCP   5m
pouring-fly-nginx-ingress-default-backend         ClusterIP      10.0.103.200   <none>         80/TCP                       5m



micah@Azure:~$ kubectl describe svc pouring-fly-nginx-ingress-controller -n kube-system
Name:                     pouring-fly-nginx-ingress-controller
Namespace:                kube-system
Labels:                   app=nginx-ingress
                          chart=nginx-ingress-0.28.3
                          component=controller
                          heritage=Tiller
                          release=pouring-fly
Annotations:              service.beta.kubernetes.io/azure-load-balancer-internal=true
Selector:                 app=nginx-ingress,component=controller,release=pouring-fly
Type:                     LoadBalancer
IP:                       10.0.219.205
IP:                       10.240.0.42
LoadBalancer Ingress:     10.240.0.42
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  30226/TCP
Endpoints:                10.244.0.28:80
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  30764/TCP
Endpoints:                10.244.0.28:443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason                Age   From                Message
  ----    ------                ----  ----                -------
  Normal  EnsuringLoadBalancer  6m    service-controller  Ensuring load balancer
  Normal  EnsuredLoadBalancer   5m    service-controller  Ensured load balancer

Although it does say External IP it is actually still isolated to your Virtual Network. So External part is a bit confusing. If you were creating an actual public IP address you would have to follow this doc instead

https://docs.microsoft.com/en-us/azure/aks/static-ip

From the error you are getting it seems the IP you are trying to assign is already in use in that subnet. I would suggest trying a different one to see if you get the same error.

[chetanku]

@MicahMcKittrick-MSFT So I can't assign a pre-existing Load Balancer IP to the controller? I will have to create the Load Balancer through helm install stable/nginx-ingress --namespace kube-system -f internal-ingress.yaml

[mimckitt]

Correct. You need to actually setup the LB and assign the IP during creation. If you already have a LB created then it already has an IP address associated. You can still use that load balancer you would just want to associate the service with that LB and the correct IP.

You can also see by following the doc that the external IP is not actually public as I mentioned:

[chetanku]

@MicahMcKittrick-MSFT Thanks for your reply. How can I achieve "If you already have a LB created then it already has an IP address associated. You can still use that load balancer you would just want to associate the service with that LB and the correct IP."?

[mimckitt]

Check out this: https://docs.microsoft.com/en-us/azure/aks/egress#create-a-service-with-the-static-ip

Essentially, you are creating a service and assigning it to the LoadBalancer type and setting the IP and the port you want to associate.

[mimckitt]

@chetanku were you able to read through the doc I provided and determine if it helped?

[chetanku]

@MicahMcKittrick-MSFT Thanks for checking, I am trying to understand and connect assigning IP at controller level like mentioned in https://docs.microsoft.com/en-us/azure/aks/ingress-internal-ip , and assigning IP at service level https://docs.microsoft.com/en-us/azure/aks/egress#create-a-service-with-the-static-ip

[mimckitt]

@iainfoulds could you help clarify on this scenario?

[iainfoulds]

They're two different things. The internal IP address for an ingress resource means that only an internal IP address is assigned to your ingress controller for use with internal apps. There is no external IP address assigned that would permit external traffic from reaching the services that get deployed behind that internal-only ingress controller.

The doc for egress traffic is for when you want any outbound traffic to always be presented with the same IP address. By default, egress traffic uses the first IP address it finds on a load balancer. If only one IP address is configured on the load balancer, that's what gets used. Defining an egress address is more for when you have multiple IP addresses configured on a single load balancer.

I don't understand the scenario here, as if you're wanting the same internal IP address to be used for egress traffic, you shouldn't need to provide any additional configuration. That IP address should be what's presented on outbound traffic, unless you're defining multiple internal IP addresses on the load balancer?

[chetanku]

@iainfoulds Thanks for the detail. Let me rephrase what I was trying to do
Method which did not work:

1. Create an internal load balancer from azure portal in the MC_ resource group (resource group created during AKS creation)
2. Assign the Loadbalancer's IP to the ingress controller via -- (https://docs.microsoft.com/en-us/azure/aks/ingress-internal-ip )

The above way didn't work and I got the error mentioned in the comment -- #16516 (comment)

So my question is in order to assign an Internal Load Balancer IP Address to ingress controller the Load Balancer has to be created via aks using helm install stable/nginx-ingress --namespace kube-system -f internal-ingress.yaml OR is there anyway to assign an IP address of an already created/existing load balancer to a new ingress controller ?

[iainfoulds]

You'd need to let AKS create the load balancer resources in Azure for your services rather than trying to manually create them ahead of them and then use them in AKS. Just create the service through the Kubernetes API, and let the networking plugin create and configure the appropriate Azure resources.

The Helm chart is a simple way of deploying an ingress controller, but is basically just wrapping the NGINX ingress controller - https://github.com/nginxinc/kubernetes-ingress. You can install that manually, and there are other ingress controllers for Kubernetes such as Heptio Contour or Træfik. You don't have to use Helm, it's just an easy wrapper for the Kubernetes resources. You're free to use the install method and ingress controller that you prefer, we don't limit to only using Helm and NGINX ingress.

@MicahMcKittrick-MSFT As there's no doc action here, #please-close

[chetanku]

@iainfoulds so basically we can't. thanks for the clarification.

[Mike-Ubezzi-MSFT]

@chetanku We will now proceed to close this thread. If there are further questions regarding this matter, please comment and we will gladly continue the discussion.

[mimckitt]

Thanks for confirming @iainfoulds. Appreciate it :)

[p10tyr]

So I am following https://docs.microsoft.com/en-us/azure/aks/ingress-static-ip
which says

• create the static ip resource in the group where the nodes are
• create a dns resource if needed
• install nginx using helm install --name az-nginx stable/nginx-ingress --set rbac.create=false --set controller.replicaCount=2 --set controller.service.loadBalancerIP="0.0.0.0" where 0.0.0.0 is the IP address reported when it was created.

I am getting the error from the original post and the comment before says we cannot do this?

This is a bit confusing now.. can somebody please clarify somewhere how to do this.

I want to create the static addres via Azure so it doesnt go away. If i just create nginx it created a new static_ip resource for me but when i delete nginx that resource goes away. As described in the doc I have linked, it demonstrates on how to do what I want... but its not working

Error creating load balancer (will retry): failed to ensure load balancer for service default/az-nginx-nginx-ingress-controller: timed out waiting for the condition

I am not using RBAC and also my user does not have permissions to the MC_ group. But Kubernetes is delegated and it can create things as it sees necessary.

--EDIT
We created the kube using Application Routing and that IP was already associated with the AZ LoadBalancer. So technically this error message means the IP is already assigned.

We recreated the cluster with that option turned off and created the nginx loadbalncer manually using a bare IP address.