GKE Internal LoadBalancer with a given reserved static LoadBalancerIP

[fabioformosa]

What happened:

I've seen it's possible to pass to the helm chart a pre-allocated private IP to the internal load balancer (as the same way as it's possible to do with the external load balancer) through the variable controller.service.internal.loadBalancerIP as shown in controller-service-internal.yaml

I ran the helm chart in a GKE cluster. The Internal Loadbalancer is created, unfortunately with no private IP associated to that. In the GCP UI, I read "This load balancer has no frontend configured".

The error I found in the kubernetes event list of the nginx service is

Error syncing load balancer: failed to ensure load balancer: googleapi: Error 400: Invalid value for field 'resource.IPAddress': '192.168.195.38'. Requested internal IP is outside the network/subnetwork range., invalid

192.168.195.38 is an IP a early reserved creating a GCP private address. It's not part of the subnet for the cluster nodes.
(https://cloud.google.com/compute/docs/ip-addresses/reserve-static-internal-ip-address)
I've also associated an internal domain name to it.

What you expected to happen:
I would like the value controller.service.internal.loadBalancerIP have the effect to assign that IP to the internal load balancer in GCP.

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

helm chart v4.4.0
nginx-ingress-controller 1.5.1

Kubernetes version (use kubectl version):
1.24.4

Environment:

• Cloud provider or hardware configuration: GCP

• OS (e.g. from /etc/os-release):

• Kernel (e.g. uname -a):

• Install tools:

  • GKE created via terraform

• Basic cluster related info:

  • kubectl version v1.22.10
  • kubectl get nodes -o wide

• How was the ingress-nginx-controller installed:

  • If helm was used then please show output of helm ls -A | grep -i ingress
  • If helm was used then please show output of helm -n <ingresscontrollernamepspace> get values <helmreleasename>
  • If helm was not used, then copy/paste the complete precise command used to install the controller, along with the flags and options used
  • if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances

ingress-nginx:
  controller:
    replicaCount: 2
    service:
      internal:
        annotations:
          networking.gke.io/load-balancer-type: Internal
        enabled: true
        loadBalancerIP: 192.168.195.38
      loadBalancerIP: ...

• Current State of the controller:
  • kubectl describe ingressclasses

Name:         nginx
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=...
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.5.1
              helm.sh/chart=ingress-nginx-4.4.0
Annotations:  meta.helm.sh/release-name: ...
              meta.helm.sh/release-namespace: nginx
Controller:   k8s.io/ingress-nginx
Events:       <none>

• kubectl -n <ingresscontrollernamespace> get all -A -o wide
• kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>
• kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>

Name:                     nginx-ingress-ingress-nginx-controller-internal
Namespace:                nginx
Labels:                   app.kubernetes.io/component=controller
                          app.kubernetes.io/instance=...
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=ingress-nginx
                          app.kubernetes.io/part-of=ingress-nginx
                          app.kubernetes.io/version=1.5.1
                          helm.sh/chart=ingress-nginx-4.4.0
Annotations:              meta.helm.sh/release-name: s-nginx-ingress
                          meta.helm.sh/release-namespace: nginx
                          networking.gke.io/load-balancer-type: Internal
Selector:                 app.kubernetes.io/component=controller,app.kubernetes.io/instance=app.kubernetes.io/name=ingress-nginx
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       192.168.192.255
IPs:                      192.168.192.255
IP:                       192.168.195.38
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  32179/TCP
Endpoints:                192.168.160.17:80,192.168.160.232:80
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  30920/TCP
Endpoints:                192.168.160.17:443,192.168.160.232:443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type     Reason                  Age                    From                Message
  ----     ------                  ----                   ----                -------
  Normal   EnsuringLoadBalancer    2m40s (x6 over 6m16s)  service-controller  Ensuring load balancer
  Warning  SyncLoadBalancerFailed  2m36s (x6 over 5m30s)  service-controller  Error syncing load balancer: failed to ensure load balancer: googleapi: Error 400: Invalid value for field 'resource.IPAddress': '192.168.195.38'. Requested internal IP is outside the network/subnetwork range., invalid

• Current state of ingress object, if applicable:

  • kubectl -n <appnnamespace> get all,ing -o wide
  • kubectl -n <appnamespace> describe ing <ingressname>
  • If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag

• Others:

  • Any other related information like ;
    • copy/paste of the snippet (if applicable)
    • kubectl describe ... of any custom configmap(s) created and in use
    • Any other related information that may help

How to reproduce this issue:

Anything else we need to know:

If I don't set the internal load balancer IP then GCP creates an internal loadbalancer with an its random private IP.

[k8s-ci-robot]

@fabioformosa: This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fabioformosa]

I've just found in this google doc page that, when a loadbalancer IP is not passed, Google assigns to the internal load balancer an IP address belonging to the primary IP address range, which is the same used to allocate cluster nodes.

Every subnet must have a primary IP address range. This is the IP address range that GKE uses to allocate IP addresses for internal load balancers and nodes.

Perhaps it's not possible to assign whatever private IP to the helm value controller.service.internal.loadBalancerIP but a private IP address reserved from the same subnet allocated for cluster nodes.

I'm looking for whether it's correct to reserve private IP address in my own, from a range passed to GKE for cluster nodes.

[fabioformosa]

Solved in my own.
At this google doc page is written that

The following Service definition uses the internal-load-balancer-subnet to reference the subnet by name. By default an available IP from the subnet will automatically be chosen. You can also specify the loadBalancerIP but it must be part of the referenced subnet

So my mistake was to set an IP to the helm value controller.service.internal.loadBalancerIP without adding the annotation:
networking.gke.io/internal-load-balancer-subnet: SUBNET_NAME with the subnet name which the reserved private IP address belongs to.

I've just submitted a PR #9406 to enrich the documentation of the helm chart, adding the missing property controller.service.internal.loadBalancerIP.

[longwuyuan]

/remove-kind bug
/kind support

[longwuyuan]

/area documentation

[k8s-ci-robot]

@longwuyuan: The label(s) area/documentation cannot be applied, because the repository doesn't have them.

In response to this:

/area documentation

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[longwuyuan]

/area docs
/kind documentation

[DylenTurnbull]

/assign