Could not find any suitable subnets for creating the ELB

[dbluxo]

What happened:

Since Amazon EKS version 1.19, the kubernetes.io/cluster/<cluster-name> subnet tag is no longer set automatically.
As also described in the release notes for 1.19

Starting with 1.19, Amazon EKS no longer adds the kubernetes.io/cluster/ tag to subnets passed in during cluster creation.

As a result, we can no longer provision internal load balancers via Kubernetes using a Kubernetes service:

27s         Normal    EnsuringLoadBalancer     service/ingress-nginx-controller                Ensuring load balancer
63s         Normal    ScalingReplicaSet        deployment/ingress-nginx-controller             Scaled up replica set ingress-nginx-controller-bd6999c6b to 1
27s         Warning   SyncLoadBalancerFailed   service/ingress-nginx-controller                Error syncing load balancer: failed to ensure load balancer: could not find any suitable subnets for creating the ELB

We have configured the following in the nginx-ingress Kubernetes service:

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
    service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:eu-central-1:000000000000000 # (anonymized)
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"

Only by "manually" setting the kubernetes.io/cluster/<cluster-name> tag we can solve the problem.

Isn't this tag optional and it should work without it:question: :thinking:

All our private subnets for internal load balancer creation have the kubernetes.io/role/internal-elb: 1 tag set.

What you expected to happen:

We expect to be able to create the load balancer without kubernetes.io/cluster/<cluster-name> subnet tag.

Environment:

• Kubernetes version (use kubectl version):

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.3", GitCommit:"01849e73f3c86211f05533c2e807736e776fcf29", GitTreeState:"clean", BuildDate:"2021-02-18T12:10:55Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.6-eks-49a6c0", GitCommit:"49a6c0bf091506e7bafcdb1b142351b69363355a", GitTreeState:"clean", BuildDate:"2020-12-23T22:10:21Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration: AWS

This issue came out of that issue: eksctl-io/eksctl#3341

/kind bug

[k8s-triage-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fredgate]

/reopen
Problem is still present.
What is the right way to automate this without manual intervention?

[k8s-ci-robot]

@fredgate: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen
Problem is still present.
What is the right way to automate this without manual intervention?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.