Target group has empty instances list after creation

[abubyr]

What happened: When creating k8s service with service.beta.kubernetes.io/aws-load-balancer-type: nlb , we constantly get an empty list target group

What you expected to happen: instances are registered in the target group just as part of target group creation process

How to reproduce it (as minimally and precisely as possible):
Create a pod and service like

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  containers:
  - image: nginx
    imagePullPolicy: IfNotPresent
    name: nginx
    ports:
    - containerPort: 80
      protocol: TCP

---
apiVersion: v1
kind: Service
metadata:
  name: nlb-sample-service
  namespace: default
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    service.beta.kubernetes.io/aws-load-balancer-internal: 'true'
spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
  type: LoadBalancer
  selector:
    app: nginx

Anything else we need to know?:
Workaround: restart of aws-cloud-controller-manager daemonset, which triggers ensureLoadBalancerv2() function again and instances registration appears

Environment:

• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"archive", BuildDate:"2021-09-23T18:53:32Z", GoVersion:"go1.17.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1+k0s", GitCommit:"b230d3e4b9d6bf4b731d96116a6643786e16ac3f", GitTreeState:"clean", BuildDate:"2021-12-24T12:59:32Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
• Cloud provider or hardware configuration: AWS EC2
• OS (e.g. from /etc/os-release): Ubuntu 20.04
• Kernel (e.g. uname -a): 5.11.0-1027-aws
• Install tools:
• Others:

/kind bug

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[olemarkus]

/remove-lifecycle rotten