Changing to IP mode breaks

[olespagnon-vdm]

Bug Description

When changing from alb.ingress.kubernetes.io/target-type: instance to alb.ingress.kubernetes.io/target-type : ip
Load balancer fails to change the targetgroup and I get this error:
{"level":"error","ts":"2025-03-20T13:04:05Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-mediaspo-servicea-80a47aec1e","namespace":"mynamespace"},"namespace":"mynamespace","name":"k8s-mediaspo-servicea-80a47aec1e","reconcileID":"f931fc53-3c5d-4f1d-a62d-44f589c0d7f2","error":"service type must be either 'NodePort' or 'LoadBalancer': mynamespace/service-api-accessservice"}

Which is weird because some of the api were properly upudated and work as intended. While some don't.
In ArgoCD which I use to deploy I get this error on the ingress resource :
Failed deploy model due to timed out waiting for the condition

Steps to Reproduce

• Step-by-step guide to reproduce the bug: Change target mode from instance to ip.
• Manifests applied while reproducing the issue:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/actions.service-api-logservice: >-
      {"Type":"redirect","RedirectConfig":{"Path":"/log-service/api/","Port":"443","Protocol":"HTTPS","Query":"#{query}","StatusCode":"HTTP_301"}}
    alb.ingress.kubernetes.io/actions.ssl-redirect: >-
      {"Type": "redirect", "RedirectConfig":{ "Protocol": "HTTPS", "Port":
      "443", "StatusCode": "HTTP_301"}}
    alb.ingress.kubernetes.io/group.name: service-alb-company
    alb.ingress.kubernetes.io/healthcheck-interval-seconds: '5'
    alb.ingress.kubernetes.io/healthcheck-path: /log-service/api/index.html
    alb.ingress.kubernetes.io/healthcheck-timeout-seconds: '2'
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/ssl-redirect: '443'
    alb.ingress.kubernetes.io/success-codes: '200'
    alb.ingress.kubernetes.io/target-group-attributes: load_balancing.algorithm.type=least_outstanding_requests
    alb.ingress.kubernetes.io/target-type: ip
    argocd.argoproj.io/tracking-id: >-
      helm-api-logservice:networking.k8s.io/Ingress:company/ingress-api-logservice
    kubernetes.io/ingress.class: alb
    service.beta.kubernetes.io/aws-load-balancer-name: service-alb-company
  creationTimestamp: '2023-10-03T10:54:58Z'
  finalizers:
    - group.ingress.k8s.aws/service-alb-company
  generation: 2
  labels:
    app.kubernetes.io/instance: testing
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: helm-company
    app.kubernetes.io/version: 1.16.0
    argocd.argoproj.io/instance: helm-api-logservice
    helm.sh/chart: helm-company-0.1.0
  name: ingress-api-logservice
  namespace: mynamespace
  resourceVersion: '526357625'
  uid: dabb1e4a-2a00-4b12-8552-8beeee6ea855
spec:
  rules:
    - host: api-test.host.io
      http:
        paths:
          - backend:
              service:
                name: service-api-logservice
                port:
                  number: 9511
            path: /log-service/api/
            pathType: Prefix

• Controller logs/error messages while reproducing the issue:
  {"level":"error","ts":"2025-03-20T13:04:05Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-mediaspo-servicea-80a47aec1e","namespace":"mynamespace"},"namespace":"mynamespace","name":"k8s-mediaspo-servicea-80a47aec1e","reconcileID":"f931fc53-3c5d-4f1d-a62d-44f589c0d7f2","error":"service type must be either 'NodePort' or 'LoadBalancer': mynamespace/service-api-accessservice"}

Expected Behavior

Ingress are valid and TargetGroups are updated properly
Actual Behavior

Regression
Was the functionality working correctly in a previous version ? [Yes / No]
If yes, specify the last version where it worked as expected

Current Workarounds

Environment

• AWS Load Balancer controller version: v2.8.2
• Kubernetes version: 1.30
• Using EKS (yes/no), if so version?: eks.27
• Using Service or Ingress: Ingress
• AWS region: eu-west-3
• How was the aws-load-balancer-controller installed:
  • If helm was used then please show output of helm ls -A | grep -i aws-load-balancer-controller
  • aws-load-balancer-controller kube-system 20 2025-03-20 12:51:44.894449318 +0100 CET deployed aws-load-balancer-controller-1.6.1 v2.6.1
  • If helm was used then please show output of helm -n <controllernamespace> get values <helmreleasename>

Name:                   aws-load-balancer-controller
Namespace:              kube-system
CreationTimestamp:      Mon, 02 Oct 2023 20:53:08 +0200
Labels:                 app.kubernetes.io/instance=aws-load-balancer-controller
                       app.kubernetes.io/managed-by=Helm
                       app.kubernetes.io/name=aws-load-balancer-controller
                       app.kubernetes.io/version=v2.6.1
                       helm.sh/chart=aws-load-balancer-controller-1.6.1
Annotations:            deployment.kubernetes.io/revision: 16
                       meta.helm.sh/release-name: aws-load-balancer-controller
                       meta.helm.sh/release-namespace: kube-system
Selector:               app.kubernetes.io/instance=aws-load-balancer-controller,app.kubernetes.io/name=aws-load-balancer-controller
Replicas:               2 desired | 2 updated | 2 total | 2 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
 Labels:           app.kubernetes.io/instance=aws-load-balancer-controller
                   app.kubernetes.io/name=aws-load-balancer-controller
 Annotations:      kubectl.kubernetes.io/restartedAt: 2024-10-01T10:32:36+02:00
                   prometheus.io/port: 8080
                   prometheus.io/scrape: true
 Service Account:  aws-load-balancer-controller
 Containers:
  aws-load-balancer-controller:
   Image:       602401143452.dkr.ecr.eu-west-3.amazonaws.com/amazon/aws-load-balancer-controller:v2.8.2
   Ports:       9443/TCP, 8080/TCP
   Host Ports:  0/TCP, 0/TCP
   Args:
     --cluster-name=EKScluster
     --ingress-class=alb
     --aws-region=eu-west-3
     --aws-vpc-id=vpc-0307f743efe855b0a
   Liveness:     http-get http://:61779/healthz delay=30s timeout=10s period=10s #success=1 #failure=2
   Environment:  <none>
   Mounts:
     /tmp/k8s-webhook-server/serving-certs from cert (ro)
 Volumes:
  cert:
   Type:               Secret (a volume populated by a Secret)
   SecretName:         aws-load-balancer-tls
   Optional:           false
 Priority Class Name:  system-cluster-critical
 Node-Selectors:       <none>
 Tolerations:          dedicated=kube-system:NoSchedule
Conditions:
 Type           Status  Reason
 ----           ------  ------
 Progressing    True    NewReplicaSetAvailable
 Available      True    MinimumReplicasAvailable
OldReplicaSets:  aws-load-balancer-controller-d75c5b56d (0/0 replicas created), aws-load-balancer-controller-6b56994c5 (0/0 replicas created), aws-load-balancer-controller-59598f7745 (0/0 replicas created), aws-load-balancer-controller-565cbf9bdf (0/0 replicas created), aws-load-balancer-controller-66f6c679b7 (0/0 replicas created), aws-load-balancer-controller-585c4fd77 (0/0 replicas created), aws-load-balancer-controller-756cbf5fd9 (0/0 replicas created), aws-load-balancer-controller-7c68c6d77 (0/0 replicas created), aws-load-balancer-controller-9556d967d (0/0 replicas created), aws-load-balancer-controller-676b8fb796 (0/0 replicas created)
NewReplicaSet:   aws-load-balancer-controller-84f5c87869 (2/2 replicas created)
Events:          <none>

• Current state of the Ingress/Service configuration:
  • kubectl describe ingressclasses

Name:         alb
Labels:       app.kubernetes.io/instance=aws-load-balancer-controller
             app.kubernetes.io/managed-by=Helm
             app.kubernetes.io/name=aws-load-balancer-controller
             app.kubernetes.io/version=v2.6.1
             helm.sh/chart=aws-load-balancer-controller-1.6.1
Annotations:  meta.helm.sh/release-name: aws-load-balancer-controller
             meta.helm.sh/release-namespace: kube-system
Controller:   ingress.k8s.aws/alb
Events:       <none>

• kubectl -n <appnamespace> describe ingress <ingressname>

[olespagnon-vdm]

Okay so uppon further investigating I found something weird. The new target groups are working as intended but the old target groups remains. Should'nt they be removed once the ingress is updated ?
Why is the ALB LC still trying to validate the ingress with the old targets ?

[zac-nixon]

Hello, please fill out this part of the template so we can properly help you.

AWS Load Balancer controller version:
Kubernetes version:
Using EKS (yes/no), if so version?:
Using Service or Ingress:
AWS region:

specifically the AWS Load Balancer controller version and Kubernetes version.

[olespagnon-vdm]

My bad forgot about this. I have edited my original post.

[zac-nixon]

Error is coming from here, https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/main/pkg/backend/endpoint_resolver.go#L87

I wonder if it's because all you did was change the target type while using the same service. Instance and IP targets use different service types: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.12/guide/ingress/annotations/#target-type

[olespagnon-vdm]

Hi, thanks for the response. You might be right. I'm using a helm template, inflated with ArgoCD and deployed by ArgoCD. But upon further inspection, it seems that indeed the modification of the Service type did not recreate a new service bu changed the existing ones.
What do you suggest I me doing ? Should I delete the TargetGroupBinding in the cluster and the aws resources would be deleted ? Should I delete both ? Or should I wait for a 'fix' for this case ?

[zac-nixon]

I think this scenario would fall under not supported behavior, moving to instance -> ip requires a service update and corresponding target group update. If you want to clean up the target group, I would suggest deleting the unused binding and deleting the target group