TargetGroupBinding is not found discovered after few horus

[pkruk]

Bug Description

We're using TargetGroupBinding in one of our applications and today we dropped target-group via terraform, but this alert was triggered after few hours.

sum(aws_api_calls_total{error_code="TargetGroupNotFound"}) by (source, error_code, status_code) > 0

There is any way to monitor missing or incorrect targetgroup definition used in TGB? AWS-LBC is able to spot this mismatches?

Steps to Reproduce

• Create TargetGroupBiding with existing targetgroup
• delete targetgroup
• this metric will fire only when pod will be rotated or aws-lc is rotated

Expected Behavior

Spot and fire errors when there is any mismatch

Actual Behavior

Regression
Was the functionality working correctly in a previous version ? [Yes / No]
If yes, specify the last version where it worked as expected

Current Workarounds

Environment

• AWS Load Balancer controller version: v2.7.1
• Kubernetes version: 1.30 and 1.32
• Using EKS (yes/no), if so version?: yes 1.32, 1.30
• Using Service or Ingress:
• AWS region: eu-west-1
• How was the aws-load-balancer-controller installed:
  • If helm was used then please show output of helm ls -A | grep -i aws-load-balancer-controller
  • If helm was used then please show output of helm -n <controllernamespace> get values <helmreleasename>
  • If helm was not used, then copy/paste the exact command used to install the controller, including flags and options. helm managed by argocd
• Current state of the Controller configuration:
  • kubectl -n <controllernamespace> describe deployment aws-load-balancer-controller
• Current state of the Ingress/Service configuration:
  • kubectl describe ingressclasses
  • kubectl -n <appnamespace> describe ingress <ingressname>
  • kubectl -n <appnamespace> describe svc <servicename>

Possible Solution (Optional)

Contribution Intention (Optional)

• Yes, I'm willing to submit a PR to fix this issue
• No, I cannot work on a PR at this time

Additional Context

[kellyyan]

Hi, thanks for reaching out. The LBC runs reconcile cycle every 10 hrs on all resources by default. So there is no way to detect any mismatches before that cycle unless the TGB itself is updated. We don't watch for target group deletion events from controller as of today.

To catch such errors earlier, you can set up a lower --sync-period for controller so reconciles happen more frequently.

[pkruk]

ah I see, thank you for your response :)