Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ALB targets not registered #3569

Closed
janrito opened this issue Feb 10, 2024 · 4 comments
Closed

ALB targets not registered #3569

janrito opened this issue Feb 10, 2024 · 4 comments

Comments

@janrito
Copy link

janrito commented Feb 10, 2024

Describe the bug
I'm trying to use annotations on an Ingress to create an Application Load Balancer for a service in a private, fargate-only EKS cluster.

Here is a list of things that are true or I have tried

  • Everything runs on Fargate, I've registered a few different profiles for coredns, kube-system, cert-manager and default
  • I have the VPC-CNI, kube-proxy and coredns addons installed
  • I installed the cert-manager and ALB controller version 2.7.0 using kubectl apply. There are a couple of things that needed changing including running the cert-manager-webhook on port 10260, and adding region, vpc, cluster name and disabling shield and waf on the alb-controller manifest.
  • The service works, I can curl to the specific pod ip address and I get a successful response. I have tried multiple replicas and they all work.
  • The application load balancer is registered, listener rules created, and a target group is defined, but no targets are registered
  • I have also tried using annotations on the service as a loadbalancer instead of defining an ingress. A very similar thing happens, I get a load balancer, listener rules and a target group with no targets registered.
  • There are no errors, the targets are just not registered. The ingress reconciles fine and the logs on the alb controller show no errors.
  • If i manually register the targets, i can access the service just fine. I did have to annotate the security group of the cluster as load balancer security groups - maybe this is a clue?

Not sure what to try next

Steps to reproduce

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: default
  name: test-deployment
  labels:
    app: test-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: test-pod
  template:
    metadata:
      labels:
        app: test-pod
    spec:
      containers:
        - name: test-container
          image: test-image
          env:
            - name: PORT
              value: '80'
          resources:
            limits:
              memory: 512Mi
              cpu: '0.25'
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  namespace: default
  name: test-service
  labels:
    app: test-service
spec:
  type: NodePort
  ports:
    - name: http
      port: 80
      targetPort: 80
      protocol: TCP
  selector:
    app: test-service
---
apiVersion: networking.k8s.io/v1n
kind: Ingress
metadata:
  name: test-ingress
  namespace: default
  annotations:
    alb.ingress.kubernetes.io/scheme: internal
    alb.ingress.kubernetes.io/target-type: ip # instance types are not available to Fargate
    alb.ingress.kubernetes.io/load-balancer-name: test-alb
    alb.ingress.kubernetes.io/healthcheck-path: /healthz
    alb.ingress.kubernetes.io/security-groups: sg-XXXXXXXXXXX # attach cluster security group to the load balancer
    alb.ingress.kubernetes.io/manage-backend-security-group-rules: 'true' # manage security group rules for the load balancer

    alb.ingress.kubernetes.io/actions.response-503: >
      {"type":"fixed-response","fixedResponseConfig":{"contentType":"text/plain","statusCode":"503","messageBody":"503 error text"}}

  labels:
    app: test-ingress
spec:
  ingressClassName: alb
  rules:
    - http:
        paths:
          - path: /503
            pathType: Exact
            backend:
              service:
                name: response-503
                port:
                  name: use-annotation
          - path: /
            pathType: Prefix
            backend:
              service:
                name: 'test-service'
                port:
                  number: 80

Expected outcome
targets to be registered

Environment

  • AWS Load Balancer controller version 2.7.0
  • Kubernetes version
  • Using EKS (yes/no), if so version? yes, latest

Additional Context:

@janrito
Copy link
Author

janrito commented Feb 12, 2024

seems related to this #2339

@janrito
Copy link
Author

janrito commented Feb 12, 2024

i'm attaching the alb logs

2024-02-12T17:48:48.974+00:00	{"log":"2024-02-12T17:48:48.974879644Z stderr F {\"level\":\"info\",\"ts\":\"2024-02-12T17:48:48Z\",\"logger\":\"backend-sg-provider\",\"msg\":\"created SecurityGroup\",\"name\":\"k8s-traffic-mlkubecluster-a083a9e7d5\",\"id\":\"sg-VVVVVVVVVVVV\"}"}
2024-02-12T17:48:48.974+00:00	{"log":"2024-02-12T17:48:48.974938249Z stderr F {\"level\":\"info\",\"ts\":\"2024-02-12T17:48:48Z\",\"logger\":\"controllers.ingress\",\"msg\":\"SG configured via annotation\",\"LB SGs\":[\"sg-XXXXXXXXXXXXX\",\"sg-VVVVVVVVVVVV\"],\"backend SG\":\"sg-VVVVVVVVVVVV\"}"}
2024-02-12T17:48:48.975+00:00	{"log":"2024-02-12T17:48:48.975231985Z stderr F {\"level\":\"info\",\"ts\":\"2024-02-12T17:48:48Z\",\"logger\":\"controllers.ingress\",\"msg\":\"successfully built model\",\"model\":\"{\\\"id\\\":\\\"default/test-ingress\\\",\\\"resources\\\":{\\\"AWS::ElasticLoadBalancingV2::Listener\\\":{\\\"80\\\":{\\\"spec\\\":{\\\"loadBalancerARN\\\":{\\\"$ref\\\":\\\"#/resources/AWS::ElasticLoadBalancingV2::LoadBalancer/LoadBalancer/status/loadBalancerARN\\\"},\\\"port\\\":80,\\\"protocol\\\":\\\"HTTP\\\",\\\"defaultActions\\\":[{\\\"type\\\":\\\"fixed-response\\\",\\\"fixedResponseConfig\\\":{\\\"contentType\\\":\\\"text/plain\\\",\\\"statusCode\\\":\\\"404\\\"}}]}}},\\\"AWS::ElasticLoadBalancingV2::ListenerRule\\\":{\\\"80:1\\\":{\\\"spec\\\":{\\\"listenerARN\\\":{\\\"$ref\\\":\\\"#/resources/AWS::ElasticLoadBalancingV2::Listener/80/status/listenerARN\\\"},\\\"priority\\\":1,\\\"actions\\\":[{\\\"type\\\":\\\"fixed-response\\\",\\\"fixedResponseConfig\\\":{\\\"contentType\\\":\\\"text/plain\\\",\\\"messageBody\\\":\\\"503 error text\\\",\\\"statusCode\\\":\\\"503\\\"}}],\\\"conditions\\\":[{\\\"field\\\":\\\"host-header\\\",\\\"hostHeaderConfig\\\":{\\\"values\\\":[\\\"test.ml-services.internal\\\"]}},{\\\"field\\\":\\\"path-pattern\\\",\\\"pathPatternConfig\\\":{\\\"values\\\":[\\\"/503\\\"]}}]}},\\\"80:2\\\":{\\\"spec\\\":{\\\"listenerARN\\\":{\\\"$ref\\\":\\\"#/resources/AWS::ElasticLoadBalancingV2::Listener/80/status/listenerARN\\\"},\\\"priority\\\":2,\\\"actions\\\":[{\\\"type\\\":\\\"forward\\\",\\\"forwardConfig\\\":{\\\"targetGroups\\\":[{\\\"targetGroupARN\\\":{\\\"$ref\\\":\\\"#/resources/AWS::ElasticLoadBalancingV2::TargetGroup/default/test-ingress-test-service:80/status/targetGroupARN\\\"}}]}}],\\\"conditions\\\":[{\\\"field\\\":\\\"host-header\\\",\\\"hostHeaderConfig\\\":{\\\"values\\\":[\\\"test.ml-services.internal\\\"]}},{\\\"field\\\":\\\"path-pattern\\\",\\\"pathPatternConfig\\\":{\\\"values\\\":[\\\"/*\\\"]}}]}}},\\\"AWS::ElasticLoadBalancingV2::LoadBalancer\\\":{\\\"LoadBalancer\\\":{\\\"spec\\\":{\\\"name\\\":\\\"test-alb\\\",\\\"type\\\":\\\"application\\\",\\\"scheme\\\":\\\"internal\\\",\\\"ipAddressType\\\":\\\"ipv4\\\",\\\"subnetMapping\\\":[{\\\"subnetID\\\":\\\"subnet-YYYYYYYYYYY\\\"},{\\\"subnetID\\\":\\\"subnet-ZZZZZZZZZZZZ\\\"},{\\\"subnetID\\\":\\\"subnet-WWWWWWWWWWW\\\"}],\\\"securityGroups\\\":[\\\"sg-XXXXXXXXXXXXX\\\",\\\"sg-VVVVVVVVVVVV\\\"]}}},\\\"AWS::ElasticLoadBalancingV2::TargetGroup\\\":{\\\"default/test-ingress-test-service:80\\\":{\\\"spec\\\":{\\\"name\\\":\\\"k8s-default-test-89d48a00c1\\\",\\\"targetType\\\":\\\"ip\\\",\\\"port\\\":80,\\\"protocol\\\":\\\"HTTP\\\",\\\"protocolVersion\\\":\\\"HTTP1\\\",\\\"ipAddressType\\\":\\\"ipv4\\\",\\\"healthCheckConfig\\\":{\\\"port\\\":\\\"traffic-port\\\",\\\"protocol\\\":\\\"HTTP\\\",\\\"path\\\":\\\"/healthz\\\",\\\"matcher\\\":{\\\"httpCode\\\":\\\"200\\\"},\\\"intervalSeconds\\\":15,\\\"timeoutSeconds\\\":5,\\\"healthyThresholdCount\\\":2,\\\"unhealthyThresholdCount\\\":2}}}},\\\"K8S::ElasticLoadBalancingV2::TargetGroupBinding\\\":{\\\"default/test-ingress-test-service:80\\\":{\\\"spec\\\":{\\\"template\\\":{\\\"metadata\\\":{\\\"name\\\":\\\"k8s-default-test-89d48a00c1\\\",\\\"namespace\\\":\\\"default\\\",\\\"creationTimestamp\\\":null},\\\"spec\\\":{\\\"targetGroupARN\\\":{\\\"$ref\\\":\\\"#/resources/AWS::ElasticLoadBalancingV2::TargetGroup/default/test-ingress-test-service:80/status/targetGroupARN\\\"},\\\"targetType\\\":\\\"ip\\\",\\\"serviceRef\\\":{\\\"name\\\":\\\"test-service\\\",\\\"port\\\":80},\\\"networking\\\":{\\\"ingress\\\":[{\\\"from\\\":[{\\\"securityGroup\\\":{\\\"groupID\\\":\\\"sg-VVVVVVVVVVVV\\\"}}],\\\"ports\\\":[{\\\"protocol\\\":\\\"TCP\\\",\\\"port\\\":80}]}]},\\\"ipAddressType\\\":\\\"ipv4\\\"}}}}}}}\"}"}
2024-02-12T17:48:49.392+00:00	{"log":"2024-02-12T17:48:49.392599147Z stderr F {\"level\":\"info\",\"ts\":\"2024-02-12T17:48:49Z\",\"logger\":\"controllers.ingress\",\"msg\":\"creating targetGroup\",\"stackID\":\"default/test-ingress\",\"resourceID\":\"default/test-ingress-test-service:80\"}"}
2024-02-12T17:48:49.634+00:00	{"log":"2024-02-12T17:48:49.634382147Z stderr F {\"level\":\"info\",\"ts\":\"2024-02-12T17:48:49Z\",\"logger\":\"controllers.ingress\",\"msg\":\"created targetGroup\",\"stackID\":\"default/test-ingress\",\"resourceID\":\"default/test-ingress-test-service:80\",\"arn\":\"arn:aws:elasticloadbalancing:eu-west-2:XXXXXXXXXXXXX:targetgroup/k8s-default-test-89d48a00c1/98770acde2b3861f\"}"}
2024-02-12T17:48:49.844+00:00	{"log":"2024-02-12T17:48:49.844360575Z stderr F {\"level\":\"info\",\"ts\":\"2024-02-12T17:48:49Z\",\"logger\":\"controllers.ingress\",\"msg\":\"creating loadBalancer\",\"stackID\":\"default/test-ingress\",\"resourceID\":\"LoadBalancer\"}"}
2024-02-12T17:48:50.425+00:00	{"log":"2024-02-12T17:48:50.425580157Z stderr F {\"level\":\"info\",\"ts\":\"2024-02-12T17:48:50Z\",\"logger\":\"controllers.ingress\",\"msg\":\"created loadBalancer\",\"stackID\":\"default/test-ingress\",\"resourceID\":\"LoadBalancer\",\"arn\":\"arn:aws:elasticloadbalancing:eu-west-2:XXXXXXXXXXXXX:loadbalancer/app/test-alb/76218ab85ffda2cd\"}"}
2024-02-12T17:48:50.505+00:00	{"log":"2024-02-12T17:48:50.50520015Z stderr F {\"level\":\"info\",\"ts\":\"2024-02-12T17:48:50Z\",\"logger\":\"controllers.ingress\",\"msg\":\"creating listener\",\"stackID\":\"default/test-ingress\",\"resourceID\":\"80\"}"}
2024-02-12T17:48:50.624+00:00	{"log":"2024-02-12T17:48:50.624623579Z stderr F {\"level\":\"info\",\"ts\":\"2024-02-12T17:48:50Z\",\"logger\":\"controllers.ingress\",\"msg\":\"created listener\",\"stackID\":\"default/test-ingress\",\"resourceID\":\"80\",\"arn\":\"arn:aws:elasticloadbalancing:eu-west-2:XXXXXXXXXXXXX:listener/app/test-alb/76218ab85ffda2cd/a0746bd379532935\"}"}
2024-02-12T17:48:50.704+00:00	{"log":"2024-02-12T17:48:50.704548053Z stderr F {\"level\":\"info\",\"ts\":\"2024-02-12T17:48:50Z\",\"logger\":\"controllers.ingress\",\"msg\":\"creating listener rule\",\"stackID\":\"default/test-ingress\",\"resourceID\":\"80:1\"}"}
2024-02-12T17:48:50.805+00:00	{"log":"2024-02-12T17:48:50.805838969Z stderr F {\"level\":\"info\",\"ts\":\"2024-02-12T17:48:50Z\",\"logger\":\"controllers.ingress\",\"msg\":\"created listener rule\",\"stackID\":\"default/test-ingress\",\"resourceID\":\"80:1\",\"arn\":\"arn:aws:elasticloadbalancing:eu-west-2:XXXXXXXXXXXXX:listener-rule/app/test-alb/76218ab85ffda2cd/a0746bd379532935/f606c4f12beaf821\"}"}
2024-02-12T17:48:50.805+00:00	{"log":"2024-02-12T17:48:50.805887666Z stderr F {\"level\":\"info\",\"ts\":\"2024-02-12T17:48:50Z\",\"logger\":\"controllers.ingress\",\"msg\":\"creating listener rule\",\"stackID\":\"default/test-ingress\",\"resourceID\":\"80:2\"}"}
2024-02-12T17:48:50.899+00:00	{"log":"2024-02-12T17:48:50.899094459Z stderr F {\"level\":\"info\",\"ts\":\"2024-02-12T17:48:50Z\",\"logger\":\"controllers.ingress\",\"msg\":\"created listener rule\",\"stackID\":\"default/test-ingress\",\"resourceID\":\"80:2\",\"arn\":\"arn:aws:elasticloadbalancing:eu-west-2:XXXXXXXXXXXXX:listener-rule/app/test-alb/76218ab85ffda2cd/a0746bd379532935/a2b86ae835a85437\"}"}
2024-02-12T17:48:50.899+00:00	{"log":"2024-02-12T17:48:50.899399981Z stderr F {\"level\":\"info\",\"ts\":\"2024-02-12T17:48:50Z\",\"logger\":\"controllers.ingress\",\"msg\":\"creating targetGroupBinding\",\"stackID\":\"default/test-ingress\",\"resourceID\":\"default/test-ingress-test-service:80\"}"}
2024-02-12T17:48:50.959+00:00	{"log":"2024-02-12T17:48:50.959168255Z stderr F {\"level\":\"info\",\"ts\":\"2024-02-12T17:48:50Z\",\"logger\":\"controllers.ingress\",\"msg\":\"created targetGroupBinding\",\"stackID\":\"default/test-ingress\",\"resourceID\":\"default/test-ingress-test-service:80\",\"targetGroupBinding\":{\"namespace\":\"default\",\"name\":\"k8s-default-test-89d48a00c1\"}}"}
2024-02-12T17:48:50.959+00:00	{"log":"2024-02-12T17:48:50.959240573Z stderr F {\"level\":\"info\",\"ts\":\"2024-02-12T17:48:50Z\",\"logger\":\"controllers.ingress\",\"msg\":\"successfully deployed model\",\"ingressGroup\":\"default/test-ingress\"}"}

@stroebs
Copy link

stroebs commented Feb 14, 2024

Your selector labels on your Service object don't match the ones on your deployment/pod, so your pods aren't actually attached to your Service, hence the load balancer controller cannot find matching pods and no pods are attached to the Target Group.

@janrito
Copy link
Author

janrito commented Feb 14, 2024

Thanks @stroebs

Sorry for the ignorance. This worked in the end:

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: default
  name: test-deployment
  labels:
    app: test
spec:
  replicas: 2
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      containers:
        - name: test-container
          image: test-image
          env:
            - name: PORT
              value: '80'
          resources:
            limits:
              memory: 512Mi
              cpu: '0.25'
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  namespace: default
  name: test-service
  labels:
    app: test
spec:
  type: NodePort
  selector:
    app: test
  ports:
    - name: http
      port: 80
      targetPort: 80
      protocol: TCP

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
  namespace: default
  annotations:
    alb.ingress.kubernetes.io/scheme: internal
    alb.ingress.kubernetes.io/target-type: ip # instance types are not available to Fargate
    alb.ingress.kubernetes.io/load-balancer-name: test-alb
    alb.ingress.kubernetes.io/healthcheck-path: /healthz
    alb.ingress.kubernetes.io/security-groups: sg-XXXXXXXXXX # attach cluster security group to the load balancer
    alb.ingress.kubernetes.io/manage-backend-security-group-rules: 'true' # manage security group rules for the load balancer
    alb.ingress.kubernetes.io/actions.response-503: >
      {"type":"fixed-response","fixedResponseConfig":{"contentType":"text/plain","statusCode":"503","messageBody":"503 error text"}}

    # external-dns specific configuration for creating route53 record-set
    external-dns.alpha.kubernetes.io/hostname: test.services.internal

  labels:
    app: test

spec:
  ingressClassName: alb
  rules:
    - host: test.ml-services.internal
      http:
        paths:
          - path: /503
            pathType: Exact
            backend:
              service:
                name: response-503
                port:
                  name: use-annotation
          - path: /
            pathType: Prefix
            backend:
              service:
                name: test-service
                port:
                  number: 80

@janrito janrito closed this as completed Feb 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants