[EKS] [BAD-DECISION]: EKS Pod Identity agent daemonset mapped to node-port 80

[ChrisMcKee]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
What do you want us to build?

Which service(s) is this request for?
EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
What outcome are you trying to achieve, ultimately, and why is it hard/impossible to do right now? What is the impact of not having this problem solved? The more details you can provide, the better we'll be able to understand and solve the problem.

We tried to install the eks-pod-identity-agent addon so that we could set the auth config to allow both options.
The addon installs as a daemonset with HostNetwork set to true, pod permissions to map to the node, and a default port set to 80.
The instant that the service started to install, all of our Haproxy ingress pods were evicted so that the identity service could map to port 80.

I'd love to know the rationale that went into choosing to map the node-port to what is literally the main http port; and then not to document how to change it to avoid collisions. Through all the documentation that mentions it the only warning is here https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html#pod-id-considerations and it's a note rather than informative. The majority of links go straight to https://docs.aws.amazon.com/eks/latest/userguide/pod-id-agent-setup.html which doesnt mention it at all.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: eks-pod-identity-agent
  namespace: kube-system
spec:
  template:    
    spec:
      containers:
        - name: eks-pod-identity-agent
          image: >-
            602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/eks-pod-identity-agent:0.1.6
          command:
            - /go-runner
            - /eks-pod-identity-agent
            - server
          args:
            - '--port'
            - '80'
            - '--cluster-name'
            - prod-eks-cluster
            - '--probe-port'
            - '2703'
          ports:
            - name: proxy
              containerPort: 80
              protocol: TCP
            - name: probes-port
              containerPort: 2703
              protocol: TCP
          securityContext:
            capabilities:
              add:
                - CAP_NET_BIND_SERVICE
      hostNetwork: true

Are you currently working around this issue?
How are you currently solving this problem?

Uninstalled the Addon

[vpineda1996]

The EKS Pod Identity Agent only binds to the address on link-local interface created by the initial setup. Specifically, the call to bind binds to port 80 on the following IP addresses:

• fd00:ec2::23
• 169.254.170.23

You can get around this limitation by specifying the address of the interface you are trying to bind to port 80. eg if you are doing a bind 0.0.0.0:80, it will fail but if you know the IP of your instance (eg 10.0.163.96) then you can do a direct bind 10.0.163.96:80.

To know the primary IP that your instance has you can run ip addr and find the interface that has the primary IP (normally named ens6)

5: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 0a:24:fe:7c:d3:99 brd ff:ff:ff:ff:ff:ff
    altname enp0s6
    altname ens6
    inet 10.0.163.96/19 brd 10.0.191.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::824:feff:fe7c:d399/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever

Alternatively you can use EC2's DescribeInstances API to find the primary network interface IP.

[ChrisMcKee]

@vpineda1996 it hijacks port-80 on the node; If I deploy haproxy ingress daemonset on hostnetwork and use a random port e.g. 30680 it will work fine.
If I have haproxy installed already attached to port 80; installing the AWS Identity addon will kick the haproxy pods causing an outage.
It sounds like you're saying it doesn't use the eks-node host-network / 80? but it is.

[ryangraham]

This just took down all my nginx ingress the same way. Had to remove the addon and delete the daemonset to fix.

[dims]

fyi code is here now - https://github.com/aws/eks-pod-identity-agent - can we please move this to an issue there? 🙏🏾

[ChrisMcKee]

@dims @ryangraham Moved to aws/eks-pod-identity-agent#10

[dims]

thanks @ChrisMcKee