New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EKS IP-addresses limits #1366

Closed
okgolove opened this Issue Oct 31, 2018 · 10 comments

Comments

Projects
None yet
3 participants
@okgolove
Copy link

okgolove commented Oct 31, 2018

Hello. EKS uses AWS CNI to assign private IP to every pod https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI
So, if you haven't free IPs your pod won't be schedulded.

Can somehow autoscaler implements autoscaling based on IP limits?

@aleksandra-malinowska

This comment has been minimized.

Copy link
Contributor

aleksandra-malinowska commented Oct 31, 2018

It works differently for every cloud provider (e.g. on GCE, each nodes is assigned a range of IPs for pods). If I understand correctly, in this case IPs would be a cluster-level resource: it doesn't limit the number of nodes, and no matter how many we add, pods may not be able to run. Currently there's no support for such resources at all. It can probably be implemented by injecting a new pod list processor, which would remove pods that won't be able to run anyway from scale-up calculations.

@johanneswuerbach

This comment has been minimized.

Copy link
Contributor

johanneswuerbach commented Jan 7, 2019

I'm not entirely sure cluster-autoscaler needs to do anything here, but your instances should be actually configured to only allow max IP address pods https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/#options.

A change for that was recently implemented in kops kubernetes/kops#6058, but I don't know whether this is done in EKS by default.

The max pods limit should also be recognised by the CA, but I'm not entirely sure whether it is, maybe @aleksandra-malinowska knows more?

@okgolove

This comment has been minimized.

Copy link
Author

okgolove commented Jan 8, 2019

@johanneswuerbach hello, thank you for the feedback.
I have specified that option (--max-pods). It's exactly that thing I meant. As far I exhaust IP limit Kubernetes can't schedule any new pods. I meant it will be great if CA can handle these situations.

@aleksandra-malinowska

This comment has been minimized.

Copy link
Contributor

aleksandra-malinowska commented Jan 8, 2019

I believe scheduler's predicates checks max pods per node limit. If it's not the case, it's probably a bug.

I have specified that option (--max-pods). It's exactly that thing I meant.

Can you verify if your nodes indeed have this set? kubectl get node <node-name> -o yaml

@okgolove

This comment has been minimized.

Copy link
Author

okgolove commented Jan 8, 2019

@aleksandra-malinowska

status:
  addresses:
  - address: 10.0.1.217
    type: InternalIP
  - address: ip-10-0-1-217.eu-west-1.compute.internal
    type: InternalDNS
  - address: ip-10-0-1-217.eu-west-1.compute.internal
    type: Hostname
  allocatable:
    cpu: "2"
    ephemeral-storage: "96625420948"
    hugepages-2Mi: "0"
    memory: 3937632Ki
    pods: "17"
  capacity:
    cpu: "2"
    ephemeral-storage: 104845292Ki
    hugepages-2Mi: "0"
    memory: 4040032Ki
    pods: "17"

Also I have next option in kubelet-config.json:
"maxPods": 17

@aleksandra-malinowska

This comment has been minimized.

Copy link
Contributor

aleksandra-malinowska commented Jan 8, 2019

Does CA ignore this (i.e. scale up assuming more than 17 pods will fit)? If so, any repro you have would be useful (sample pods etc.) Scheduler code looks fairly straightforwad, not sure what may be wrong here:/

@okgolove

This comment has been minimized.

Copy link
Author

okgolove commented Jan 9, 2019

Hm. It's strange, but when I tried to deploy 100 pods to my EKS cluster CNI wasn't able to assign IP to pods and those pods weren't in status "Pending" (I don't remember what status was).
But now when I deploy 100 pods I have a lot of pods in "Pending" status and CA scales my nodes correctly:

Warning FailedScheduling 8s (x7 over 39s) default-scheduler 0/3 nodes are available: 3 Insufficient pods.

nginx-bucket-6f8b645d58-vg92h   0/1     Pending   0          4m
nginx-bucket-6f8b645d58-vpkkk   0/1     Pending   0          4m
nginx-bucket-6f8b645d58-w5px7   0/1     Pending   0          4m
nginx-bucket-6f8b645d58-w6fv4   0/1     Pending   0          4m
nginx-bucket-6f8b645d58-w8dxd   0/1     Pending   0          4m
nginx-bucket-6f8b645d58-wbn5v   0/1     Pending   0          4m
nginx-bucket-6f8b645d58-wkc26   0/1     Pending   0          4m
nginx-bucket-6f8b645d58-wq926   0/1     Pending   0          4m
nginx-bucket-6f8b645d58-ws7bz   0/1     Pending   0          4m
nginx-bucket-6f8b645d58-x9k6d   0/1     Pending   0          4m
nginx-bucket-6f8b645d58-xcgnf   0/1     Pending   0          4m
nginx-bucket-6f8b645d58-xxp4b   0/1     Pending   0          4m
nginx-bucket-6f8b645d58-zfcd7   0/1     Pending   0          4m
nginx-bucket-6f8b645d58-zlpr4   0/1     Pending   0          4m
nginx-bucket-6f8b645d58-zmsz6   0/1     Pending   0          4m
nginx-bucket-6f8b645d58-znjjt   0/1     Pending   0          4m

It seems CA works as expected.

@okgolove

This comment has been minimized.

Copy link
Author

okgolove commented Jan 9, 2019

Oh, I've reproduced it!
Pod has status

0/1     Running
Warning  FailedCreatePodSandBox  58s (x12 over 70s)  kubelet, ip-10-0-2-196.eu-west-1.compute.internal  Failed create pod sandbox: rpc error: code = Unknown
 desc = NetworkPlugin cni failed to set up pod "nginx-develop-7845f449bc-lnlqv_nginx-develop" network: add cmd: failed to assign an IP a
ddress to container
  Normal   SandboxChanged          58s (x11 over 68s)  kubelet, ip-10-0-2-196.eu-west-1.compute.internal  Pod sandbox changed, it will be killed and re-create
d.```
@aleksandra-malinowska

This comment has been minimized.

Copy link
Contributor

aleksandra-malinowska commented Jan 9, 2019

CA only makes sure there are enough nodes to schedule pods on. In this case, it seems pod was scheduled, but kubelet wasn't actually able to run it. I'd look for the scheduling constraints that were supposed to prevent this and ensure they're in place. Perhaps 17 pods per node is too many in this case, or there's some global limit on number of pods?

@okgolove

This comment has been minimized.

Copy link
Author

okgolove commented Jan 9, 2019

Thank you for your help.
As I think it is not the CA problem.
This issue may be closed, if you think it should.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment