Remove per-node pod limit

[deliahu]

Description

Pod limits per node have been reached (17 for t3.medium, 11 for t3.small). The limits exist due to IP address allocation from the AWS cni plugin, see here.

This should be resolved in the new version of the AWS CNI (aws-cni roadmap)

Replacing the AWS CNI

If this becomes an urgent issue, it may also be fixed by using a different CNI

• [EKS] [CNI]: Optional Default CNI Plugin Installation aws/containers-roadmap#71
• Opt-Out AWS VPC CNI (And Any Other EKS "Magics") awslabs/amazon-eks-ami#117
• [EKS] Increased pod density on smaller instance types aws/containers-roadmap#138
• Ability to choose another CNI aws/amazon-vpc-cni-k8s#214
• Pods stuck in ContainerCreating due to CNI Failing to Assing IP to Container Until aws-node is deleted aws/amazon-vpc-cni-k8s#59

Alternative CNI plugins

• weave-net
  • Running Weave Net on EKS
  • By far the simplest installation process
  • metrics-server and istio seem to not work out of the box
    • Metric server with weave net can't collect monitoring data from the node where his pod is placed kubernetes-sigs/metrics-server#166
    • Error from server (ServiceUnavailable): the server is currently unable to handle the request kubernetes-sigs/metrics-server#188
    • couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request kubernetes-sigs/metrics-server#157
    • Metrics server issue with hostname resolution of kubelet and apiserver unable to communicate with metric-server clusterIP kubernetes-sigs/metrics-server#131
    • Metrics server api not getting registered kubernetes-sigs/metrics-server#45
• calico
  • Running Calico on EKS
  • GKE uses it?
  • EKS supports running Calico alongside AWS CNI
• flannel
  • Running Flannel on EKS
• istio
  • is not be a replacement, but in addition to the aws cni plugin; see here or here and search "chained"
• cni-genie (to install other CNIs)
  • install it on the master node and configure the default to a different CNI before creating the ec2 worker nodes
  • Example project of using cni-genie to install weave on EKS

May need to run kubectl delete --namespace kube-system daemonset/aws-node before adding worker nodes to uninstall the AWS cni. May also need to start kubelet without --network-plugin=cni - otherwise kubelet may refuse to start because the configured CNI plugin cannot be brought up (aws-node container is not running). Another way to remove the AWS cni is to build a custom AMI with the desired CNI plugin prefixed with 00 instead of the standard 10 so that it circumvents the loading of the AWS VPC CNI plugin. source

Things that will help:

• Increase default node size
• Increase default CPU request (to reach CPU limits before pod limits)
• Replace argo with custom DAG management