Skip to content

create eni-max-pods.txt file in generated LT#1403

Closed
bwagner5 wants to merge 1 commit intoaws:mainfrom
bwagner5:max-eni
Closed

create eni-max-pods.txt file in generated LT#1403
bwagner5 wants to merge 1 commit intoaws:mainfrom
bwagner5:max-eni

Conversation

@bwagner5
Copy link
Copy Markdown
Contributor

1. Issue, if available:
N/A

2. Description of changes:

  • This change uses Karpenterr's knowledge of max-pods to generate the eni-max-pods.txt file which the EKS bootstrap script reads when starting the k8s components. There have been several issues when new instance types are released, the eks-optimized AMIs do not have the new instance types listed in the eni-max-pods.txt file. Karpenter can pick up new instance types very early which makes this a problem. Further, the Ubuntu eks-optimized AMIs are further behind than the AL2 AMIs.
  • NOTE: there is a 16KB limit to user-data. The current user-data, with this change, is 5.2KB so there is quite a bit of headroom.

3. How was this change tested?

  • Tested on a manual cluster with AL2 & Ubuntu on instance types that previously were not listed (r6i.32xlarge on Ubuntu and c6a.4x.large.

4. Does this change impact docs?

  • Yes, PR includes docs updates
  • Yes, issue opened: link to issue
  • No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@netlify
Copy link
Copy Markdown

netlify bot commented Feb 22, 2022

✔️ Deploy Preview for karpenter-docs-prod canceled.

🔨 Explore the source changes: 7c55bd0

🔍 Inspect the deploy log: https://app.netlify.com/sites/karpenter-docs-prod/deploys/621568a62d0220000762943d

@bwagner5 bwagner5 force-pushed the max-eni branch 2 times, most recently from d245d87 to 45c33d3 Compare February 22, 2022 22:27
%s
/etc/eks/bootstrap.sh '%s' %s \
--apiserver-endpoint '%s'`,
p.ReplaceENIMaxPodsFile(ctx, constraints),
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I worry that we're gonna run headlong into userdata limits with this approach.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could double the size of our user-data today and still be within limits.

We could also only populate the instance type options that we are going to launch. I'm doing all of them here for cache efficiency for the LT. Since we clean the LTs up now, it might be less of a big deal, other than the small bit of latency creating the LT.

@stevehipwell
Copy link
Copy Markdown
Contributor

@bwagner5 have you checked out the various issues with the current AL2 max pods implementation? My specific issue is awslabs/amazon-eks-ami#782 which might shed a bit more light on this? It also covers the reserved values which might also be something Karpenter should look at? I'm not sure what BottleRocket uses for these values but it's probably also worth considering?

@bwagner5 bwagner5 closed this Mar 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants