Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
eb05b1e
commit 205c4db
Showing
75 changed files
with
7,936 additions
and
86 deletions.
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| --- | ||
| title: "AWS" | ||
| linkTitle: "AWS" | ||
| weight: 70 | ||
| description: > | ||
| Use AWS cloud provider features with Karpenter | ||
| --- | ||
|
|
||
| Check out the [Karpenter EKS Best Practices](https://aws.github.io/aws-eks-best-practices/karpenter/) guide. | ||
|
|
||
| Check out the [EC2 Spot Workshop](https://ec2spotworkshops.com/karpenter.html) for Karpenter and the [EKS Karpenter Workshop](https://www.eksworkshop.com/beginner/085_scaling_karpenter/set_up_the_provisioner/). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,241 @@ | ||
| --- | ||
| title: "Launch Templates and Custom Images" | ||
| linkTitle: "Launch Templates" | ||
| weight: 80 | ||
| description: > | ||
| Create custom launch templates for Karpenter | ||
| --- | ||
|
|
||
| By default, Karpenter generates launch templates with the following features: | ||
| - [EKS Optimized AMI](https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html) for nodes. | ||
| - Encrypted EBS root volumes with the default (AWS managed) KMS key for nodes. | ||
|
|
||
| If these features are not sufficient for your use case (customizing node image, customizing EBS KMS key, etc), you need a custom launch template. | ||
|
|
||
| Karpenter supports using custom launch templates. | ||
|
|
||
| Note: When using a custom launch template, **you are taking responsibility** for maintaining the launch template, including updating which AMI is used (i.e., for security updates). In the default configuration, Karpenter will use the latest version of the EKS optimized AMI, which is maintained by AWS. Without a custom launch template, Karpenter will create its own. If these launch templates aren't used for sixty seconds, Karpenter will clean them up. | ||
|
|
||
|
|
||
| ## Introduction | ||
|
|
||
| Karpenter follows existing AWS patterns for customizing the base image of | ||
| instances. More specifically, Karpenter uses [EC2 launch templates](https://docs.aws.amazon.com/autoscaling/ec2/userguide/LaunchTemplates.html). Launch | ||
| templates may specify many values. The pivotal value is the base image (AMI). | ||
| Launch templates further specify many different parameters related to networking, authorization, instance type, and more. | ||
|
|
||
| Launch Templates and AMIs are unique to AWS regions, similar to EKS clusters. IAM resources are global. | ||
|
|
||
| **Karpenter only implements a subset of launch template fields, and some fields should not be set.** | ||
|
|
||
| This guide describes requirements for using launch templates with Karpenter, and later an example procedure. | ||
|
|
||
| ## Launch Template Requirements | ||
|
|
||
| The Launch Template resource includes a large number of fields. AWS accepts launch templates with any subset of these fields defined. | ||
|
|
||
| Certain fields are obviously critical, such as AMI and User Data. Some fields are useful for particular workloads, such as storage and IAM Instance Profile. | ||
|
|
||
| Finally, **the majority of Launch Template fields should not be set** (or will have no effect), such as network interfaces and instance type. | ||
|
|
||
| ## Important Fields | ||
|
|
||
| When creating a custom launch template, the AMI and User Data are the defining characteristics. Instance Profile (IAM Role) and Security Group (firewall rules) are also important for Karpenter. | ||
|
|
||
| ### AMI | ||
|
|
||
| AMI (Amazon Machine Image), is the base image/VM for a launch template. | ||
|
|
||
| [Review the instructions for importing a VM to AWS.](https://docs.aws.amazon.com/vm-import/latest/userguide/vmimport-image-import.html) Note the AMI id generated by this process, such as, | ||
| `ami-074cce78125f09d61`. | ||
|
|
||
| ### User Data - Autoconfigure | ||
|
|
||
| Importantly, the AMI must support automatically connecting to a cluster based | ||
| on "user data", or a base64 encoded string passed to the instance at startup. | ||
| The syntax and purpose of the user data varies between images. The Karpenter | ||
| default OS, Amazon Linux 2 (AL2), accepts shell scripts (bash commands). | ||
|
|
||
| [AWS calls data passed to an instance at launch time "user | ||
| data".](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html#user-data-shell-scripts) | ||
|
|
||
| In the default configuration, Karpenter uses an EKS optimized version of AL2 and passes the hostname of the Kubernetes API server, and a certificate. The EKS Optimized AMI includes a `bootstrap.sh` script which connects the instance to the cluster, based on the passed data. | ||
|
|
||
| Alternatively, you may reference AWS's [`bootstrap.sh` | ||
| file](https://github.com/awslabs/amazon-eks-ami/blob/master/files/bootstrap.sh) | ||
| when building a custom base image. | ||
|
|
||
| ``` | ||
| #!/bin/bash | ||
| /etc/eks/bootstrap.sh <my-cluster-name> \ | ||
| --kubelet-extra-args <'--max-pods=40'> \ | ||
| --b64-cluster-ca <certificateAuthority> \ | ||
| --apiserver-endpoint <endpoint> \ | ||
| --dns-cluster-ip <dnsClusterIP> \ | ||
| --use-max-pods false | ||
| ``` | ||
|
|
||
| Note, you must populate this command with live values. Karpenter will | ||
| not change the user data in the launch template. | ||
|
|
||
| Encode using yaml function `!Base64` yaml function or `cat userdata.sh | base64 > userdata-encoded.txt` shell command. | ||
|
|
||
| **Bootstrap Script Parameters** | ||
|
|
||
| The sample bootstrap script requires information to join the cluster. | ||
|
|
||
| These values may be found using: | ||
| ``` | ||
| aws eks describe-cluster --name MyKarpenterCluster | ||
| ``` | ||
|
|
||
| **Kubelet Arguments** | ||
|
|
||
| Specifying max-pods can break Karpenter's binpacking logic (it has no way to know what this setting is). If Karpenter attempts to pack more than this number of pods, the instance may be oversized, and additional pods will reschedule. | ||
|
|
||
| ## Situational Fields | ||
|
|
||
| Configure these values in response to a particular use case, such as nodes interacting with another AWS service, or using EBS storage on the node. | ||
|
|
||
| ### Instance Profile - IAM | ||
|
|
||
| The launch template must include an "instance profile" -- an IAM role. | ||
|
|
||
| The instance profile must include *at least* the permissions of the default Karpenter node instance profile. See the default role, `KarpenterNodeRole`, in the full example below for more information. | ||
|
|
||
| See also, [the managed policy "AmazonEKSWorkerNodePolicy"](https://docs.aws.amazon.com/eks/latest/userguide/security-iam-awsmanpol.html#security-iam-awsmanpol-AmazonEKSWorkerNodePolicy) which includes permission to describe clusters and subnets. | ||
|
|
||
| ### Storage | ||
|
|
||
| Karpenter expects nothing of node storage. Configure as needed for your base | ||
| image. | ||
|
|
||
| ### Security Groups - Firewall | ||
|
|
||
| The launch template may include a security group (i.e., instance firewall rules) and the security group must be associated with the virtual private cloud (VPC) of the EKS cluster. If none is specified, the default security group of the cluster VPC is used. | ||
|
|
||
| The security group must permit communication with EKS control plane. Outbound access should be permitted for at least: HTTPS on port 443, DNS (UDP and TCP) on port 53, and your subnet's network access control list (network ACL). | ||
|
|
||
| ## Fields with Undefined Behavior | ||
|
|
||
| Resources referenced by these fields are controlled by EKS/Karpenter, and not the launch template. | ||
|
|
||
| ### Instance Type | ||
|
|
||
| The instance type should not be specified in the launch template. Karpenter | ||
| will determine the launch template at run time. | ||
|
|
||
| ### Network Interfaces | ||
|
|
||
| The [AWS CNI](https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html) will configure the network interfaces. Do not configure network instances in the launch template. | ||
|
|
||
| ## Creating the Launch Template | ||
|
|
||
| Launch Templates may be created via the web console, the AWS CLI, or | ||
| CloudFormation. | ||
|
|
||
| ### CloudFormation | ||
|
|
||
| The procedure, in summary, is to: | ||
| 1. [Create an AMI as described in the EC2 documentation.](https://docs.aws.amazon.com/vm-import/latest/userguide/vmimport-image-import.html) | ||
| 2. Write a EC2 Launch Template specification including the AMI. | ||
| 3. Push the specification to AWS with CloudFormation. | ||
| 4. Update the Provisioner CRD to specify the new Launch Template. | ||
|
|
||
| An example yaml cloudformation definition of a launch template for Karpenter is | ||
| provided below. | ||
|
|
||
| CloudFormation yaml is suited for the moderately high configuration density of | ||
| launch templates, and creating the unusual InstanceProfile resource. | ||
|
|
||
| You must manually replace these values in the template: | ||
| - SecurityGroupID | ||
| - list all security groups with `aws ec2 describe-security-groups` | ||
| - Parameters in UserData | ||
| - AMI | ||
|
|
||
| ```yaml | ||
| AWSTemplateFormatVersion: '2010-09-09' | ||
| Resources: | ||
| # create InstanceProfile wrapper on NodeRole | ||
| KarpenterNodeInstanceProfile: | ||
| Type: "AWS::IAM::InstanceProfile" | ||
| Properties: | ||
| InstanceProfileName: "KarpenterNodeInstanceProfile" | ||
| Path: "/" | ||
| Roles: | ||
| - Ref: "KarpenterNodeRole" | ||
| # create role with basic permissions for EKS node | ||
| KarpenterNodeRole: | ||
| Type: "AWS::IAM::Role" | ||
| Properties: | ||
| RoleName: "KarpenterNodeRole" | ||
| Path: / | ||
| AssumeRolePolicyDocument: | ||
| Version: "2012-10-17" | ||
| Statement: | ||
| - Effect: Allow | ||
| Principal: | ||
| Service: | ||
| !Sub "ec2.${AWS::URLSuffix}" | ||
| Action: | ||
| - "sts:AssumeRole" | ||
| ManagedPolicyArns: | ||
| - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonEKSWorkerNodePolicy" | ||
| - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonEKS_CNI_Policy" | ||
| - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly" | ||
| - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonSSMManagedInstanceCore" | ||
| MyLaunchTemplate: | ||
| Type: AWS::EC2::LaunchTemplate | ||
| Properties: | ||
| LaunchTemplateData: | ||
| IamInstanceProfile: | ||
| # Get ARN of InstanceProfile defined above | ||
| Arn: !GetAtt | ||
| - KarpenterNodeInstanceProfile | ||
| - Arn | ||
| ImageId: ami-074cce78125f09d61 | ||
| # UserData is Base64 Encoded | ||
| UserData: !Base64 > | ||
| #!/bin/bash | ||
| /etc/eks/bootstrap.sh 'MyClusterName' \ | ||
| --kubelet-extra-args '--node-labels=node.k8s.aws/capacity-type=spot' \ | ||
| --b64-cluster-ca 'LS0t....0tCg==' \ | ||
| --apiserver-endpoint 'https://B0385BE29EA792E811CB5866D23C856E.gr7.us-east-2.eks.amazonaws.com' | ||
| BlockDeviceMappings: | ||
| - Ebs: | ||
| VolumeSize: 80 | ||
| VolumeType: gp3 | ||
| DeviceName: /dev/xvda | ||
| # The SecurityGroup must be associated with the cluster VPC | ||
| SecurityGroupIds: | ||
| - sg-a69adfdb | ||
| LaunchTemplateName: KarpenterCustomLaunchTemplate | ||
| ``` | ||
|
|
||
| Create the Launch Template by uploading the CloudFormation yaml file. The | ||
| sample yaml creates an IAM Object (InstanceProfile), so `--capabilities | ||
| CAPABILITY_NAMED_IAM` must be indicated. | ||
|
|
||
| ``` | ||
| aws cloudformation create-stack \ | ||
| --stack-name KarpenterLaunchTemplateStack \ | ||
| --template-body file://$(pwd)/lt-cfn-demo.yaml \ | ||
| --capabilities CAPABILITY_NAMED_IAM | ||
| ``` | ||
|
|
||
| ### Define LaunchTemplate for Provisioner | ||
|
|
||
| The LaunchTemplate is ready to be used. Specify it by name in the [Provisioner | ||
| CRD](../../provisioner/). Karpenter will use this template when creating new instances. | ||
| The following is an example of a provisioner using the new template. Please replace the `CLUSTER_NAME` with the correct value. | ||
|
|
||
| ```yaml | ||
| apiVersion: karpenter.sh/v1alpha5 | ||
| kind: Provisioner | ||
| spec: | ||
| provider: | ||
| launchTemplate: KarpenterCustomLaunchTemplate | ||
| subnetSelector: | ||
| karpenter.sh/discovery: CLUSTER_NAME | ||
| ``` |
Oops, something went wrong.