| title | linkTitle | weight |
|---|---|---|
FAQs |
FAQs |
90 |
See [Configuring provisioners]({{< ref "./concepts/#configuring-provisioners" >}}) for information on how Karpenter provisions and manages nodes.
AWS is the first cloud provider supported by Karpenter, although it is designed to be used with other cloud providers as well. See [Cloud provider]({{< ref "./concepts/#cloud-provider" >}}) for details.
Yes, but there is no documentation yet for it. Start with Karpenter's GitHub cloudprovider documentation to see how the AWS provider is built, but there are other sections of the code that will require changes too.
By default, Karpenter uses Amazon Linux 2 images.
Karpenter allows you to create your own AWS AMIs using custom launch templates. See [Launch Templates and Custom Images]({{< ref "./aws/launch-templates/" >}}) for details.
Yes. Build and prepare custom arm images as described in [Launch Templates and Custom Images]({{< ref "./aws/launch-templates/" >}}). Specify the desired architecture when you deploy workloads.
All of the required RBAC rules can be found in the helm chart template. See clusterrolebinding.yaml, clusterrole.yaml, rolebinding.yaml, and role.yaml files for details.
Yes, as long as the controller has network and IAM/RBAC access to the Kubernetes API and your provider API.
Karpenter is tested with Kubernetes v1.19 and later.
Karpenter documents integration with a fresh install of the latest AWS Elastic Kubernetes Service (EKS). Existing EKS distributions can be used, but this use case has not yet been documented. Other Kubernetes distributions (KOPs, etc.) can be used, but setting up cloud provider permissions for those distributions has not been documented.
Provisioners are designed to work alongside static capacity management solutions like EKS Managed Node Groups and EC2 Auto Scaling Groups. You can manage all capacity using provisioners, use a mixed model with dynamic and statically managed capacity, or use a fully static approach. We expect most users will use a mixed approach in the near term and provisioner-managed in the long term.
- Kubernetes Cluster Autoscaler: Karpenter can work alongside cluster autoscaler. See [Kubernetes cluster autoscaler]({{< ref "./concepts/#kubernetes-cluster-autoscaler" >}}) for details.
- Kubernetes Scheduler: Karpenter focuses on scheduling pods that the Kubernetes scheduler has marked as unschedulable. See [Scheduling]({{< ref "./concepts/#scheduling" >}}) for details on how Karpenter interacts with the Kubernetes scheduler.
See [Provisioner API]({{< ref "./provisioner" >}}) for provisioner examples and descriptions of features.
Yes, provisioners can identify multiple teams based on labels. See [Provisioner API]({{< ref "./provisioner" >}}) for details.
By default, pods will use the rules defined by a provisioner named default.
This is analogous to the default scheduler.
To select an alternative provisioner, use the node selector karpenter.sh/provisioner-name: alternative-provisioner.
You must either define a default provisioner or explicitly specify karpenter.sh/provisioner-name node selector.
Yes, the setting is provider-specific. See examples in [Accelerators, GPU]({{< ref "./aws/provisioning/#accelerators-gpu" >}}) Karpenter documentation.
Yes, see [Example Provisioner Resource]({{< ref "./provisioner/#example-provisioner-resource" >}}) for an example.
- Attribute-based requests are currently not possible.
- You can select instances with special hardware, such as gpu.
Karpenter batches pending pods and then binpacks them based on CPU, memory, and GPUs required, taking into account node overhead, VPC CNI resources required, and daemon sets that will be packed when bringing up a new node.
By default Karpenter uses all available instance types, but it can be constrained in the provisioner spec with the instance-type well-known label in the requirements section.
After the pods are binpacked on the most efficient instance type (i.e. the smallest instance type that can fit the pod batch), Karpenter takes 19 other instance types that are larger than the most efficient packing, and passes all 20 instance type options to an API called Amazon EC2 Fleet.
The EC2 fleet API attempts to provision the instance type based on a user-defined allocation strategy.
If you are using the on-demand capacity type, then Karpenter uses the lowest-price allocation strategy.
So fleet will provision the lowest price instance type it can get from the 20 Karpenter passed it.
If the instance type is unavailable for some reason, then fleet will move on to the next cheapest instance type.
If you are using the spot capacity type, Karpenter uses the capacity-optimized-prioritized allocation strategy which tells fleet to find the instance type that EC2 has the most capacity of which will decrease the probability of a spot interruption happening in the near term.
See Choose the appropriate allocation strategy for information on fleet optimization.
Karpenter will fallback to on-demand, if your provisioner specifies both spot and on-demand.
More specifically, Karpenter maintains a concept of "offerings" for each instance type, which is a combination of zone and capacity type (equivalent in the AWS cloud provider to an EC2 purchase option). Spot offerings are prioritized, if they're available. Whenever the Fleet API returns an insufficient capacity error for Spot instances, those particular offerings are temporarily removed from consideration (across the entire provisioner) so that Karpenter can make forward progress through fallback. The retry will happen immediately within milliseconds.
See [Application developer]({{< ref "./concepts/#application-developer" >}}) for descriptions of how Karpenter matches nodes with pod requests.
- Set the ALB target type to IP mode for the pods. Use IP targeting if you want the pods to receive equal weight. Instance balancing could greatly skew the traffic being sent to a node without also managing host spread of the workload.
- Set readiness gate on the namespace.
The default is round robin at the node level.
For Karpenter, not all nodes are equal.
For example, each node will have different performance characteristics and a different number of pods running on it.
A
t3.smallwith three instances should not receive the same amount of traffic as am5.4xlargewith dozens of pods. If you don't specify a spread at the workload level, or limit what instances should be picked, you could get the same amount of traffic sent to thet3andm5.
Yes. See [Persistent Volume Topology]({{< ref "./tasks/scheduling#persistent-volume-topology" >}}) for details.
Not yet.
See [Deprovisioning nodes]({{< ref "./tasks/deprovisioning" >}}) for information on how Karpenter deprovisions nodes.
Karpenter is a controller that runs in your cluster, but it is not tied to a specific Kubernetes version, as the Cluster Autoscaler is. Use your existing upgrade mechanisms to upgrade your core add-ons in Kubernetes and keep Karpenter up to date on bug fixes and new features.
Karpenter requires proper permissions in the KarpenterNode IAM Role and the KarpenterController IAM Role.
To upgrade Karpenter to version $VERSION, make sure that the KarpenterNode IAM Role and the KarpenterController IAM Role have the right permission described in https://karpenter.sh/$VERSION/getting-started/cloudformation.yaml.
Next, locate KarpenterController IAM Role ARN (i.e., ARN of the resource created in Create the KarpenterController IAM Role) and the cluster endpoint, and pass them to the helm upgrade command
helm upgrade --install --namespace karpenter --create-namespace \
karpenter karpenter/karpenter \
--version ${$VERSION} \
--set clusterName=${CLUSTER_NAME} \
--set clusterEndpoint=${CLUSTER_ENDPOINT} \
--set serviceAccount.annotations.eks\.amazonaws\.com/role-arn=${KARPENTER_IAM_ROLE_ARN}
--set aws.defaultInstanceProfile=KarpenterNodeInstanceProfile-${CLUSTER_NAME} \
--wait # for the defaulting webhook to install before creating a Provisioner