Skip to content

Commit

Permalink
docs: Add updated information for threat model (#4966)
Browse files Browse the repository at this point in the history
  • Loading branch information
@jonathan-innis
jonathan-innis committed Oct 30, 2023
1 parent 43e51ea commit e2af747
Showing 1 changed file with 32 additions and 14 deletions.
46 changes: 32 additions & 14 deletions website/content/en/preview/reference/threat-model.md
View file
Expand Up @@ -21,7 +21,7 @@ When running in AWS, Karpenter is typically installed onto EC2 instances that ru

### Cluster Operator

The Cluster Operator has full control to install and configure Karpenter including all [`NodePools`]{{<ref "../concepts/nodepools" >}} and [`EC2NodeClasses`]{{<ref "../concepts/nodeclasses" >}}. The Cluster Operator has privileges to manage the cloud identities and permissions for Nodes, and the cloud identity and permissions for Karpenter.
The Cluster Operator has full control to install and configure Karpenter including all [`NodePools`]({{<ref "../concepts/nodepools" >}}) and [`EC2NodeClasses`]({{<ref "../concepts/nodeclasses" >}}). The Cluster Operator has privileges to manage the cloud identities and permissions for Nodes, and the cloud identity and permissions for Karpenter.

### Cluster Developer

Expand All @@ -39,15 +39,15 @@ Karpenter has permissions to create and manage cloud instances. Karpenter has Ku

## Assumptions

| Category | Assumption | Comment |
| --- | --- | --- |
| Generic | The Karpenter pod is operated on a node in the cluster, and uses a Service Account for authentication to the Kubernetes API | Cluster Operators may want to isolate the node running the Karpenter pod to a system-pool of nodes to mitigate the possibility of container breakout with Karpenter’s permissions. |
| Generic | Cluster Developer does not have any Kubernetes permissions to manage Karpenter running in the cluster (The deployment, pods, clusterrole, etc) | |
| Generic | Restrictions on the fields of pods a Cluster Developer can create are out of scope. | Cluster Operators can use policy frameworks to enforce restrictions on Pod capabilities |
| Generic | No sensitive data is included in non-Secret resources in the Kubernetes API. The Karpenter controller has the ability to list all pods, nodes, deployments, and many other pod-controller and storage resource types. | Karpenter does not have permission to list/watch cluster-wide ConfigMaps or Secrets |
| Generic | Karpenter has permissions to create, modify, and delete nodes from the cluster, and evict any pod. | Cluster Operators running applications with varying security profiles in the same cluster may want to configure dedicated nodes and scheduling rules for Karpenter to mitigate potential container escapes from other containers |
| AWS-Specific | The Karpenter IAM policy is encoded in the GitHub repo. Any additional permissions possibly granted to that role by the administrator are out of scope | |
| AWS-Specific | The Karpenter pod uses IRSA for AWS credentials | Setup of IRSA is out of scope for this document |
| Category | Assumption | Comment |
|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Generic | The Karpenter pod is operated on a node in the cluster, and uses a Service Account for authentication to the Kubernetes API | Cluster Operators may want to isolate the node running the Karpenter pod to a system-pool of nodes to mitigate the possibility of container breakout with Karpenter’s permissions. |
| Generic | Cluster Developer does not have any Kubernetes permissions to manage Karpenter running in the cluster (The deployment, pods, clusterrole, etc) | |
| Generic | Restrictions on the fields of pods a Cluster Developer can create are out of scope. | Cluster Operators can use policy frameworks to enforce restrictions on Pod capabilities |
| Generic | No sensitive data is included in non-Secret resources in the Kubernetes API. The Karpenter controller has the ability to list all pods, nodes, deployments, and many other pod-controller and storage resource types. | Karpenter does not have permission to list/watch cluster-wide ConfigMaps or Secrets |
| Generic | Karpenter has permissions to create, modify, and delete nodes from the cluster, and evict any pod. | Cluster Operators running applications with varying security profiles in the same cluster may want to configure dedicated nodes and scheduling rules for Karpenter to mitigate potential container escapes from other containers |
| AWS-Specific | The Karpenter IAM policy is encoded in the GitHub repo. Any additional permissions possibly granted to that role by the administrator are out of scope | |
| AWS-Specific | The Karpenter pod uses IRSA for AWS credentials | Setup of IRSA is out of scope for this document |

## Generic Threats and Mitigations

Expand All @@ -68,6 +68,7 @@ Karpenter has permissions to create and manage cloud instances. Karpenter has Ku
* `karpenter.sh/managed-by`
* `karpenter.sh/nodepool`
* `kubernetes.io/cluster/${CLUSTER_NAME}`
* `karpenter.sh/provisioner-name` (prior to `v0.32.0`)

Any user that has the ability to Create/Delete tags on CloudProvider instances will have the ability to orchestrate Karpenter to Create/Delete CloudProvider instances as a side effect.

Expand All @@ -81,17 +82,34 @@ In addition, as of v0.29.0, Karpenter will Drift on Security Groups and Subnets.

**Background**: Many IAM roles in an AWS account may trust the EC2 service principal. IAM administrators must grant the `iam:PassRole` permission to IAM principals to allow those principals in the account to launch instances with specific roles.

**Threat:** A Cluster Operator attempts to create a Node Template with an IAM role not intended for Karpenter
**Threat:** A Cluster Operator attempts to create an `EC2NodeClass` with an IAM role not intended for Karpenter

**Mitigation**: Cluster Operators must enumerate the roles in the resource section of the IAM policy granted to the Karpenter role for the `iam:PassRole` action.
**Mitigation**: Cluster Operators must enumerate the roles in the resource section of the IAM policy granted to the Karpenter role for the `iam:PassRole` action. Karpenter will fail to generate an instance profile if role that is specified in the `spec.role` section of the `EC2NodeClass` is not enumerated in the `iam:PassRole` permission.

### Threat: Karpenter can be used to create or terminate EC2 instances outside of the cluster
### Threat: Karpenter can orchestrate the creation/deletion of IAM Instance Profiles it doesn't own

**Background**: Karpenter has permission to create/update/delete instance profiles as part of its controller permissions to ensure that it can auto-generate instance profiles when EC2NodeClasses are created.

**Threat**: An actor who obtains control of the Karpenter pod’s IAM role may delete instance profiles not owned by Karpenter, causing workload disruption to other instances using the profile in the account.

**Mitigation**: Karpenter's controller permissions enforce that it creates instance profiles with tags which provide ownership information. These tags include:

* `karpenter.sh/managed-by`
* `kubernetes.io/cluster/${CLUSTER_NAME}`
* `karpenter.k8s.aws/ec2nodeclass`
* `topology.kubernetes.io/region`

These tags ensure that instance profiles created by Karpenter in the account are unique to that cluster. Karpenter's controller permissions _only_ allow it to act on instance profiles that contain these tags which match the cluster information.

### Threat: Karpenter can be used to create or terminate EC2 instances outside the cluster

**Background**: EC2 instances can exist in an AWS account outside of the Kubernetes cluster.

**Threat**: An actor who obtains control of the Karpenter pod’s IAM role may be able to create or terminate EC2 instances not part of the Kubernetes cluster managed by Karpenter.

**Mitigation**: Karpenter creates instances with tags, several of which can be enforced in the IAM policy granted to the Karpenter IAM role that restrict the instances Karpenter can terminate. One tag can require that the instance was provisioned by a Karpenter controller, another tag can include a cluster name to mitigate any termination between two clusters with Karpenter in the same account. Cluster Operators also can restrict the region to prevent two clusters in the same account with the same name in different regions.
**Mitigation**: Karpenter creates instances with tags, several of which are enforced in the IAM policy granted to the Karpenter IAM role that restrict the instances Karpenter can terminate. One tag requires that the instance was provisioned by a Karpenter controller (`karpenter.sh/nodepool`), another tag can include a cluster name to mitigate any termination between two clusters with Karpenter in the same account (`kubernetes.io/cluster/${CLUSTER_NAME}`. Cluster Operators also can restrict the region to prevent two clusters in the same account with the same name in different regions.

Additionally, Karpenter does not allow tags to be modified on instances unowned by Karpenter after creation, except for the `Name` and `karpenter.sh/nodeclaim` tags. Though these tags can be changed after instance creation, `aws:ResourceTag` conditions enforce that the Karpenter controller is only able to change these tags on instances that it already owns, enforced through the `karpenter.sh/nodepool` and `kubernetes.io/cluster/${CLUSTER_NAME}` tags.

### Threat: Karpenter launches an EC2 instance using an unintended AMI

Expand Down

0 comments on commit e2af747

Please sign in to comment.