AWS uses Identity and Access Management (IAM) roles to manage authentication and authorization for AWS Services. An IAM role is an IAM entity that defines a set of permissions for making AWS service requests. For more information about IAM, please see here.
Out of the box, a Kubernetes cluster created by kops has two IAM roles: a master IAM role and a node IAM role. Additional policies may be associated with these IAM roles. More details about the IAM roles and associated policies is available at https://github.com/kubernetes/kops/blob/master/docs/iam_roles.md.
A Pod may be assigned a specific IAM role using different mechanisms. A third-party plugin such as kube2iam can be configured to apply IAM roles at the pod level. Hashicorp’s Vault can be used to assign IAM dynamic IAM roles as well.
This chapter will walk you through:
-
Update default policies for the cluster
-
Use kube2iam to assign IAM roles at runtime
-
Use Hashicorp Vault to assign IAM roles at runtime
This chapter uses a cluster with 3 master nodes and 5 worker nodes as described here: multi-master, multi-node gossip based cluster.
All configuration files for this chapter are in the roles directory.
Kubernetes cluster created using kops has two IAM roles - one for all master nodes and one for all worker nodes. The master IAM role is associated with a number of policies related to EC2, Elastic Load Balancer (ELB), EC2 Container Registry (ECR), Route53, and Key Management Service (KMS). The worker nodes require a number of EC2, ECR, and Route53 policies applied. kops automatically configures these roles and policies for you (see here for a full list).
These policies may be augmented via kops. For example, to allow all nodes ElasticSearch access, perform the following steps:
Edit your cluster using the command:
$ kops edit cluster example.cluster.k8s.local
Add the following to the spec:
additionalPolicies:
node: |
[
{
"Effect": "Allow",
"Action": ["es:*"],
"Resource": ["*"]
}
]
Note, you need to use spaces and proper indentation instead of tab for this configuration to be recognized.
Update the cluster for changes to take effect:
$ kops update cluster example.cluster.k8s.local --yes
In a similar fashion, policies may be applied to all masters by setting the additionalPolicies.master property as below:
additionalPolicies:
master: |
[
{
"Effect": "Allow",
"Action": ["es:*"],
"Resource": ["*"]
}
]
Once again, the cluster needs to be updated for these changes to take effect.
Kube2iam provides different AWS IAM roles for pods running on Kubernetes based on annotations.
Kube2iam works by intercepting traffic from the containers to the EC2 Metadata API, calling the AWS Security Token Service (STS) API to obtain temporary credentials using the pod configured role, then using these temporary credentials to perform the original request.
The following steps describe how to configure kube2iam:
-
Edit your cluster using the command:
$ kops edit cluster example.cluster.k8s.local
-
Update the
specto allow STS to assume roles:additionalPolicies: node: | [ { "Effect": "Allow", "Action": ["sts:AssumeRole"], "Resource": ["*"] } ] -
Update the cluster for changes to take effect:
kops update cluster example.cluster.k8s.local --yes
-
Deploy kube2iam as a DaemonSet:
$ kubectl apply -f templates/kube2iam-ds.yaml daemonset "kube2iam" created
Note that the
kube2iam-ds.yamlis assuming that your network configuration is based onkubenet, which is also the default kops configuration. If you have changed this as part of another exercise (e.g. switched to Calico) you will need to change the value of the--host-interfacearg in the configuration file. For example, set tocal+if using Calico.
It is necessary to create an IAM role which can assume other roles and assign it to each Kubernetes worker.
We need the Amazon Resource Name (ARN) of the IAM Role assigned to the worker nodes. We will need this as part of creating pod roles. We can obtain this by running the following command:
$ kubectl get no \
--selector=kubernetes.io/role==node \
-o jsonpath='{.items[0].spec.externalID}' | \
xargs aws ec2 describe-instances \
--instance-id \
--query 'Reservations[*].Instances[*].IamInstanceProfile.Arn' | \
sed -e 's/instance-profile/role/g'
This command retrieves the AWS EC2 instance id (stored as .spec.externalID) of a worker node. It then uses the AWS CLI to query the ARN for the given EC2 instance id.
It shows an output like:
[
[
"arn:aws:iam::<account-id>:role/nodes.example.cluster.k8s.local"
]
]
Note down the ARN from this output.
Edit the templates/pod-role-trust-policy.json file, replace {{NodeIAMRoleARN}} with the IAM Role ARN obtained from the previous step.
We will first create a role with no permissions. By configuring the Trusted Policy of the role, we are allowing kube2iam (via the worker node IAM Instance Profile Role) to assume the pod role. Make note of the role ARN from the response:
$ aws iam create-role \ --role-name MyPodRole \ --assume-role-policy-document \ file://templates/pod-role-trust-policy.json
It shows an output as:
{
"Role": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Effect": "Allow",
"Sid": ""
},
{
"Action": "sts:AssumeRole",
"Principal": {
"AWS": "arn:aws:iam::<account-id>:role/nodes.cluster.k8s.local"
},
"Effect": "Allow",
"Sid": ""
}
]
},
"RoleId": "AROAJANTQ2EP23B2BE2YQ",
"CreateDate": "2017-10-25T01:59:51.585Z",
"RoleName": "MyPodRole",
"Path": "/",
"Arn": "arn:aws:iam::<account-id>:role/MyPodRole"
}
}
iam.amazonaws.com/role annotation on the pod is used to assign an IAM role to a pod. Let’s set this annotation on our pod. The templates/pod-with-kube2iam.yaml file looks like:
apiVersion: v1
kind: Pod
metadata:
name: aws-cli
labels:
name: aws-cli
annotations:
iam.amazonaws.com/role: MyPodRole
spec:
containers:
- image: cgswong/aws:aws
command:
- "sleep"
- "9999999"
name: aws-cli
Run the following command:
$ kubectl create -f templates/pod-with-kube2iam.yaml pod "aws-cli" created
This will create a pod with the AWS CLI already installed, with the MyPodRole IAM role assigned.
Run the AWS CLI to attempt to access S3:
$ kubectl exec -it aws-cli aws s3 ls
An error occurred (AccessDenied) when calling the ListBuckets operation: Access DeniedRecall that the MyPodRole IAM role that we created has no permissions and so the access is denied.
Terminate the pod:
$ kubectl delete po aws-cli --force pod "aws-cli" deleted
Let’s update the role to grant S3 permissions:
$ aws iam attach-role-policy --role-name MyPodRole --policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess
Recreate the pod and then try to access S3 again.
$ kubectl create -f templates/pod-with-kube2iam.yaml
pod "aws-cli" created
$ kubectl exec -it aws-cli aws s3 lsWe should now be authorized and the output should show the list of S3 buckets in this IAM account.
As kube2iam caches STS tokens for 15 minutes, if you make any changes to a role and need it to take effect immediately, you will need to restart the pod.
To govern what roles a pod can assume, you can use the iam.amazonaws.com/allowed-roles namespace annotation. Edit the templates/namespace-role-annotation.yaml file, replace {{NodeIAMRoleARN}} with the IAM Role ARN obtained from the previous step and run the following command:
$ kubectl apply -f templates/namespace-role-annotation.yaml namespace "default" configured
From now on, pods in the default namespace will only be able to optain the MyPodRole.
Hashicorp Vault is a tool for securely accessing secrets. The secrets could be static where they are retrieved from a key-value store or could be dynamically generated when needed such as IAM credentials. This is enabled by the pluggable architecture of Vault that supports different backends. Vault behaves like a virtual filesystem where each backend is mounted at a specific path. For example, by default, AWS backend is mounted at the path aws. An alternative path can be specified using -path at mount time.
Each Vault backend reacts differently to different Vault CLI commands. For example, the command vault read aws/deploy will generate an access key based on the "deploy" role in the context of Vault/AWS secret backend. This role contains an IAM role or ARN mapping.
Make sure to setup Vault as explained in:
And now follow the steps outlined here.
The steps in this section are inspired from https://github.com/calvn/vault-kubernetes-demo/blob/master/5-deploy-aws.md.
Create an IAM user with enough permissions to manage IAM resources. We will be using the IAMFullAccess policy here, but a more locked down custom policy can be provided. For an example on such policy template, refer to the Vault documentation regarding the backend’s root credentials.
-
Create the user:
$ aws iam create-user \ --user-name vault-root { "User": { "UserName": "vault-root", "Path": "/", "CreateDate": "2017-11-22T20:13:39.273Z", "UserId": "AIDAI6SQOXCLURVV2J7BK", "Arn": "arn:aws:iam::<account-id>:user/vault-root" } } -
Attach the
IAMFullAccesspolicy to the user:$ aws iam attach-user-policy \ --user-name vault-root \ --policy-arn arn:aws:iam::aws:policy/IAMFullAccess -
Generate a access key and secret access key pair:
$ aws iam create-access-key \ --user-name vault-root { "AccessKey": { "UserName": "vault-root", "Status": "Active", "CreateDate": "2017-11-22T20:14:00.682Z", "SecretAccessKey": "<secret-key>", "AccessKeyId": "<access-key>" } }
-
Create a policy for this role:
$ vault policy-write kube-auth templates/kube-auth.hcl Policy 'kube-auth' written.
-
Check the policy:
$ vault policies kube-auth path "secret/creds" { capabilities = ["read"] } path "aws/creds/readonly" { capabilities = ["read"] } -
Mount AWS backend:
$ vault mount aws Successfully mounted 'aws' at 'aws'!
-
Configure the AWS secret backend:
$ vault write aws/config/root \ secret_key=<secret-key> \ access_key=<access-key> \ region=us-east-1 Success! Data written to: aws/config/root
-
Create a AWS secret backend role that has read-only permissions on EC2 instances for the account using a AWS managed policy this policy:
$ vault write aws/roles/readonly \ arn=arn:aws:iam::aws:policy/AmazonEC2ReadOnlyAccess Success! Data written to: aws/roles/readonly
Pod will be using this role to request IAM credentials during runtime.
-
Edit
templates/deplloymnet-with-vault.yamlto replace<public-ip-address>with the public IP address of the EC2 instance where Vault server is running. -
Create the Pod:
$ kubectl apply -f templates/deployment-with-vault.yaml deployment "vault-sidecar" created
-
Get the list of Deployment:
$ kubectl get deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE vault-sidecar 1 1 1 1 10s
-
Get the list of Pod:
$ kubectl get pods -l app=vault-sidecar NAME READY STATUS RESTARTS AGE vault-sidecar-79bbc86955-wwlkb 2/2 Running 0 35s
-
Use AWS CLI:
$ kubectl exec -it $(kubectl get pods -l app=vault-sidecar -o jsonpath={.items[0].metadata.name}) aws s3 ls Defaulting container name to aws-cli. Use 'kubectl describe pod/vault-sidecar-79bbc86955-h4pkf' to see all of the containers in this pod. An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied command terminated with exit code 255This is expected because the AWS secret backend gives access to
AmazonEC2ReadOnlyAccessonly. -
Update the AWS secret backend to allow access to S3:
$ vault write aws/roles/readonly \ arn=arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess Success! Data written to: aws/roles/readonly
-
We need to generate a new set of AWS credentials after the policy is updated. In order for this policy to take effect, we need to restart the sidecar container. But its easier to delete and create the Deployment again.
that reads the credentials from Vault. This will allow the new policy to take effect. But its easier to delete the Deployment.
$ kubectl delete -f templates/deployment-with-vault.yaml deployment "vault-sidecar" deleted
-
Create the Deployment again:
$ kubectl apply -f templates/deployment-with-vault.yaml deployment "aws-sidecar-example" created
-
Use AWS CLI:
$ kubectl exec -it $(kubectl get pods -l app=vault-sidecar -o jsonpath={.items[0].metadata.name}) aws s3 lsAnd this shows the listing of S3 buckets in your account.
-
Kubernetes Auth backend is configured for the token to expire in 60 seconds. This was done during Kubernetes Auth backend configuration by setting the
-period=60son thevault write auth/kubernetes/role/democommand. This means the temporary IAM role created for this Deployment will eventually disappear. Trying to access the S3 listing after a few minutes will give an error:$ kubectl exec -it $(kubectl get pods -l app=vault-sidecar -o jsonpath={.items[0].metadata.name}) aws s3 ls Defaulting container name to aws-cli. Use 'kubectl describe pod/vault-sidecar-79bbc86955-7wpt6' to see all of the containers in this pod. An error occurred (InvalidAccessKeyId) when calling the ListBuckets operation: The AWS Access Key Id you provided does not exist in our records. command terminated with exit code 255