Skip to content
This repo will walk customers through setting up the aws kubernetes cni plugin on a self managed kubernetes cluster on AWS. It will also demonstrate how to leverage Calico to enforce Kubernetes network policy.
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github Creating initial file from template Feb 22, 2018
images first commit Feb 22, 2018
CODE_OF_CONDUCT.md Creating initial file from template Feb 22, 2018
CONTRIBUTING.md Creating initial file from template Feb 22, 2018
LICENSE Creating initial file from template Feb 22, 2018
README.adoc fix typos Mar 6, 2018
aws-cni.yaml first commit Feb 22, 2018
calico.yaml first commit Feb 22, 2018
cni-policy.json first commit Feb 22, 2018
deny-all-policy.yaml first commit Feb 22, 2018

README.adoc

Using the Amazon VPC CNI for Kubernetes + Calico network policy enforcement in a kops cluster.

tl:dr steps:

  • Install a cluster with kops

  • Modify IAM policy on ec2 instances (cni.policy.yaml included in this repo)

  • Assure all traffic is allowed between the master and node security groups and all traffic is allowed within the security groups.

  • Pass --node-ip <ip-address> to kubelet on all master and node instances, where <ip-address> equals the primary ip address on eth0 of the ec2 instance

  • Apply the aws-cni.yaml file to your cluster

  • Apply the calico.yaml file to your cluster

  • Apply and test network policy enforcement

Pre-Requisites

Install a cluster with kops

Create an s3 bucket for Cluster State

Add a custom prefix to the bucket name below:

aws s3api create-bucket --bucket prefix-example-com-state-store --region us-east-1
bucket
Setup environment variables

Assure you use the same prefix you defined above to the state store below:

export NAME=awscni.k8s.local
export kops_STATE_STORE=s3://prefix-example-state-store
env
Deploy cluster with kops

This example will deploy a kubernetes cluster in us-east-2a:

kops create cluster --zones us-east-2a --networking cni --yes --name ${NAME}
cluster
Verify nodes

In a few minutes a single-master/dual-node cluster will be deployed into your account. kops will automatically configure the Kubernetes contexts in your config file. From here let’s make sure we can access the node from kubectl:

kubectl get nodes

This will only display the master node in a not ready status for now. We need to perform a few tasks to get the CNI up and running properly.

nodes

Modify IAM policy on ec2 instances

Create a customer managed policy

First we need to modify the IAM policy attached to the masters and nodes to allow them to allocate IP addresses and ENIs from the VPC network.

From the AWS console navigate to the IAM page:

iam-console

Click on "Policies to the left"

policies

Click on the "Create Policy" button and then click on the "JSON" tab:

create-policy

From here copy and paste the contents from the file "cni-policy.json" (included in the root of this repository) and then click "Review Policy".

The next screen will allow you to review the permissions and provide the policy a name, for the purpose of this walkthrough let’s name the policy "aws-cni".

Apply the policy to the nodes and the masters

From the IAM console select the roles menu from the left and then search for "awscni" in the search box.

roles

Next select the "masters.awscni.k8s.local" role and apply the customer managed policy we created in the previous bullet point. Click the "attach policy" button and search for the "aws-cni" policy we previously created and click the "attach policy" button in the lower right hand corner.

attach

Repeat for the "nodes.awscni.k8s.local" policy.

Modify Security Groups

Next we need to modify the security groups to allow all traffic from/to and within the master and node security groups.

First, go to the aws ec2 console and click instances on the left side:

ec2-console

Select the master instance and click on the security group (from the description tab).

Create a new inbound rule to allow all traffic from the nodes group.

security-group

All of the other rules should already exist, at the end what you need to verify is that all traffic flows freely between the masters and nodes security group in addition to all traffic within the groups themselves (all traffic from master security group to master security group).

Set node-ip flag on kubelet

ssh into each instance and retrieve the primary ip address:

curl http://169.254.169.254/latest/meta-data/local-ipv4

Add the --node-ip flag with the value retrieved from above to the kubelet config /etc/sysconfig/kubelet and restart kubelet:

sudo systemctl restart kubelet

Deploy the AWS CNI plugin

Apply the manifest file, this is included in the root of this repository.

kubectl apply -f aws-cni.yaml
cni-apply

Verify the nodes are in a ready state

watch kubectl get nodes
not-ready

Eventually the display will change and all of the nodes will have a status of "ready". If not, stop here, and go back and verify all of the previous steps as the next steps will not work.

ready

Next verify that all of the pods in the kube-system namespace are successfully running, this will validate that the CNI is properly functioning:

kubectl get pods --namespace=kube-system
get-pods

Again if there are any problems here go back and verify all of the previous steps.

Deploy the Calico plugin for network policy

kubectl apply -f calico.yaml
calico-apply

Again verify all of the pods in the kube-system namespace are in good running order, you will have three more pods than you did the first time we performed this step:

kubectl get pods --namespace=kube-system
get-pods-2

Test Network Policy Enforcement

Create a namespace for testing:

kubectl create ns policy-demo
create-namespace

Deploy 3 Nginx pods

kubectl run nginx-deployment --replicas=3 --image=nginx --port=80 --namespace=policy-demo
nginx

Verify the pods are in running state:

kubectl get pods --namespace=policy-demo
nginx-pods

Expose the deployment:

kubectl expose deployment nginx-deployment --type=LoadBalancer --port=80 --namespace=policy-demo
expose

Retrieve the service endpoint:

kubectl get svc --namespace=policy-demo -o=wide
service

Browse to the External IP and you should see the Nginx home page

homepage

Apply default deny all network policy, this file is available in the root of this repository:

kubectl apply -f deny-all-policy.yaml --namespace=policy-demo

Again browse to the external IP and you should now be blocked (note your existing brower may re-use an existing connection, try another browswer or curl the endpoint)

Conclusion

This demonstration has shown how you can leverage the aws kubernetes cni in your own cluster and how to use Calico network policy to enforce your Kubernetes network policy objects.

Cleaning up the example resources

To remove all resources created by this example do the following:

  1. Delete the policy-demo namespace (this will delete all of the resources in the namespace as well):

    kubectl delete ns --policy-demo
  2. Remove the customer policy from the node and master IAM roles

  3. Delete the cluster with kops:

    kubectl delete cluster awscni.k8s.local --yes

License Summary

This sample code is made available under a modified MIT license. See the LICENSE file.

You can’t perform that action at this time.