EKS-Terraform-Setup

EKS setup implementation using combination of Terraform and Cloudformation template by AWS with tweaks & customization to user-data. Fully functional templates to deploy your VPC and Kubernetes clusters together with addons like kube-proxy & kube DNS. Also, worker nodes are part of AutoScallingGroup which consists of spot and on-demand instances.

Curretly it launches only spot to save cost. You can go head & modify the variable.tf, eks-node-group config to launch on-demand instances.

Important Assumption

This doc assumes a working aws-cli setup with suffcient AWS IAM permission to create AWS resource in the module. Please use aws-vault to setup aws credentials securely
Assumes working kubectl cli matching the version of EKS cluster & helm3 local setip for installing cluster addons. local-null-provisioner module will fail without it
The cluster identity is attached to the role/user creating the tf resource. This is automatically done for the user & role declared in variables.tf
RBAC permission must be setup to ensure other user/group h'v permission towards the cluster. More details here
Managed node group is created via TF module & unamanged spot node group via aws provided CF teamplate via resource block
Local provisioner setups extra utilities in the cluster. Make sure the default values.yaml are modified or use your own custom kubernetes manifest before enabling the create_eks_utilities variable.

Note: The terraform apply can very rarely **abruptly exit** while installing EKS-addons. **This is a known issue**. Re-run terraform apply again

Tips & Suggestion

The terraform version version.tf is >=1.0.0. Ideally any version >=0.15 should work. Please report any issue
The module uses env variable for to pass the iam role to setup the cluster
Supports kube-bench integration via service account pod-reader which is automatically created. Follow the instructions in additional resource link to set your cluster for audit
The Unmanaged spot node group are tainted with spotInstance=true:PreferNoSchedule. Incoming pods must h'v toleration for the taint set
Unmanaged spot Node group ASG+LT can be a playground for SRE team to experiment with ami/asg/LT configuration before prod rollout, assuming no critical workloads are scheduled. Always use a standby EKS environment for experiments

Features

Creates two Managed spot node group & one Unmanaged spot node group by default
Initializes custom launch template for EKS Managed node group and Unmanaged spot node group
Nodes in both Managed/Unmanaged spot node group joins the cluster automatically via user-data setup.
Modifies the auth configmap to include the iam role used by Unamanged nodegroup
Creates S3 & ECR endpoint for the VPC
Install cluster addons listed below along with control plan logging enabled
Cillium is the preferred network plugin over VPC CNI plugin(Disabled by default)
Automatically Configures cluster & bastion host access from your public IP instead of opening cluster endpoint to WWW. Ideally, cluster endpoint should be private & accessible via VPN.
kubelet garbage collection to clean up the image cache in the worker node when the disk usage reaches 70%
irsa for pods (pod_reader) to assume role with permission to SSM, ECR, S3 & iam assume role
Repo is integrated with bridge cloud for infrastruce security scanning & vulnerability scanning against CIS benmarks. Generates detailed categorized error report and Infrastructure as Code analysis
Predictive Horizontal Pod Autoscaler which preditcs replica count ahead of time. Pls check addons section

Note: EKS Managed Node Grp, behind the scenes creates a clone of the custom launch template and binds it to the EKS nodegroup. Please note that incrementing the version of the launch template will cause graceful node rollout to the new version. Depends on how soon the running pod can be evicted

IAC checks

Pull request Warnings:

Upcoming feature (WIP...)

kyverno integration instead of PSP
Cross-plane intergration
Custom Addons (more on this later..)
Terragoat integration
Bottle-rocket ami support
containerd as default-runtime instead of docker in EKS-1.21. Use the flag --container-runtime containerd during kubelet initialization (supported in EKS-1.21 release ami)

terraform-aws

To kick-start, you must configure your environment variables in variables.tf file, to enable or disable the components that you wish to install as part of EKS setup. Please make sure, to run init/fmt/validate/plan in the order before applying. Optionally you can save run information if you are running locally or use TF_LOG env variable.

terraform init;terraform fmt;terraform validate;terraform plan
terraform apply -auto-approve 2>&1 | tee <filename>

You will find latest setup of following components:

VPC with public/private subnets, enabled flow logs and VPC endpoints for ECR and S3
EKS controlplane with worker nodes in private subnets. Launched both spot and ondemnd (disabled) instances based on variables flag)
Enable or disable configuration of Managed/Unmanaged spot Node Groups via variables.tf
Automatically configure aws-auth configmap for worker nodes for both Managed & Unmanaged spot node group to join the cluster
OpenID Connect provider for assigning IAM roles to service accounts (irsa) in k8s
Defautl EKS cluster add-ons (CoreDNS + kube-proxy) supported by EKS
NodeDrainer lambda which will drain worker nodes during rollingUpdate of the nodes (This is only applicable to Unmanaged spot Node group).
IRSA setup for aws-node authorization, cluster-autoscaler, secrets EBS csi driver & more...
Automatic draining od spot instance using node-drainer implementation via Asg lifecycle hook

Kubernetes Tools + Addons

All the templates for additional deployments/daemonsets can be found in eks_manifeest.

To apply templates simply run kubectl apply -f . from a desired folder with respective template. Please note some of the addons are automatically installed via local-null-provisioner.

Following are the componets that are installed by default:

Following addons installation is controlled via tf-variable create_eks_utilities. This variable is disabled by default. You need to enable it to support addon installation mentioned below.

Addons(WIP..):

Docs and other additional resources

Check the Terraform module from modules dir

Useful resources

EKS platforms information Worker nodes upgrades

Generate kubeconfig file

The local provisioner automatically configures kubeconfig, assuming the aws cli setup is working. Anyway for manual config, here is the command..

$ aws eks list-clusters
$ aws eks update-kubeconfig --name ${cluster_name}

License

calshankar/terraform-eks

ErrorLooks like something went wrong!

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Languages

Error
Looks like something went wrong!

Packages