title | description |
---|---|
AWS EKS Kubernetes with AWS Load Balancer Controller |
Kubernetes Role and Role Binding in combination with AWS IAM Role, Group and User on EKS Cluster |
- Comment Public Node Group and its outputs
- Uncomment Private Node Group and its outputs
- Create IAM Policy and make a note of Policy ARN
- Create IAM Role and k8s Service Account and bound them together
- Install AWS Load Balancer Controller using HELM Terraform Provider
- Understand IngressClass Concept and create a default Ingress Class
Step-02: Update EKS Cluster Node Groups - Comment Public Node Group and Uncomment Private Node Group
- Project Folder: 01-ekscluster-terraform-manifests
- Comment all code related to Public Node Group
/*
# Create AWS EKS Node Group - Public
resource "aws_eks_node_group" "eks_ng_public" {
cluster_name = aws_eks_cluster.eks_cluster.name
node_group_name = "${local.name}-eks-ng-public"
node_role_arn = aws_iam_role.eks_nodegroup_role.arn
subnet_ids = module.vpc.public_subnets
version = var.cluster_version #(Optional: Defaults to EKS Cluster Kubernetes version)
ami_type = "AL2_x86_64"
capacity_type = "ON_DEMAND"
disk_size = 20
instance_types = ["t3.medium"]
remote_access {
ec2_ssh_key = "eks-terraform-key"
}
scaling_config {
desired_size = 1
min_size = 1
max_size = 2
}
# Desired max percentage of unavailable worker nodes during node group update.
update_config {
max_unavailable = 1
#max_unavailable_percentage = 50 # ANY ONE TO USE
}
# Ensure that IAM Role permissions are created before and deleted after EKS Node Group handling.
# Otherwise, EKS will not be able to properly delete EC2 Instances and Elastic Network Interfaces.
depends_on = [
aws_iam_role_policy_attachment.eks-AmazonEKSWorkerNodePolicy,
aws_iam_role_policy_attachment.eks-AmazonEKS_CNI_Policy,
aws_iam_role_policy_attachment.eks-AmazonEC2ContainerRegistryReadOnly,
kubernetes_config_map_v1.aws_auth
]
tags = {
Name = "Public-Node-Group"
}
}
*/
- Project Folder: 01-ekscluster-terraform-manifests
- Uncomment all code related to Private Node Group
- What is Happening with this step ?
- We are going to deploy all our Application workloads in Private Subnets of a VPC with this approach
- Our Load Balancer gets created in Public Subnet and sends the traffic to our Application Kubernetes Workloads running in Private Node Groups in Private Subnets of a VPC
# Create AWS EKS Node Group - Private
resource "aws_eks_node_group" "eks_ng_private" {
cluster_name = aws_eks_cluster.eks_cluster.name
node_group_name = "${local.name}-eks-ng-private"
node_role_arn = aws_iam_role.eks_nodegroup_role.arn
subnet_ids = module.vpc.private_subnets
version = var.cluster_version #(Optional: Defaults to EKS Cluster Kubernetes version)
ami_type = "AL2_x86_64"
capacity_type = "ON_DEMAND"
disk_size = 20
instance_types = ["t3.medium"]
remote_access {
ec2_ssh_key = "eks-terraform-key"
}
scaling_config {
desired_size = 1
min_size = 1
max_size = 2
}
# Desired max percentage of unavailable worker nodes during node group update.
update_config {
max_unavailable = 1
#max_unavailable_percentage = 50 # ANY ONE TO USE
}
# Ensure that IAM Role permissions are created before and deleted after EKS Node Group handling.
# Otherwise, EKS will not be able to properly delete EC2 Instances and Elastic Network Interfaces.
depends_on = [
aws_iam_role_policy_attachment.eks-AmazonEKSWorkerNodePolicy,
aws_iam_role_policy_attachment.eks-AmazonEKS_CNI_Policy,
aws_iam_role_policy_attachment.eks-AmazonEC2ContainerRegistryReadOnly,
kubernetes_config_map_v1.aws_auth
]
tags = {
Name = "Private-Node-Group"
}
}
- Project Folder: 01-ekscluster-terraform-manifests
- Comment Public Node Group Outputs and Uncomment Private Node Group Outputs
# EKS Cluster Outputs
output "cluster_id" {
description = "The name/id of the EKS cluster."
value = aws_eks_cluster.eks_cluster.id
}
output "cluster_arn" {
description = "The Amazon Resource Name (ARN) of the cluster."
value = aws_eks_cluster.eks_cluster.arn
}
output "cluster_certificate_authority_data" {
description = "Nested attribute containing certificate-authority-data for your cluster. This is the base64 encoded certificate data required to communicate with your cluster."
value = aws_eks_cluster.eks_cluster.certificate_authority[0].data
}
output "cluster_endpoint" {
description = "The endpoint for your EKS Kubernetes API."
value = aws_eks_cluster.eks_cluster.endpoint
}
output "cluster_version" {
description = "The Kubernetes server version for the EKS cluster."
value = aws_eks_cluster.eks_cluster.version
}
output "cluster_iam_role_name" {
description = "IAM role name of the EKS cluster."
value = aws_iam_role.eks_master_role.name
}
output "cluster_iam_role_arn" {
description = "IAM role ARN of the EKS cluster."
value = aws_iam_role.eks_master_role.arn
}
output "cluster_oidc_issuer_url" {
description = "The URL on the EKS cluster OIDC Issuer"
value = aws_eks_cluster.eks_cluster.identity[0].oidc[0].issuer
}
output "cluster_primary_security_group_id" {
description = "The cluster primary security group ID created by the EKS cluster on 1.14 or later. Referred to as 'Cluster security group' in the EKS console."
value = aws_eks_cluster.eks_cluster.vpc_config[0].cluster_security_group_id
}
/*
# EKS Node Group Outputs - Public
output "node_group_public_id" {
description = "Public Node Group ID"
value = aws_eks_node_group.eks_ng_public.id
}
output "node_group_public_arn" {
description = "Public Node Group ARN"
value = aws_eks_node_group.eks_ng_public.arn
}
output "node_group_public_status" {
description = "Public Node Group status"
value = aws_eks_node_group.eks_ng_public.status
}
output "node_group_public_version" {
description = "Public Node Group Kubernetes Version"
value = aws_eks_node_group.eks_ng_public.version
}
*/
# EKS Node Group Outputs - Private
output "node_group_private_id" {
description = "Node Group 1 ID"
value = aws_eks_node_group.eks_ng_private.id
}
output "node_group_private_arn" {
description = "Private Node Group ARN"
value = aws_eks_node_group.eks_ng_private.arn
}
output "node_group_private_status" {
description = "Private Node Group status"
value = aws_eks_node_group.eks_ng_private.status
}
output "node_group_private_version" {
description = "Private Node Group Kubernetes Version"
value = aws_eks_node_group.eks_ng_private.version
}
# Change Directory
cd 01-ekscluster-terraform-manifests
# Terraform Initialize
terraform init
# Terraform Validate
terraform validate
# Terraform Plan
terraform plan
# Terraform Apply
terraform apply -auto-approve
# Configure kubeconfig for kubectl
aws eks --region <region-code> update-kubeconfig --name <cluster_name>
aws eks --region us-east-1 update-kubeconfig --name hr-dev-eksdemo1
# Verify Kubernetes Worker Nodes using kubectl
kubectl get nodes
kubectl get nodes -o wide
# Stop the Bastion Host
Go to AWS Mgmt Console -> Services -> EC2 -> Instances -> hr-dev-Bastion-Host -> Instance State -> Stop Instance
- Project Folder: 02-lbc-install-terraform-manifests
- Create DynamoDB Table
dev-aws-lbc
- Create S3 Bucket Key as
dev/aws-lbc/terraform.tfstate
- Important Note:
lbc
stands forLoad Balancer Controller
# Terraform Settings Block
terraform {
required_version = ">= 1.0.0"
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 4.12"
}
helm = {
source = "hashicorp/helm"
#version = "2.5.1"
version = "~> 2.5"
}
http = {
source = "hashicorp/http"
#version = "2.1.0"
version = "~> 2.1"
}
kubernetes = {
source = "hashicorp/kubernetes"
version = "~> 2.11"
}
}
# Adding Backend as S3 for Remote State Storage
backend "s3" {
bucket = "terraform-on-aws-eks"
key = "dev/aws-lbc/terraform.tfstate"
region = "us-east-1"
# For State Locking
dynamodb_table = "dev-aws-lbc"
}
}
# Terraform AWS Provider Block
provider "aws" {
region = var.aws_region
}
# Terraform HTTP Provider Block
provider "http" {
# Configuration options
}
# Terraform Remote State Datasource - Remote Backend AWS S3
data "terraform_remote_state" "eks" {
backend = "s3"
config = {
bucket = "terraform-on-aws-eks"
key = "dev/eks-cluster/terraform.tfstate"
region = var.aws_region
}
}
# Input Variables - Placeholder file
# AWS Region
variable "aws_region" {
description = "Region in which AWS Resources to be created"
type = string
default = "us-east-1"
}
# Environment Variable
variable "environment" {
description = "Environment Variable used as a prefix"
type = string
default = "dev"
}
# Business Division
variable "business_divsion" {
description = "Business Division in the large organization this Infrastructure belongs"
type = string
default = "SAP"
}
# Define Local Values in Terraform
locals {
owners = var.business_divsion
environment = var.environment
name = "${var.business_divsion}-${var.environment}"
common_tags = {
owners = local.owners
environment = local.environment
}
eks_cluster_name = "${data.terraform_remote_state.eks.outputs.cluster_id}"
}
# Datasource: AWS Load Balancer Controller IAM Policy get from aws-load-balancer-controller/ GIT Repo (latest)
data "http" "lbc_iam_policy" {
url = "https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/install/iam_policy.json"
# Optional request headers
request_headers = {
Accept = "application/json"
}
}
output "lbc_iam_policy" {
value = data.http.lbc_iam_policy.body
}
# Resource: Create AWS Load Balancer Controller IAM Policy
resource "aws_iam_policy" "lbc_iam_policy" {
name = "${local.name}-AWSLoadBalancerControllerIAMPolicy"
path = "/"
description = "AWS Load Balancer Controller IAM Policy"
policy = data.http.lbc_iam_policy.body
}
output "lbc_iam_policy_arn" {
value = aws_iam_policy.lbc_iam_policy.arn
}
# Resource: Create IAM Role
resource "aws_iam_role" "lbc_iam_role" {
name = "${local.name}-lbc-iam-role"
# Terraform's "jsonencode" function converts a Terraform expression result to valid JSON syntax.
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = "sts:AssumeRoleWithWebIdentity"
Effect = "Allow"
Sid = ""
Principal = {
Federated = "${data.terraform_remote_state.eks.outputs.aws_iam_openid_connect_provider_arn}"
}
Condition = {
StringEquals = {
"${data.terraform_remote_state.eks.outputs.aws_iam_openid_connect_provider_extract_from_arn}:aud": "sts.amazonaws.com",
"${data.terraform_remote_state.eks.outputs.aws_iam_openid_connect_provider_extract_from_arn}:sub": "system:serviceaccount:kube-system:aws-load-balancer-controller"
}
}
},
]
})
tags = {
tag-key = "AWSLoadBalancerControllerIAMPolicy"
}
}
# Associate Load Balanacer Controller IAM Policy to IAM Role
resource "aws_iam_role_policy_attachment" "lbc_iam_role_policy_attach" {
policy_arn = aws_iam_policy.lbc_iam_policy.arn
role = aws_iam_role.lbc_iam_role.name
}
output "lbc_iam_role_arn" {
description = "AWS Load Balancer Controller IAM Role ARN"
value = aws_iam_role.lbc_iam_role.arn
}
# Datasource: EKS Cluster Auth
data "aws_eks_cluster_auth" "cluster" {
name = data.terraform_remote_state.eks.outputs.cluster_id
}
# HELM Provider
provider "helm" {
kubernetes {
host = data.terraform_remote_state.eks.outputs.cluster_endpoint
cluster_ca_certificate = base64decode(data.terraform_remote_state.eks.outputs.cluster_certificate_authority_data)
token = data.aws_eks_cluster_auth.cluster.token
}
}
# Install AWS Load Balancer Controller using HELM
# Resource: Helm Release
resource "helm_release" "loadbalancer_controller" {
depends_on = [aws_iam_role.lbc_iam_role]
name = "aws-load-balancer-controller"
repository = "https://aws.github.io/eks-charts"
chart = "aws-load-balancer-controller"
namespace = "kube-system"
set {
name = "image.repository"
value = "602401143452.dkr.ecr.us-east-1.amazonaws.com/amazon/aws-load-balancer-controller" # Changes based on Region - This is for us-east-1 Additional Reference: https://docs.aws.amazon.com/eks/latest/userguide/add-ons-images.html
}
set {
name = "serviceAccount.create"
value = "true"
}
set {
name = "serviceAccount.name"
value = "aws-load-balancer-controller"
}
set {
name = "serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
value = "${aws_iam_role.lbc_iam_role.arn}"
}
set {
name = "vpcId"
value = "${data.terraform_remote_state.eks.outputs.vpc_id}"
}
set {
name = "region"
value = "${var.aws_region}"
}
set {
name = "clusterName"
value = "${data.terraform_remote_state.eks.outputs.cluster_id}"
}
}
# Helm Release Outputs
output "lbc_helm_metadata" {
description = "Metadata Block outlining status of the deployed release."
value = helm_release.loadbalancer_controller.metadata
}
# Terraform Kubernetes Provider
provider "kubernetes" {
host = data.terraform_remote_state.eks.outputs.cluster_endpoint
cluster_ca_certificate = base64decode(data.terraform_remote_state.eks.outputs.cluster_certificate_authority_data)
token = data.aws_eks_cluster_auth.cluster.token
}
- Understand what is Ingress Class
- Understand how it overrides the default deprecated annotation
#kubernetes.io/ingress.class: "alb"
- Ingress Class Documentation Reference
- Different Ingress Controllers available today
# Resource: Kubernetes Ingress Class
resource "kubernetes_ingress_class_v1" "ingress_class_default" {
depends_on = [helm_release.loadbalancer_controller]
metadata {
name = "my-aws-ingress-class"
annotations = {
"ingressclass.kubernetes.io/is-default-class" = "true"
}
}
spec {
controller = "ingress.k8s.aws/alb"
}
}
# Change Directory
02-lbc-install-terraform-manifests
# Terraform Initialize
terraform init
# Terraform Validate
terraform validate
# Terraform Plan
terraform plan
# Terraform Apply
terraform apply -auto-approve
# Verify that the controller is installed.
kubectl -n kube-system get deployment
kubectl -n kube-system get deployment aws-load-balancer-controller
kubectl -n kube-system describe deployment aws-load-balancer-controller
# Sample Output
Kalyans-Mac-mini:02-lbc-install-terraform-manifests kalyanreddy$ kubectl -n kube-system get deployment aws-load-balancer-controller
NAME READY UP-TO-DATE AVAILABLE AGE
aws-load-balancer-controller 2/2 2 2 18m
Kalyans-Mac-mini:02-lbc-install-terraform-manifests kalyanreddy$
# Verify AWS Load Balancer Controller Webhook service created
kubectl -n kube-system get svc
kubectl -n kube-system get svc aws-load-balancer-webhook-service
kubectl -n kube-system describe svc aws-load-balancer-webhook-service
# Sample Output
Kalyans-Mac-mini:02-lbc-install-terraform-manifests kalyanreddy$ kubectl -n kube-system get svc aws-load-balancer-webhook-service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
aws-load-balancer-webhook-service ClusterIP 172.20.110.33 <none> 443/TCP 18m
Kalyans-Mac-mini:02-lbc-install-terraform-manifests kalyanreddy$
# Verify Labels in Service and Selector Labels in Deployment
kubectl -n kube-system get svc aws-load-balancer-webhook-service -o yaml
kubectl -n kube-system get deployment aws-load-balancer-controller -o yaml
Observation:
1. Verify "spec.selector" label in "aws-load-balancer-webhook-service"
2. Compare it with "aws-load-balancer-controller" Deployment "spec.selector.matchLabels"
3. Both values should be same which traffic coming to "aws-load-balancer-webhook-service" on port 443 will be sent to port 9443 on "aws-load-balancer-controller" deployment related pods.
# List Pods
kubectl get pods -n kube-system
# Review logs for AWS LB Controller POD-1
kubectl -n kube-system logs -f <POD-NAME>
kubectl -n kube-system logs -f aws-load-balancer-controller-86b598cbd6-5pjfk
# Review logs for AWS LB Controller POD-2
kubectl -n kube-system logs -f <POD-NAME>
kubectl -n kube-system logs -f aws-load-balancer-controller-86b598cbd6-vqqsk
# List Service Account and its secret
kubectl -n kube-system get sa aws-load-balancer-controller
kubectl -n kube-system get sa aws-load-balancer-controller -o yaml
kubectl -n kube-system get secret <GET_FROM_PREVIOUS_COMMAND - secrets.name> -o yaml
kubectl -n kube-system get secret aws-load-balancer-controller-token-5w8th
kubectl -n kube-system get secret aws-load-balancer-controller-token-5w8th -o yaml
## Decoce ca.crt using below two websites
https://www.base64decode.org/
https://www.sslchecker.com/certdecoder
## Decode token using below two websites
https://www.base64decode.org/
https://jwt.io/
Observation:
1. Review decoded JWT Token
# List Deployment in YAML format
kubectl -n kube-system get deploy aws-load-balancer-controller -o yaml
Observation:
1. Verify "spec.template.spec.serviceAccount" and "spec.template.spec.serviceAccountName" in "aws-load-balancer-controller" Deployment
2. We should find the Service Account Name as "aws-load-balancer-controller"
# List Pods in YAML format
kubectl -n kube-system get pods
kubectl -n kube-system get pod <AWS-Load-Balancer-Controller-POD-NAME> -o yaml
kubectl -n kube-system get pod aws-load-balancer-controller-65b4f64d6c-h2vh4 -o yaml
Observation:
1. Verify "spec.serviceAccount" and "spec.serviceAccountName"
2. We should find the Service Account Name as "aws-load-balancer-controller"
3. Verify "spec.volumes". You should find something as below, which is a temporary credentials to access AWS Services
CHECK-1: Verify "spec.volumes.name = aws-iam-token"
- name: aws-iam-token
projected:
defaultMode: 420
sources:
- serviceAccountToken:
audience: sts.amazonaws.com
expirationSeconds: 86400
path: token
CHECK-2: Verify Volume Mounts
volumeMounts:
- mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
name: aws-iam-token
readOnly: true
CHECK-3: Verify ENVs whose path name is "token"
- name: AWS_WEB_IDENTITY_TOKEN_FILE
value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
# List aws-load-balancer-tls secret
kubectl -n kube-system get secret aws-load-balancer-tls -o yaml
# Verify the ca.crt and tls.crt in below websites
https://www.base64decode.org/
https://www.sslchecker.com/certdecoder
# Make a note of Common Name and SAN from above
Common Name: aws-load-balancer-controller
SAN: aws-load-balancer-webhook-service.kube-system, aws-load-balancer-webhook-service.kube-system.svc
# List Pods in YAML format
kubectl -n kube-system get pods
kubectl -n kube-system get pod <AWS-Load-Balancer-Controller-POD-NAME> -o yaml
kubectl -n kube-system get pod aws-load-balancer-controller-65b4f64d6c-h2vh4 -o yaml
Observation:
1. Verify how the secret is mounted in AWS Load Balancer Controller Pod
CHECK-2: Verify Volume Mounts
volumeMounts:
- mountPath: /tmp/k8s-webhook-server/serving-certs
name: cert
readOnly: true
CHECK-3: Verify Volumes
volumes:
- name: cert
secret:
defaultMode: 420
secretName: aws-load-balancer-tls