K8s-Microservices-Platform (Migration of ECS Microservices CICD Platfom Project)

Check ECS-MICROSERVICES-CICD-PLATFORM Project for relations and relationg Architectural Diagram

Project Summary

Migrated a production microservices workload from AWS ECS Fargate to Amazon EKS, replacing AWS-native tooling with cloud-agnostic, industry-standard alternatives. Same application architecture, same SQS-based design — rebuilt on Kubernetes, Terraform, and GitHub Actions.

---

Key Outcomes

• Migrated containerized microservices from ECS Fargate to Amazon EKS
• Replaced CloudFormation with modular Terraform
• Replaced CodePipeline/CodeBuild with GitHub Actions
• Implemented IRSA for keyless AWS authentication from pods
• Deployed AWS Load Balancer Controller to manage ALB provisioning from Kubernetes
• Deployed Prometheus, Grafana, and Alertmanager for Observability
• Multi-environment promotion strategy (dev → staging → prod) with manual approval gates

---

Architecture

Three loosely coupled services communicating via Amazon SQS:

• Frontend — Nginx serving static UI, proxying /api/* requests to the API internally
• API — Python Flask exposing /api/health and /api/job endpoints
• Worker — Background processor polling SQS and executing jobs asynchronously

Internet → ALB (public subnet)
    → /api/* → API Service → Flask pod
    → /*     → Frontend Service → Nginx pod
                                      ↓
                              Nginx reverse proxy → API Service (internal)
 
Worker pod → SQS (outbound only, no Service needed)

---

Infrastructure — Terraform

Modular Terraform replacing CloudFormation nested stacks.

terraform/
├── main.tf
├── variables.tf
├── outputs.tf
├── envs/
│   ├── dev.tfvars
│   ├── staging.tfvars
│   └── prod.tfvars
└── modules/
    ├── vpc/       # VPC, subnets, IGW, NAT Gateways
    ├── eks/       # EKS cluster, node group, access entries
    ├── ecr/       # ECR repositories (frontend, api, worker)
    ├── sqs/       # SQS job queue
    └── iam/       # All IAM roles — cluster, node group, IRSA, GitHub Actions, LB Controller

Each environment has its own Terraform workspace and .tfvars file — isolated state stored in S3.

---

Kubernetes Manifests

k8/
├── serviceaccount.yaml               # IRSA identity for app pods → SQS role
├── lb-controller-serviceaccount.yaml # IRSA identity for LB Controller → ALB role
├── frontend-deployment.yaml          # Deployment + ClusterIP Service
├── api-deployment.yaml               # Deployment + ClusterIP Service
├── worker-deployment.yaml            # Deployment only
├── ingress.yaml                      # ALB routing rules
├── sevicemonitor.yaml                # Prometheus scraping for app
└── alertingrules.yaml                # Prometheus alert rules

ECS → Kubernetes Mapping

ECS	Kubernetes
Task Definition	Pod spec
ECS Service (desired count)	Deployment
ECS Service (load balancing)	ClusterIP Service
Task Role	ServiceAccount + IRSA
ALB + Target Groups	Ingress + AWS Load Balancer Controller
Cloud Map	Kubernetes DNS

---

GitHub Actions

Two workflows replacing CodePipeline + CodeBuild:

terraform.yml — Manual trigger. Choose action (plan, apply, destroy) and environment (dev, staging, prod) from a dropdown. Runs Terraform against the selected workspace and .tfvars file.

deploy.yml — Triggers automatically on push to dev, staging, or main branches.

Deployment Lifecycle (To be done for all environments i.e prod, staging, and main)

Step 1 — Terraform Init

Initialize the project and configure the S3 backend:

Step 2 — Terraform Plan

Review what will be created before applying:

Step 3 — Terraform Apply

Provision the full infrastructure — VPC, EKS cluster, node group, ECR repos, SQS queue, IAM roles, This

Step 4 - Add Secrets to GitHub

Add Github Actions workflow secrets to Github from Terraform outputs

Step 5 - Add , Commit and Push to Git

Commit and Push your Git repo on your local computer to GitHub

Step 6 - Workflow Trigger

Push triggers the deploy workflow to build the images, push images to ECR, Apply Kubernetes Manifests to EKS , Installs Helm, Installs Load Balancer Controller to route traffic to pods, Installs kube-prometheus-stack for observability(Prometheus,Grafana,AlertManager), Apply Manifests for the prometheus stack

Code pushed
    → OIDC authentication to AWS (no stored keys)
        → Docker build → tag with git commit SHA
            → Push to ECR
                → kubectl apply manifests
                    → Install Helm
                      → Install AWS LoadBalancer Controller
                        → Install kube-prometheus-stack
                            → kubectl apply manifests(kube-prometheus-stack)

                      
        

Workflow Completed

EKS Pods and K8 LoadBalancer showing DNS Name

Working Web Application

Images are tagged with the git commit SHA for full traceability. A sed command substitutes the IMAGE_TAG placeholder in manifests before kubectl apply runs — no latest tag required.

Multi-environment promotion uses GitHub Environment protection rules — deployments pause for manual approval.

Manual Approval Section

---

CI/CD

Make Changes to your code in the App file , in this case app/frontend/index.html and push to Git, This will trigger deploy workflow, and make changes to your app. Refresh Loadbalancer Domain Web Page.

Check for previous app/frontend/index.html version in the commit history

Security

• No hardcoded credentials anywhere
• Pods authenticate to AWS via IRSA (OIDC token → IAM role)
• GitHub Actions authenticates via OIDC (no stored AWS keys)
• Worker nodes and pods in private subnets — ALB is the only public entry point
• Environment-scoped GitHub Secrets per environment

---

Observability

Deployed via kube-prometheus-stack Helm chart into a dedicated monitoring namespace. Observability runs alongside the application workloads on the same EKS cluster.

Stack

• Prometheus — scrapes metrics from the cluster and application every 15 seconds
• Grafana — dashboards for visualizing cluster and application metrics
• Alertmanager — routes alerts based on defined rules
• Node Exporter — exposes EC2 node-level metrics (CPU, memory, disk)
• kube-state-metrics — exposes Kubernetes object metrics (pod status, replica counts)

Application Metrics

The Flask API is instrumented with prometheus-client. A /metrics endpoint exposes:

• api_requests_total — total requests by method, endpoint, and status code
• api_request_duration_seconds — request latency histogram
• api_jobs_sent_total — successful SQS job sends
• api_job_errors_total — failed SQS job sends

A ServiceMonitor resource tells Prometheus to scrape the API Service every 15 seconds automatically.

Grafana Dashboards

Four custom panels built on application metrics:

• API Request Rate — rate(api_requests_total[5m])
• API Error Rate — rate(api_requests_total{status=~"5.."}[5m])
• API P95 Latency — histogram_quantile(0.95, rate(api_request_duration_seconds_bucket[5m]))
• Jobs Sent to SQS — rate(api_jobs_sent_total[5m])

Alerting Rules

Five PrometheusRule alerts defined in k8/alertingrules.yaml:

Alert	Condition
APIPodCrashing	Pod restart rate > 0 for 1 min
APIHighErrorRate	5xx rate > 0.1 req/s for 2 min
APIJobFailures	SQS error rate > 0 for 1 min
APIPodNotRunning	Available replicas < 1 for 1 min
APIHighLatency	P95 latency > 1s for 2 min

---

Evolution from ECS Project

Direct migration of ECS-Microservices-CICD-Platform.

	ECS Project	This Project
Orchestration	ECS Fargate	Kubernetes (EKS)
IaC	CloudFormation	Terraform
CI/CD	CodePipeline + CodeBuild	GitHub Actions
Pod Identity	Task Role	IRSA
Load Balancing	ALB (CloudFormation)	ALB (Ingress + LB Controller)
Secrets	SSM Parameter Store	GitHub Secrets + K8s Secrets

---

Repository Structure

├── app/
│   ├── api/
│   ├── frontend/
│   └── worker/
├── k8/
├── terraform/
└── .github/workflows/