Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,15 @@ Then browse to [http://localhost:8080](http://localhost:8080). The exact namespa

Alternatively, you can use a terminal UI such as [k9s](https://k9scli.io/) to select the service and start a port-forward interactively (press `<shift-f>` on a selected service or pod).

## Tutorials

Beyond the Vacation Planner samples, the repository includes standalone tutorials that exercise specific AKS capabilities. Unlike the samples above, they do not deploy the web app.

| Tutorial | Description |
| ------ | ----------- |
| [policies](policies/) | Kubernetes network policy tutorials that enforce zero-trust traffic control with [Calico](https://docs.tigera.io/calico/latest/about/) and [Cilium](https://docs.cilium.io/): cluster-wide default-deny, DNS-aware (FQDN) egress, and L3/L4/L7 ingress. |
| [ccm](ccm/scripts/) | Exercises the [Azure cloud controller manager](https://cloud-provider-azure.sigs.k8s.io/) load-balancer reconcile on the emulator: public and internal `Service` type `LoadBalancer`, `loadBalancerSourceRanges` NSG rules, the nodeIP backend-pool variant, and an NGINX ingress controller. |

## Tools

The following tools are useful when working with these samples:
Expand Down
83 changes: 83 additions & 0 deletions ccm/scripts/00-variables.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# Shared variables for the Cloud Controller Manager (CCM) sample tests.
#
# Source this file from every test script with: source ./00-variables.sh
#
# The AKS cluster is created by scripts/01-user-assigned-managed-identity.sh;
# these values MUST match that script (prefix "local", suffix "test", location "ItalyNorth") so the
# tests target the cluster it creates instead of standing up their own.

# Azure Kubernetes Service (AKS)
PREFIX="local"
SUFFIX="test"
AKS_NAME="${PREFIX}-aks-${SUFFIX}"
AKS_RESOURCE_GROUP_NAME="${PREFIX}-rg"
LOCATION="ItalyNorth"

# Node resource group (the MC_* resource group the AKS RP manages). The CCM writes the per-service
# public IP, the LoadBalancer frontend and rule, and the NSG allow-rules here, so the tests assert
# against it. Derived from the cluster so it is never hardcoded (empty when the cluster does not exist).
NODE_RESOURCE_GROUP=$(az aks show \
--name $AKS_NAME \
--resource-group $AKS_RESOURCE_GROUP_NAME \
--query nodeResourceGroup \
--output tsv \
--only-show-errors 2>/dev/null)

# The primary Standard load balancer the AKS RP pre-creates in the node resource group. The CCM adds
# each public LoadBalancer Service's frontend, rule, and backend pool to it. Real AKS and the upstream
# cloud-provider-azure Helm chart default the name (and the inbound backend pool) to "kubernetes".
PUBLIC_LOAD_BALANCER_NAME="kubernetes"
# Internal LoadBalancer Services land on a separate load balancer named "kubernetes-internal".
INTERNAL_LOAD_BALANCER_NAME="kubernetes-internal"

# Kubernetes namespace and the nginx workload the LoadBalancer Services select (tests 1 to 4).
NAMESPACE="ccm-test"
DEPLOYMENT_NAME="nginx"
APP_LABEL="nginx"
CONTAINER_IMAGE="nginx:1.27-alpine"
SERVICE_PORT=80

# One Service name per scenario, so the tests can coexist on the same cluster.
PUBLIC_SERVICE_NAME="nginx-public-lb"
INTERNAL_SERVICE_NAME="nginx-internal-lb"
RESTRICTED_SERVICE_NAME="nginx-restricted-lb"
NODE_IP_SERVICE_NAME="nginx-nodeip-lb"

# Annotation that turns a Service into an internal LoadBalancer (test 2). The frontend IP is then
# allocated privately from the cluster subnet instead of a public IP.
# https://learn.microsoft.com/en-us/azure/aks/internal-lb
INTERNAL_LB_ANNOTATION="service.beta.kubernetes.io/azure-load-balancer-internal"

# Client CIDR allowed to reach the restricted public Service (test 3). The CCM reconciles this into an
# inbound Allow rule on the node resource group NSG. 203.0.113.0/24 is the RFC 5737 TEST-NET-3 range.
ALLOWED_SOURCE_RANGE="203.0.113.0/24"

# NGINX ingress controller (test 5). Its own front Service is type LoadBalancer, so the CCM assigns it
# an EXTERNAL-IP the same way. Mirrors 01-user-assigned-managed-identity.sh's ingress install.
INGRESS_NAMESPACE="ingress-basic"
INGRESS_RELEASE_NAME="nginx-ingress"
INGRESS_REPO_NAME="ingress-nginx"
INGRESS_REPO_URL="https://kubernetes.github.io/ingress-nginx"
INGRESS_CHART_NAME="ingress-nginx"

# Backend workload behind the ingress (test 5 extension). A Deployment plus a ClusterIP Service, with
# an Ingress object routing to it, so a request that passes through the controller reaches a real
# backend. The backend serves a recognizable string (via a ConfigMap) so the pass-through is assertable.
BACKEND_DEPLOYMENT_NAME="ingress-backend"
BACKEND_SERVICE_NAME="ingress-backend"
BACKEND_APP_LABEL="ingress-backend"
BACKEND_CONFIG_MAP_NAME="ingress-backend-content"
BACKEND_RESPONSE_TEXT="Hello from the ingress backend"

# The Ingress routes every request (path "/", no host) to the backend Service, so no Host header is
# needed to reach it through the controller. It targets the "nginx" IngressClass the chart installs.
INGRESS_NAME="ingress-backend"
INGRESS_CLASS_NAME="nginx"

# Local port used by the kubectl port-forward pass-through check against the controller Service.
PORT_FORWARD_LOCAL_PORT=8080

# EXTERNAL-IP polling. The CCM writes the address only after the whole reconcile (frontend public IP,
# LB rule, backend-pool membership) completes, which can take a few minutes on a loaded emulator.
EXTERNAL_IP_TIMEOUT_SECONDS=300
SLEEP=5
137 changes: 137 additions & 0 deletions ccm/scripts/01-test-public-loadbalancer.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
#!/bin/bash

# Test 1: a public Service of type LoadBalancer receives an EXTERNAL-IP, and the Cloud Controller
# Manager creates a dedicated public IP for it in the node resource group plus a frontend and a rule
# on the "kubernetes" load balancer.
#
# The EXTERNAL-IP is a synthetic, non-routable placeholder (the emulated load balancer has no real
# dataplane): to actually reach the service, use kubectl port-forward.
# https://learn.microsoft.com/en-us/azure/aks/load-balancer-standard

# Variables
source ./00-variables.sh

# Make sure the AKS cluster exists (it is created by 01-user-assigned-managed-identity.sh)
if [[ -z $NODE_RESOURCE_GROUP ]]; then
echo "Could not resolve the node resource group for the [$AKS_NAME] AKS cluster"
echo "Create the cluster first with scripts/01-user-assigned-managed-identity.sh"
exit 1
fi

# Merge the cluster credentials into kubeconfig and set it as the current context
echo "Merging credentials for the [$AKS_NAME] AKS cluster into kubeconfig..."
az aks get-credentials \
--name $AKS_NAME \
--resource-group $AKS_RESOURCE_GROUP_NAME \
--overwrite-existing \
--only-show-errors

# Create the namespace if it does not already exist
RESULT=$(kubectl get namespace $NAMESPACE -o jsonpath='{.metadata.name}' 2>/dev/null)
if [[ -n $RESULT ]]; then
echo "The [$NAMESPACE] namespace already exists"
else
echo "Creating the [$NAMESPACE] namespace..."
kubectl create namespace $NAMESPACE
fi

# Deploy the nginx workload the Service selects (idempotent)
echo "Deploying the [$DEPLOYMENT_NAME] nginx deployment to the [$NAMESPACE] namespace..."
cat <<EOF | kubectl apply -n $NAMESPACE -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: $DEPLOYMENT_NAME
spec:
replicas: 1
selector:
matchLabels:
app: $APP_LABEL
template:
metadata:
labels:
app: $APP_LABEL
spec:
containers:
- name: nginx
image: $CONTAINER_IMAGE
ports:
- containerPort: $SERVICE_PORT
EOF

# Create the public LoadBalancer Service
echo "Creating the [$PUBLIC_SERVICE_NAME] public LoadBalancer service in the [$NAMESPACE] namespace..."
cat <<EOF | kubectl apply -n $NAMESPACE -f -
apiVersion: v1
kind: Service
metadata:
name: $PUBLIC_SERVICE_NAME
spec:
type: LoadBalancer
selector:
app: $APP_LABEL
ports:
- port: $SERVICE_PORT
targetPort: $SERVICE_PORT
protocol: TCP
EOF

# Wait for the Cloud Controller Manager to assign the EXTERNAL-IP
echo "Waiting up to [$EXTERNAL_IP_TIMEOUT_SECONDS] seconds for the [$PUBLIC_SERVICE_NAME] service EXTERNAL-IP..."
EXTERNAL_IP=""
DEADLINE=$((SECONDS + EXTERNAL_IP_TIMEOUT_SECONDS))
while [[ $SECONDS -lt $DEADLINE ]]; do
EXTERNAL_IP=$(kubectl get service $PUBLIC_SERVICE_NAME -n $NAMESPACE \
-o jsonpath='{.status.loadBalancer.ingress[0].ip}' 2>/dev/null)
if [[ -n $EXTERNAL_IP ]]; then
break
fi
sleep $SLEEP
done

if [[ -n $EXTERNAL_IP ]]; then
echo "The [$PUBLIC_SERVICE_NAME] service received EXTERNAL-IP [$EXTERNAL_IP]"
else
echo "The [$PUBLIC_SERVICE_NAME] service did not receive an EXTERNAL-IP within [$EXTERNAL_IP_TIMEOUT_SECONDS] seconds"
exit 1
fi

# The CCM creates a dedicated inbound public IP for the service in the node resource group
echo "Looking for a public IP with address [$EXTERNAL_IP] in the [$NODE_RESOURCE_GROUP] node resource group..."
PUBLIC_IP_NAME=$(az network public-ip list \
--resource-group $NODE_RESOURCE_GROUP \
--query "[?ipAddress=='$EXTERNAL_IP'].name | [0]" \
--output tsv \
--only-show-errors)

if [[ -n $PUBLIC_IP_NAME ]]; then
echo "Found public IP [$PUBLIC_IP_NAME] with address [$EXTERNAL_IP] in the [$NODE_RESOURCE_GROUP] node resource group"
else
echo "No public IP with address [$EXTERNAL_IP] found in the [$NODE_RESOURCE_GROUP] node resource group"
exit 1
fi

# The CCM adds a frontend IP configuration and a load-balancing rule to the "kubernetes" load balancer
echo "Verifying the [$PUBLIC_LOAD_BALANCER_NAME] load balancer frontend and rule in the [$NODE_RESOURCE_GROUP] node resource group..."
FRONTEND_COUNT=$(az network lb show \
--resource-group $NODE_RESOURCE_GROUP \
--name $PUBLIC_LOAD_BALANCER_NAME \
--query "length(frontendIPConfigurations || frontendIpConfigurations)" \
--output tsv \
--only-show-errors)
RULE_COUNT=$(az network lb rule list \
--resource-group $NODE_RESOURCE_GROUP \
--lb-name $PUBLIC_LOAD_BALANCER_NAME \
--query "length(@)" \
--output tsv \
--only-show-errors)

if [[ -n $FRONTEND_COUNT && $FRONTEND_COUNT -ge 1 && -n $RULE_COUNT && $RULE_COUNT -ge 1 ]]; then
echo "The [$PUBLIC_LOAD_BALANCER_NAME] load balancer has [$FRONTEND_COUNT] frontend(s) and [$RULE_COUNT] rule(s)"
else
echo "The [$PUBLIC_LOAD_BALANCER_NAME] load balancer is missing a frontend or a rule (frontends: [$FRONTEND_COUNT], rules: [$RULE_COUNT])"
exit 1
fi

echo "SUCCESS: the [$PUBLIC_SERVICE_NAME] public LoadBalancer service is backed by node resource group resources"
echo "The EXTERNAL-IP [$EXTERNAL_IP] is a synthetic placeholder; run 'kubectl port-forward -n $NAMESPACE svc/$PUBLIC_SERVICE_NAME 8080:$SERVICE_PORT' to reach nginx"
136 changes: 136 additions & 0 deletions ccm/scripts/02-test-internal-loadbalancer.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
#!/bin/bash

# Test 2: a Service annotated as an internal LoadBalancer receives a private EXTERNAL-IP allocated
# from the cluster subnet, materialised on the separate "kubernetes-internal" load balancer, and NO
# public IP is created for it in the node resource group.
# https://learn.microsoft.com/en-us/azure/aks/internal-lb

# Variables
source ./00-variables.sh

# Make sure the AKS cluster exists (it is created by 01-user-assigned-managed-identity.sh)
if [[ -z $NODE_RESOURCE_GROUP ]]; then
echo "Could not resolve the node resource group for the [$AKS_NAME] AKS cluster"
echo "Create the cluster first with scripts/01-user-assigned-managed-identity.sh"
exit 1
fi

# Merge the cluster credentials into kubeconfig and set it as the current context
echo "Merging credentials for the [$AKS_NAME] AKS cluster into kubeconfig..."
az aks get-credentials \
--name $AKS_NAME \
--resource-group $AKS_RESOURCE_GROUP_NAME \
--overwrite-existing \
--only-show-errors

# Create the namespace if it does not already exist
RESULT=$(kubectl get namespace $NAMESPACE -o jsonpath='{.metadata.name}' 2>/dev/null)
if [[ -n $RESULT ]]; then
echo "The [$NAMESPACE] namespace already exists"
else
echo "Creating the [$NAMESPACE] namespace..."
kubectl create namespace $NAMESPACE
fi

# Deploy the nginx workload the Service selects (idempotent)
echo "Deploying the [$DEPLOYMENT_NAME] nginx deployment to the [$NAMESPACE] namespace..."
cat <<EOF | kubectl apply -n $NAMESPACE -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: $DEPLOYMENT_NAME
spec:
replicas: 1
selector:
matchLabels:
app: $APP_LABEL
template:
metadata:
labels:
app: $APP_LABEL
spec:
containers:
- name: nginx
image: $CONTAINER_IMAGE
ports:
- containerPort: $SERVICE_PORT
EOF

# Create the internal LoadBalancer Service (the annotation switches the frontend to a private IP)
echo "Creating the [$INTERNAL_SERVICE_NAME] internal LoadBalancer service in the [$NAMESPACE] namespace..."
cat <<EOF | kubectl apply -n $NAMESPACE -f -
apiVersion: v1
kind: Service
metadata:
name: $INTERNAL_SERVICE_NAME
annotations:
$INTERNAL_LB_ANNOTATION: "true"
spec:
type: LoadBalancer
selector:
app: $APP_LABEL
ports:
- port: $SERVICE_PORT
targetPort: $SERVICE_PORT
protocol: TCP
EOF

# Wait for the Cloud Controller Manager to assign the private EXTERNAL-IP
echo "Waiting up to [$EXTERNAL_IP_TIMEOUT_SECONDS] seconds for the [$INTERNAL_SERVICE_NAME] service EXTERNAL-IP..."
EXTERNAL_IP=""
DEADLINE=$((SECONDS + EXTERNAL_IP_TIMEOUT_SECONDS))
while [[ $SECONDS -lt $DEADLINE ]]; do
EXTERNAL_IP=$(kubectl get service $INTERNAL_SERVICE_NAME -n $NAMESPACE \
-o jsonpath='{.status.loadBalancer.ingress[0].ip}' 2>/dev/null)
if [[ -n $EXTERNAL_IP ]]; then
break
fi
sleep $SLEEP
done

if [[ -n $EXTERNAL_IP ]]; then
echo "The [$INTERNAL_SERVICE_NAME] service received private EXTERNAL-IP [$EXTERNAL_IP]"
else
echo "The [$INTERNAL_SERVICE_NAME] service did not receive an EXTERNAL-IP within [$EXTERNAL_IP_TIMEOUT_SECONDS] seconds"
exit 1
fi

# The internal frontend lives on a separate load balancer named "kubernetes-internal"
echo "Verifying the [$INTERNAL_LOAD_BALANCER_NAME] load balancer exists in the [$NODE_RESOURCE_GROUP] node resource group..."
INTERNAL_LB_NAME_FOUND=$(az network lb show \
--resource-group $NODE_RESOURCE_GROUP \
--name $INTERNAL_LOAD_BALANCER_NAME \
--query name \
--output tsv \
--only-show-errors 2>/dev/null)

if [[ -n $INTERNAL_LB_NAME_FOUND ]]; then
echo "Found the [$INTERNAL_LB_NAME_FOUND] internal load balancer; its frontend private IPs are:"
az network lb show \
--resource-group $NODE_RESOURCE_GROUP \
--name $INTERNAL_LOAD_BALANCER_NAME \
--query "(frontendIPConfigurations || frontendIpConfigurations)[].privateIPAddress" \
--output tsv \
--only-show-errors
else
echo "The [$INTERNAL_LOAD_BALANCER_NAME] internal load balancer was not found in the [$NODE_RESOURCE_GROUP] node resource group"
exit 1
fi

# An internal LoadBalancer must NOT allocate a public IP: the EXTERNAL-IP is a private subnet address
echo "Confirming no public IP with address [$EXTERNAL_IP] exists in the [$NODE_RESOURCE_GROUP] node resource group..."
PUBLIC_IP_NAME=$(az network public-ip list \
--resource-group $NODE_RESOURCE_GROUP \
--query "[?ipAddress=='$EXTERNAL_IP'].name | [0]" \
--output tsv \
--only-show-errors)

if [[ -z $PUBLIC_IP_NAME ]]; then
echo "Confirmed: the [$INTERNAL_SERVICE_NAME] service is backed by a private frontend, not a public IP"
else
echo "Unexpected: a public IP [$PUBLIC_IP_NAME] with address [$EXTERNAL_IP] exists for an internal service"
exit 1
fi

echo "SUCCESS: the [$INTERNAL_SERVICE_NAME] internal LoadBalancer service received a private EXTERNAL-IP [$EXTERNAL_IP] on the [$INTERNAL_LOAD_BALANCER_NAME] load balancer"
echo "The EXTERNAL-IP [$EXTERNAL_IP] is a private IP address; run 'kubectl port-forward -n $NAMESPACE svc/$INTERNAL_SERVICE_NAME 8080:$SERVICE_PORT' to reach nginx"
Loading