SRE Agent Azure Lab

Overview

SRE Agent Azure Lab is a Terraform-first Azure sandbox for practicing native incident detection and guided remediation. It provisions Windows VM incident targets plus optional modern app-platform targets: Azure Kubernetes Service, Azure App Service, Azure Container Apps, and Azure Functions.

It uses Azure Monitor Agent, Data Collection Rules, Log Analytics, KQL alerts, Workbooks, Azure Automation runbooks, Update Manager, Policy, Resource Health alerts, cost controls, and an optional portal-visible Azure SRE Agent (Microsoft.App/agents).

The lab is intentionally Azure-native: Terraform provisions the platform, Azure Monitor detects signals, and Automation runbooks perform safe lab remediation. No third-party monitoring stack is required in v1.

Quality Review Path

Step	Command or file	What it proves
Lowest-cost profile	environments/cheap-lab.tfvars	One jumpbox, one IIS target, native monitoring, SRE runbooks, optional portal Azure SRE Agent, no premium network services
Quality gate	.\scripts\Invoke-QualityGate.ps1	Formatting, validation, script parsing, compile checks, reserved wording, and no-refresh plans
No-apply plan	.\scripts\Invoke-LocalPlan.ps1 -VarFile environments/lab.tfvars	Azure graph can be planned before creating resources
Script validation	.\scripts\Invoke-SreLabValidation.ps1	Resource groups, telemetry, alerts, AMA, runbooks, and optionally the portal Azure SRE Agent exist after deployment
Incident exercises	.\scripts\Invoke-SreIncident.ps1 -Scenario IisOutage	Lab can generate signals for alert and remediation practice
Readiness review	Operational readiness	Security, reliability, cost, governance, and operations review points

Architecture

The deployment creates five main resource-group roles:

Role	Purpose
network	Hub, management, and workload VNets, subnets, NSGs, peering, optional Firewall/VPN
windows	Jumpbox and Windows IIS incident targets
apps	Optional AKS, App Service, Container Apps, and Functions targets
sre	Log Analytics, DCRs, alerts, Workbooks, dashboard, Automation runbooks, optional Backup
governance	Optional Azure Policy assignments and guardrail resources
sreagent	Optional portal-visible Azure SRE Agent, managed identity, and agent telemetry

Native SRE Flow

1. Windows VMs run Azure Monitor Agent.
2. Data Collection Rules send performance counters, Windows events, heartbeat, IIS, and disk signals to Log Analytics.
3. Metric and KQL alerts detect CPU, VM availability, missing heartbeat, IIS outage, disk pressure, and critical Windows events.
4. Action Groups notify operators.
5. Azure Automation runbooks can restart IIS, start stopped VMs, collect diagnostics, or clean lab-safe temporary files.
6. Direct alert-to-runbook webhooks stay off by default and require enable_alert_runbook_webhooks = true.
7. When deploy_azure_sre_agent = true, a real Azure SRE Agent can observe the lab resource groups from sre.azure.com.

Deployment Profiles

Profile	File	Best for	Cost posture
cheap-lab	environments/cheap-lab.tfvars	First review and low-cost demos	Lowest
dev	environments/dev.tfvars	Terraform and module testing	Very low
lab	environments/lab.tfvars	Normal SRE incident lab with AKS, App Service, Container Apps, and Functions	Moderate
full	environments/full.tfvars	Expanded demo with app platform, backup, and extra targets	Highest

Start with cheap-lab unless you specifically need AKS, App Service, Container Apps, Functions, backup, Firewall, VPN, or extra Windows targets.

Quick Start

Prerequisites:

• Azure CLI logged in with az login
• Terraform 1.9 or later
• Azure subscription selected with az account set --subscription <id> or ARM_SUBSCRIPTION_ID
• A private VM password supplied through TF_VAR_admin_password or an ignored terraform.tfvars

terraform init -backend=false -reconfigure
terraform validate
.\scripts\Invoke-LocalPlan.ps1 -VarFile environments/cheap-lab.tfvars

For a real apply:

terraform init
terraform apply -var-file=environments/cheap-lab.tfvars
.\scripts\Invoke-SreLabValidation.ps1 -Environment cheap-lab

Master Control Panel

The main feature switches live in terraform.tfvars or the committed environment profiles.

deploy_monitoring             = true
deploy_log_analytics          = true
deploy_azure_monitor_agent    = true
deploy_data_collection_rules  = true
deploy_alerts                 = true
deploy_log_query_alerts       = true
deploy_workbooks              = true
deploy_sre_agent              = true
deploy_azure_sre_agent        = false
enable_alert_runbook_webhooks = false
deploy_update_management      = true
deploy_backup                 = false
deploy_policy                 = true
deploy_cost_management        = true
deploy_windows_targets        = true
deploy_iis_farm               = true
deploy_firewall               = false
deploy_vpn_gateway            = false
deploy_aks                    = true
deploy_app_service            = true
deploy_container_apps         = true
deploy_functions              = true

Lab Scenarios

Scenario	What you validate
VM health monitoring	Heartbeat, VM availability, CPU, memory, disk, and Windows event visibility
IIS outage	Stop W3SVC, detect service-control events, then restart IIS through a runbook
High CPU	Generate CPU pressure and confirm metric alert behavior
Disk pressure	Create a temporary lab load file and confirm disk-free-space alert behavior
Stopped VM	Stop a VM and validate VM availability signal plus runbook recovery
Diagnostics collection	Use Collect-VMDiagnostics to capture service, event, volume, and process snapshots
Update governance	Use Update Manager maintenance configs and PatchGroup tags
Cost guardrail	Track spend with an Azure budget on the SRE resource group
Modern app platform	Deploy AKS, App Service, Container Apps, and Functions as additional Resource Health and operations targets

Testing

terraform fmt -check -recursive
terraform init -backend=false -reconfigure
terraform validate
.\scripts\Invoke-LocalPlan.ps1 -VarFile environments/cheap-lab.tfvars
.\scripts\Invoke-QualityGate.ps1 -SkipPlans
Get-ChildItem scripts -Filter *.ps1 | ForEach-Object {
  $null = [System.Management.Automation.Language.Parser]::ParseFile($_.FullName, [ref]$null, [ref]$null)
}

Terratest smoke tests live in tests/ and skip live Azure checks when ARM_SUBSCRIPTION_ID is not set.

Cost And Safety Notes

• Keep enable_alert_runbook_webhooks = false until you intentionally want alerts to invoke runbooks.
• Keep deploy_azure_sre_agent = false unless you want a real Azure SRE Agent; it starts Azure SRE Agent billing when created.
• Keep allowed_rdp_source_ips = [] unless you have a trusted CIDR.
• Use cheap-lab first; lab enables modern app-platform services, and full adds more VMs and Backup.
• AKS nodes, App Service plans, Firewall, VPN Gateway, Backup storage, and Managed Grafana can materially increase cost.
• Destroy temporary deployments when finished.

Documentation Map

• Wiki home
• Architecture overview
• Monitoring and dashboards
• Update management
• Operational readiness
• Variables reference
• Outputs reference
• Pipeline