Operations Runbooks

Operations Runbooks

This page documents repeatable operational procedures for the lab.

Run A Plan

GitHub UI:

1. Go to Terraform Pipeline.
2. Select Run workflow.
3. Choose environment.
4. Choose action = plan.
5. Run and review the plan summary.

GitHub CLI:

gh workflow run "Terraform Pipeline" --repo Jamonygr/azure-landing-zone-lab -f environment=cheap-lab -f action=plan

Review:

• Add/change/destroy counts.
• Security scan results.
• OPA policy results.
• Cost estimate.
• Plan artifact.

Apply A Plan

Only apply after plan review:

gh workflow run "Terraform Pipeline" --repo Jamonygr/azure-landing-zone-lab -f environment=cheap-lab -f action=apply

Expected pipeline behavior:

• Reuses plan job output.
• Runs state backup before apply.
• Applies saved plan artifact.
• Generates resource inventory.
• Writes changelog and metrics.

Source:

• .github/actions/apply
• .github/actions/state-backup
• .github/actions/resource-inventory
• .github/actions/changelog
• .github/actions/metrics

Destroy An Environment

Destroy is intentionally guarded:

gh workflow run "Terraform Pipeline" --repo Jamonygr/azure-landing-zone-lab -f environment=cheap-lab -f action=destroy -f destroy_confirm=DESTROY

Before destroy:

• Confirm this is the correct environment.
• Confirm state key is correct.
• Confirm no shared resources are being reused.
• Confirm state backup succeeded.
• Confirm GitHub environment approval is required.

Source:

• .github/actions/destroy

Rotate GitHub OIDC Identity

1. Create or update the Microsoft Entra application registration.
2. Add federated credential for this repo and branch/environment pattern.
3. Update GitHub secrets:
   • AZURE_CLIENT_ID
   • AZURE_TENANT_ID
   • AZURE_SUBSCRIPTION_ID
4. Run plan for cheap-lab.
5. Confirm Azure login succeeds.

See CI/CD Pipeline and Remote State and Secrets.

Add A New Environment Profile

1. Copy an existing profile under environments.
2. Set environment, location, owner, and toggles.
3. Add the profile to workflow_dispatch.inputs.environment.options in terraform.yml.
4. Add GitHub environment protection.
5. Run format and validate.
6. Run plan.
7. Add documentation to Deployment Profiles.

Enable An Expensive Service Safely

For Firewall, VPN Gateway, Application Gateway, NAT Gateway, or AKS:

1. Start from a working plan.
2. Enable one toggle.
3. Run plan.
4. Review cost estimate.
5. Review public IP changes.
6. Apply only if needed.
7. Destroy or disable after the lab exercise.

Troubleshoot A Failed Plan

1. Read the failed job summary.
2. Check whether failure is format, validate, scan, policy, backend, or Azure authorization.
3. If backend failed, verify TF_STATE_RG and TF_STATE_SA.
4. If Azure login failed, verify OIDC federated credential subject and repo.
5. If OPA failed, inspect the generated plan JSON and policy message.
6. If Checkov/tfsec failed, confirm whether the finding is intentional or requires code change.

See Troubleshooting.

Failed Deployment Recovery

When apply fails:

1. Stop additional apply attempts until the failure is understood.
2. Save the workflow run URL and failed job logs.
3. Check whether the failure happened before or after resource creation started.
4. Run a fresh plan against the same profile.
5. Compare planned changes to the failed apply.
6. Fix Terraform, permissions, policy, or Azure quota issue.
7. Apply only after the new plan is reviewed.

Do not manually delete Azure resources unless the Terraform state impact is understood.

Public Exposure Incident

Use this when an unexpected public IP, RDP rule, public SQL setting, or public storage access is found:

1. Treat the issue as active until proven otherwise.
2. Disable the exposure through Terraform or Azure emergency action.
3. Preserve evidence: plan, Azure setting, timestamp, resource name.
4. Rotate credentials if there is any chance of exposure.
5. Add or update a regression check.
6. Update Security Operations with lessons learned if the process changed.

Weekly Cost Review

Run weekly while resources exist:

az consumption usage list --top 20 -o table

Review:

• Firewall, VPN Gateway, App Gateway, NAT Gateway, AKS, SQL, and VM spend.
• Any resource group that should have been destroyed.
• Cost alert recipients and thresholds.
• Whether the lab should be destroyed or moved to cheap-lab.

Evidence Capture Runbook

For every completed apply:

1. Save workflow plan and apply URLs.
2. Export resource group, VNet, public IP, private endpoint, and diagnostic setting summaries.
3. Capture sanitized portal screenshots only where they add clarity.
4. Attach evidence to the relevant GitHub issue or PR.
5. Update wiki pages if observed behavior differs from documentation.

Update The Architecture Diagram

Primary diagram path in repo:

• docs/images/azure_architecture_with_icons_v2.png

Wiki image reference:

![Actual Azure Architecture Plan](images/azure_architecture_with_icons_v2.png)

The Wiki now stores its own copy under images/azure_architecture_with_icons_v2.png, so update the Wiki image when the repository diagram changes.

Next page: Troubleshooting