A PowerShell, cloud-init, and Ansible IaC pipeline that installs, configures, and monitors a headless Linux node from a single command — no installer prompts, no hypervisor wizard.
Infrastructure as Code across four layers, each owned by the tool that should own it.
%%{init: {'themeVariables': { 'background': '#ffffff'}}}%%
flowchart TD
classDef prov fill:#e6f3ff,stroke:#0066cc,stroke-width:2px,color:#003366,rx:5px,ry:5px;
classDef conf fill:#e6ffe6,stroke:#009933,stroke-width:2px,color:#004d1a,rx:5px,ry:5px;
classDef obs fill:#fff4e6,stroke:#cc6600,stroke-width:2px,color:#663300,rx:5px,ry:5px;
style Provisioning fill:#ffffff,stroke:#dee2e6,stroke-width:2px,stroke-dasharray: 5 5,color:#333
style Configuration fill:#ffffff,stroke:#dee2e6,stroke-width:2px,stroke-dasharray: 5 5,color:#333
style Observability fill:#ffffff,stroke:#dee2e6,stroke-width:2px,stroke-dasharray: 5 5,color:#333
subgraph Provisioning [Provision & Install -- PowerShell, cloud-init]
direction LR
A[01: Host Prep]:::prov --> B[02: VM Provisioning]:::prov
B --> C[03: OS Installation]:::prov
C --> D[04: Boot & Await SSH]:::prov
end
subgraph Configuration [Configure -- Ansible]
direction LR
E[docker role]:::conf --> F[monitoring role]:::conf
F --> G[tailscale role]:::conf
end
subgraph Observability [Observe -- left running on the node]
direction LR
H[node_exporter]:::obs --> I["Prometheus<br/>5 alert rules"]:::obs
I --> J[Grafana dashboard]:::obs
I --> K[Alertmanager]:::obs
K --> L["Discord<br/>+ runbook link"]:::obs
end
D --> E
F --> H
Provisioning is Windows-specific by design — VirtualBox on a Windows host. Configuration is not: the Ansible roles target any Debian-family host.
| Problem | Why it matters |
|---|---|
| ⌨️ Interactive OS install | The Ubuntu installer prompts for disk, bootloader, and credentials before networking exists. |
| 🖱️ VM defined through a GUI | Specs set in the VirtualBox wizard are not versioned, reviewable, or repeatable. |
| 🔓 Inbound access | Reaching SSH from outside the LAN requires a router port-forward to the VM. |
| 📉 No metric history | VirtualBox reports point-in-time figures, readable only at the host's screen. |
After the pipeline finishes, this is what is already running.
Datasource and dashboard are provisioned from this repository, never clicked in. Panels mirror the alert rules, so what is watched and what pages you are the same signals.
node_exporter stopped on purpose. TargetDown waited out its for: 2m grace period, then fired with the labels defined in alert_rules.yml. The other four rules stayed quiet.
Alertmanager delivers the alert, then the resolved notice once the container returns. The runbook link rides on the notification itself.
📦 ztp-linux-node/
│
├── ⚙️ Deploy-Node.ps1 # Master execution entrypoint
│
├── 📁 config/ # Single source of truth: VM name, hardware sizing, ISO URL
│
├── 📁 scripts/ # 01-04: provision the machine (PowerShell)
│ ├── 📁 cloud-init/ # Unattended Ubuntu autoinstall configuration
│ └── 📁 tests/ # Pester tests for Get-LabConfig
│
├── 📁 ansible/ # Configure the running node (docker, tailscale, monitoring)
│
├── 📁 monitoring/ # Observability configuration
│ ├── 🐳 docker-compose.yml # Prometheus & Grafana stack
│ └── 📁 tests/ # promtool unit tests for the alert rules
│
├── 📁 docs/ # Changelog, troubleshooting, roadmap, Ansible setup
│
└── 📁 logs/ # Per-stage execution transcripts (gitignored)
| Decision | Why |
|---|---|
| Provisioning ≠ configuration | PowerShell drives VBoxManage, diskpart and bcdedit. Ansible owns desired state. Stage 04 runs VBoxManage, so it stays PowerShell |
| Idempotency is proved, not claimed | A second run must report changed=0. The Grafana password persists on the control node so it cannot rotate and break that |
| Secrets never enter git | The SSH key renders into a throwaway user-data. Grafana and Tailscale credentials stay on the control node, gitignored |
| Every run recovers | Stale media registrations and known_hosts pins cleared on rebuild. Mounted VHDs released in a finally |
| No silent success | $LASTEXITCODE checked after every native call. The install loop fails on a deadline instead of hanging |
| NAT over bridged | Bridged Wi-Fi stalled Docker pulls at ~50% after an hour. NAT + virtio measured ~210-290 Mbps |
Each of these came from a failure. The ones with a root-cause writeup are in TROUBLESHOOTING.md.
Important
Required — an Ansible control node on WSL 1. One-time setup: ANSIBLE-SETUP.md. Miss it and the pipeline provisions the VM, then stops and says so.
Both optional, both gitignored, both written once:
| Optional | Write it | Skip it |
|---|---|---|
| Tailscale key — unattended tailnet enrolment Settings → Keys, reusable |
echo 'tskey-auth-...' > ansible/.tailscale_auth_key |
Approve the device in a browser once |
| Discord webhook — alerts reach you, not a web page Text channel → ⚙️ → Integrations → Webhooks |
echo 'https://discord.com/api/webhooks/...' > ansible/.discord_webhook_url |
Alerts still group and silence, they just go nowhere |
VirtualBox 7+ and the ISO are handled by stage 01. Then, as Administrator:
.\Deploy-Node.ps1Prompts once for the node's sudo password. Re-running the playbook alone should report changed=0:
cd ansible && ANSIBLE_CONFIG=$PWD/ansible.cfg ansible-playbook site.yml -K| Service | Endpoint |
|---|---|
| Grafana | http://localhost:3000 — user admin, dashboard already provisioned |
| Prometheus | http://localhost:9090 — /targets for scrape health, /alerts for rule state |
| Alertmanager | http://localhost:9093 — grouped alerts and silences |
| SSH | ssh -p 2222 sysadmin@127.0.0.1 |
The Grafana password is generated once, then reused. It stays on your machine, gitignored:
cat ansible/.grafana_admin_password| Metric | Manual Provisioning | Automated Pipeline |
|---|---|---|
| Time to provision | 20-30 min, interactive (est.) | 3 min 53 s to a booted OS, 7 min 35 s end-to-end (single run, 2 vCPU / 4 GB) |
| Reproducibility | Undocumented manual steps | Single command against config/node.json |
| Remote access | Router port forwarding | Tailscale mesh VPN, zero inbound ports |
| Host visibility | No history, host screen only | node_exporter scraped every 15s, viewable from any device on the tailnet |
- CHANGELOG.md - Version history and bug fixes.
- TROUBLESHOOTING.md - Detailed root-cause analysis for advanced edge cases.
- RUNBOOK.md - One section per alert: what it means, how to confirm it, what to do.
- FUTURE-ROADMAP.md - Tracker: what is built, what is next, what is deliberately not being built.
- ANSIBLE-SETUP.md - Control-node prerequisites on Windows, and why WSL 1 rather than WSL 2.
Seven gates on every push and PR, in five parallel jobs. All must pass.
| Gate | What it checks |
|---|---|
| Lint | Invoke-ScriptAnalyzer over every script, and that config/node.json has every key the stages read |
| Unit tests | Pester against Get-LabConfig, the one script with logic rather than side effects |
| Compose | docker compose config against .env.example |
| Ansible | --syntax-check and ansible-lint over the playbook and all three roles |
| Secret scan | gitleaks across the full history, not just the tip |
| Observability config | promtool and amtool over the Prometheus and Alertmanager configs, cloud-init, the dashboard, and the runbook links |
| Alert behaviour | Synthetic series driven through the five rules: each must fire when it should, and stay quiet when it should not |
Dependabot proposes updates monthly, for Actions and the pinned images.
Worth running before you push:
Invoke-ScriptAnalyzer -Path . -Recurse -Severity Error,Warning -ExcludeRule PSUseBOMForUnicodeEncodedFile
Invoke-Pester -Path scripts/testsdocker run --rm -v "$PWD/monitoring:/etc/prometheus:ro" --entrypoint promtool \
prom/prometheus:v3.13.2 test rules /etc/prometheus/tests/alert_rules_test.ymlThe rest are one-liners in ci.yml, which is the copy that matters.
CI never builds a VM or connects to a node. A green check means the config is valid and the rules behave — not that the pipeline provisions anything.
Important
The real test is running site.yml twice. The second run must report changed=0.