Deploy Alloy with one script; split the agent config by where it can run (#88)

[Gerrrt]

What changed

• scripts/deploy-agent.sh (and make deploy-agent): one command that converges any host on this checkout's Alloy — image and version from compose.yaml, the alloy service's hardening written out as docker run, or the release .deb matching the same tag for a host that should not run Docker. Config shipped over ssh and checksummed; gate is ten seconds up, then a sixty-second window with no unexpected level=error, then Prometheus and Loki asked whether the host arrived.
• stacks/observability/alloy/ is now a directory Alloy loads as one config: config.alloy (every host), docker.alloy (hosts with a Docker socket), syslog.alloy (monitoring host only — the listener morpheus sends to). compose.yaml mounts the directory; validate, CI and check_dashboards.py read every file. Node exporter's rootfs prefix is ALLOY_ROOTFS (default unchanged).
• docs/runbooks/add-monitored-device.md rewritten around the script, both runtimes. Its previous docker run was still --privileged and its verification grep could never fail (Alloy logs to stderr).
• Docs reconciled: hardware.md three agents, guarded by a new check_docs.py claim against the architecture table; observability.md no longer hard-codes the pushed-job count; architecture.md diagram and host table show the agents; network.md, security.md, stack README, two other runbooks.
• ADR-0007 and ADR-0013 carry a NOTE (text left as written): Saruman's own host agent remote-writes to Winterfell over one unlogged pass above the ADR-0014 tripwire; guest telemetry still does not. ADR-0012 left untouched — its prediction came true as written.
• Roadmap: Deploy Alloy to Saruman and oracle #88 moved to Done, with what was actually found.

Why

Closes #88. The issue was stale on arrival: oracle has reported since 2026-08-30. What was actually wrong is in the commit message and the roadmap entry — oracle was a version behind, privileged, on a config with no self-scrape and no WAL volume, deployed from a runbook whose check passed for the wrong reason. Saruman needed a decision (ADR-0007 vs ADR-0012) more than a deploy.

Blast radius

• Monitoring host: the alloy service definition changed (directory mount, directory argument), so the next make up from the main checkout recreates it and reload-config.sh restarts it once more. Seconds of gap, WAL-backed. syslog.alloy must load or morpheus's logs stop — {host="morpheus"} is the check.

• oracle: container replaced (done, below). Gained alloy-config and alloy-data volumes; /opt/alloy/config.alloy left on disk, unused.

• Saruman: new package, new /etc/alloy, /etc/default/alloy, alloy user in systemd-journal and adm. Not yet run — see below.

• No change to network segmentation or firewall rules — one new pass is required and is the operator's to add: on the ImaginationLAN interface, TCP 10.0.30.110 → 10.0.99.20 ports 9090, 3100, not logged, above the ADR-0014 tripwire. The runbook has the table and the reason for the placement.

• No new port published to a VLAN that could not already reach the service

• No credential added outside secrets/*.sops.yaml

Verification

• make validate and make lint pass in the worktree. The new agent-count guard was shown to fail on the stale "two" and pass on "three".

• The config directory was formatted with alloy fmt -w and run for 20 s in a throwaway container from the pinned image: parsed, graph evaluated, only the expected socket/journal/textfile errors. (That run resolved prometheus through the host's DNS and pushed four instance="smoke" series into the live Prometheus; there is no admin API to delete them, so RemoteWriteJobStale{instance="smoke"} is silenced for 25 h and the ship-firewall-logs runbook now says to point the URLs at 127.0.0.1:1 for such runs.)

• oracle redeployed from this worktree with ./scripts/deploy-agent.sh atropos@10.0.99.30:

  ✓ alloy running: grafana/alloy:v1.19.2@sha256:b8ec653c… privileged=false cgroupns=host caps=[ALL]
  ✓ no errors in the last 60s
  ✓ Prometheus has 3 job(s) for instance="oracle"
  ✓ Loki has host="oracle"
  

  Confirmed from Prometheus: oracle-metrics, oracle-alloy, integrations/cadvisor all up, node_textfile_scrape_error 0, three containers seen, alloy_build_info v1.19.2. docker inspect: CapDrop=[ALL] GroupAdd=[4 473] Cgroup=host Pids=1024 Init=true, no /var/lib/docker/containers mount, stage directory cleaned.

• Found on the way, not fixed here: every Docker-host agent has logged cAdvisor's rootDiskErr every few minutes since Reduce Alloy's privilege — it is root on the host and can read the age key #188 (the monitoring host: 83 in six hours). Partial — container_fs_usage_bytes still exists — and nothing charted depends on that walk. The gate ignores exactly that line and node_exporter's one-time udev line, and prints anything else.

After merge

1. Main checkout on the monitoring host: git pull && make up, then confirm up{job="alloy"}, up{job="prometheus-alloy"} and {host="morpheus"} still flowing.
2. Add the pfSense rule above.
3. From the Mac: ./scripts/deploy-agent.sh --runtime native root@10.0.30.110. The label will be whatever hostname prints there (probably saruman); if the docs' Saruman is wrong, fix the docs. Then check {app="filterlog", interface=~"vlan30.*"} shows no pushes logged on the tripwire.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

The new deploy script’s Prometheus arrival-check loop can terminate early under set -euo pipefail when the query initially returns no matching jobs, preventing the intended retry behavior.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR standardizes deployment of the Grafana Alloy agent across hosts by introducing a single convergent deployment script, and by splitting the Alloy config into multiple files that are loaded as a directory depending on where they can run (base vs Docker socket vs monitoring-host-only syslog listener). It also updates CI/validation and docs to match the new deployment and configuration model.

Changes:

• Added scripts/deploy-agent.sh (+ make deploy-agent) to deploy/redeploy Alloy on remote hosts using either a Docker container (compose-equivalent hardening) or the native .deb matching the pinned compose version.
• Converted stacks/observability/alloy/ into a multi-file directory config (config.alloy, docker.alloy, syslog.alloy) and updated compose + CI/validation tooling to format-check all *.alloy files.
• Updated runbooks and documentation consistency checks (including a new doc guard for the “Alloy agents” count).

File summaries

File	Description
stacks/observability/README.md	Updates stack file map to describe Alloy as a directory config with per-environment files.
stacks/observability/prometheus/rules/stack.rules.yaml	Clarifies alert-rule commentary around job counting and historical agent job shape.
stacks/observability/compose.yaml	Switches Alloy to mount /etc/alloy as a directory and run Alloy against the directory path.
stacks/observability/alloy/syslog.alloy	Adds monitoring-host-only network syslog receiver pipeline and relabeling.
stacks/observability/alloy/docker.alloy	Adds Docker-socket-dependent metrics/log collection components (cAdvisor + docker logs).
stacks/observability/alloy/config.alloy	Refactors “common” agent config to be directory-loaded and adds ALLOY_ROOTFS support for native installs.
scripts/validate.sh	Extends local validation to run alloy fmt --test over all *.alloy files.
scripts/deploy-agent.sh	Adds new convergent remote deploy script for Docker/native runtimes with gating and arrival verification.
scripts/check_docs.py	Adds doc assertion for Alloy agent count derived from the architecture table.
scripts/check_dashboards.py	Updates level-vocabulary extraction to read all alloy/*.alloy files rather than a single file.
README.md	Updates repository tree description to reflect Alloy as a directory config.
Makefile	Adds deploy-agent target to invoke the new deployment script.
docs/security.md	Updates security posture description to include deploy script parity and native runtime behavior.
docs/runbooks/ship-firewall-logs.md	Updates syslog listener documentation to reflect syslog.alloy and directory-loading behavior.
docs/runbooks/schedule-maintenance.md	Updates troubleshooting note to reference ALLOY_ROOTFS for textfile collector path correctness.
docs/runbooks/add-monitored-device.md	Rewrites host onboarding around deploy-agent.sh (Docker and native package paths).
docs/roadmap.md	Moves #88 to Done and documents what was actually fixed/found.
docs/observability.md	Updates observability doc to reflect current pushed job counts and remote_write behavior.
docs/network.md	Documents Saruman’s limited cross-segment pass for its host telemetry.
docs/hardware.md	Updates hardware capacity statement to reflect three Alloy agents.
docs/architecture.md	Updates architecture diagram and host table to show agents and syslog flow.
docs/adr/0013-segment-access-as-implemented.md	Adds NOTE about the new cross-segment rule for Saruman’s agent.
docs/adr/0007-defensive-estate-and-offensive-range.md	Adds NOTE amending telemetry decision for Saruman’s host agent only.
.github/workflows/ci.yml	Updates CI to run alloy fmt --test across all *.alloy files.

Review details

• Files reviewed: 24/24 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.