Build stacks/lab/ — Prometheus, Loki, Grafana, Alloy

[Gerrrt]

Split out of #101. Composition
settled by ADR-0019:
Prometheus, Loki, Grafana, Alloy.

ADR-0007 named three services and, in the same sentence, said config.alloy is
reused unchanged with "the two *_URL variables the only difference". That file
has two sinks — loki.write "grafana_loki" and prometheus.remote_write "metrics_service", lines 386 and 395. With no lab Prometheus the second has
nowhere to point but 10.0.99.20, which the Decision forbids. The three-item
list was an omission.

Blocked behind the guest to run it on, and behind the validators, both of which
are separate issues.

What it is

• stacks/lab/compose.yaml, name: lab, following
  stacks/observability/compose.yaml's shape — the shared x-service-defaults,
  the pinned-plus-digested images, cap_drop: [ALL] and cgroup: host on
  Alloy, the reasoning comments that make the settings survivable.
• stacks/lab/alloy/ — config.alloy and docker.alloy, not
  syslog.alloy: that listener belongs to the monitoring host alone.
• secrets/lab.example.yaml and secrets/lab.sops.yaml.
• stacks/lab/.env.example, stacks/lab/README.md.
• Rows in docs/architecture.md and docs/observability.md.

The traps, in the order they will be hit

• .sops.yaml needs a second creation_rule, and the tooling walks you past
  it. The one rule there matches secrets/.*\.sops\.ya?ml$, so a recipient
  added to it decrypts every file under secrets/ — the estate's SNMP
  communities and Grafana admin password included. make secrets-init STACK=lab on the lab guest finds a different recipient already present and
  prints "Add this key as an additional recipient by hand"
  (scripts/bootstrap.sh:61-63), which is right for a second estate host and
  exactly wrong here. A lab host that can decrypt the estate's credentials
  inverts the trust direction ADR-0007 exists to protect. Wanted: a rule scoped
  to the lab file, ordered before the general one, with only the guest's key.
• render-config.sh's REQUIRED array is not stack-aware. It demands all
  four SNMP communities and all four Alertmanager URLs from any stack, and the
  lab has neither an snmp-exporter nor an Alertmanager. The renders themselves
  are already guarded by if [[ -f ... ]]; the required-key list is not.
• Certificates. Grafana serves https from a leaf issued by the lab CA, and
  render-config.sh refuses to render without it. A second Grafana means a
  second leaf — from the same CA, not a second one.
• Retention has to be sized down. The estate budgets 12 GiB of Prometheus
  and 30 days of Loki on a host that does nothing else. This one shares a
  mirrored pair of 7.2K disks with a Windows domain, Wazuh, Velociraptor and
  PBS. ADR-0007: "the fleet is sized against spindles, not RAM." Do not start
  from the estate's numbers.
• Dashboards. The estate's seven are built on the estate's datasources and
  metrics. What the lab needs is not a copy, and a copy is what will happen if
  nobody decides otherwise first.

Done when

make up STACK=lab on the guest brings up four services, make validate
checks them, and the lab's Grafana is reachable from Hicks over the existing
50 → 30 rule with no new firewall rule anywhere.

[Gerrrt]

Built in #276 — all three CI jobs green. Not deployed: the guest is still #262, and VLAN 30 is not reachable from the monitoring host.

Four services on alexander (10.0.30.40), a guest on Saruman. The agent config is mounted from stacks/observability/alloy/ rather than copied — two files, so syslog.alloy stays on the monitoring host.

All four traps this issue listed were real and are handled:

• .sops.yaml has the lab rule, above the catch-all — and bootstrap.sh now refuses when the key it would write is already a recipient of another rule, which is what make secrets-init STACK=lab on the monitoring host would have done.
• render-config.sh derives its required keys per stack. The estate's ten are byte-identical to before, verified by running the real code path against both stacks rather than by reading it.
• Certificates: a second leaf from the same CA, with the grafana DNS SAN the scrape job verifies against.
• Retention is 15d/4GB and labelled a bound rather than a measurement, with the re-derivation queries in compose.yaml.

Two more found on the way and fixed in the same PR: reload-config.sh died on services a stack does not declare, and backup-firewall.sh would have encrypted every firewall export to the lab guest the moment the lab rule got a real key.

No dashboards, deliberately — stacks/lab/grafana/dashboards/README.md says why a copy of the estate's seven would be four rows of empty panels.

Still open and untouched: #263 (no validator runs against this stack), and converge.sh runs a bare make up, so nothing deploys it either — both written down in the stack README rather than left to be discovered.