ADR 0003 detection and orchestration

ADR-0003: Detection & orchestration strategy

Status: Accepted (2026-07-22)

Context

The source design orchestrated the review with a supervisor LLM delegating to sub-agents, and had an "Infrascan" agent map live telemetry by reasoning over raw AWS config. Two problems with that for this workload:

1. The flow is fixed and fully specifiable. A nightly WA review is always the same sequence — hydrate → scan (live + code) → reconcile → score → write back → report. There is no open-ended decision for a model to make at the orchestration level, and this is a governance tool that files High-Risk Issues and mutates the AWS Well-Architected Tool, so determinism, reproducibility, and auditability are hard requirements.
2. Detection should not be an LLM's job. Mature, deterministic scanners already implement hundreds of maintained checks and emit structured, evidence-backed findings. Having an LLM "notice" misconfigurations is slower, more expensive, non-reproducible, and hallucination-prone — and reinvents tools like Prowler badly.

Decision drivers

• Reproducibility & auditable evidence for every finding filed.
• Per-stage least-privilege (the read-only scan must not share context or identity with the WA-Tool write-back).
• Cost and testability (structured, pure-function stages).
• Reserve the LLM for what only it can do: cross-source reasoning and language.

Decision

Orchestration is a deterministic, code-orchestrated pipeline; detection is a deterministic scanner portfolio; the LLM does bounded reconciliation.

1. Orchestration — deterministic. The buildspec / driver owns the fixed phase flow. There is no supervisor LLM deciding the flow. The three original roles survive as pipeline stages, not as an agent society. The "Supervisor" is demoted to a deterministic orchestrator.

2. Detection — deterministic scanner portfolio (per-pillar tool mapping in ADR-0004), run symmetrically on both sides:

   • Live account → Prowler + Steampipe/Powerpipe (WA/Thrifty mods) + AWS Compute Optimizer — actual/runtime state.
   • Terraform code → Checkov + Infracost — declared/intended state. Each emits structured findings, normalised to a common finding schema.

3. Reconciliation — bounded LLM. Claude consumes the structured findings (not raw config) and does the work scanners cannot: correlate live vs code, classify drift, map heterogeneous findings into WA pillar language, dedupe, prioritise, and draft remediation — plus reason over the residual questionnaire items no scanner can answer. This stage is bounded (structured JSON-schema output, scoped tools, capped iterations), not an open-ended agent loop.

4. Drift classification is the core LLM verdict:

   • code finding and matching live finding → code-origin HRI (fix in code)
   • live finding without code cause → out-of-band drift (live ≠ code)
   • code finding not present live → not-yet-applied (advisory)

Consequences

• Reproducible & auditable — same flow every night; each HRI carries the scanner finding (check ID + evidence) that produced it.
• Per-stage least-privilege — the scan stages run under broad read-only identity; only the write-back stage holds UpdateAnswer (see the IAM ADR).
• Lower token cost — the LLM reads compact structured findings, not raw tfstate/HCL/config dumps; no supervisor overhead or inter-agent chatter.
• Testable — every stage is a structured-output function with fixed inputs.
• Determinism-vs-adaptiveness trade-off — the pipeline will not autonomously decide to dig deeper across pillars. Accepted: within-stage reasoning is still adaptive; cross-pillar depth is a revisit trigger.
• Strands' role shrinks — it is available for a bounded within-stage agent if reconciliation ever needs a tool-use loop, but it does not orchestrate.
• Detection tooling must be installed and version-pinned in the build image (ADR-0001), and Prowler/Steampipe require broad read-only IAM (IAM ADR).

Alternatives considered

Option	Verdict	Why
Deterministic pipeline + scanner portfolio + bounded LLM reconciliation	Chosen	Reproducible/auditable, per-stage least-privilege, cheap, testable; LLM used only for irreducible reasoning.
Supervisor multi-agent (LLM orchestrates)	Rejected (now)	Non-deterministic flow undermines the audit story for a tool that files HRIs; highest cost. Revisit if dynamic cross-pillar depth is needed.
Single agent, all tools	Rejected	One context + one over-broad IAM role; prompt-injection in HCL would sit in the same context that can write to the WA Tool.
LLM-driven detection (no scanners)	Rejected	Non-reproducible, hallucination-prone, slower, and reinvents mature maintained tools.
Pure single-shot LLM calls (no within-stage loops at all)	Viable variant	Even more locked-down; adopt if within-stage tool loops prove unnecessary.

Revisit trigger

• Introduce LLM-driven delegation (supervisor pattern) only if reviews need autonomous, dynamic cross-pillar depth ("this finding looks severe — pull more telemetry and cross-check") that a fixed pipeline cannot express.
• Add a bounded within-stage agent (Strands) if reconciliation needs an actual tool-use loop rather than a single structured call.

References

• HLD diagram: ../diagrams/nwaf-agent-hld.png
• Related: ADR-0004 (detection tooling — per-pillar portfolio & the deterministic-vs-judgment split), ADR-0001 (compute — hosts the scanners), ADR-0002 (LLM provider — the reconciliation model), the least-privilege IAM ADR, the human-in-the-loop ADR (findings only, no auto-remediation).