ADR 0004 detection tooling portfolio

ADR-0004: Detection tooling — per-pillar scanner portfolio

Status: Accepted (2026-07-22)

Context

ADR-0003 commits to deterministic detection feeding a bounded LLM reconciliation stage. But the AWS Well-Architected Framework has six pillars, and deterministic OSS coverage is uneven: Security is richly tooled, Cost well tooled, while Performance Efficiency, the process side of Operational Excellence, and Sustainability are only partially — or not — machine-checkable.

A credible design must (a) map a concrete tool to each pillar where determinism exists, and (b) state honestly where it does not, so those answers are clearly LLM/questionnaire-assessed rather than presented as scanner-derived fact. Overclaiming "full WA review from scanners" would be the failure mode.

Decision

Adopt a per-pillar scanner portfolio, run symmetrically on live account and Terraform code, with an explicit deterministic-vs-judgment boundary.

Per-pillar tool mapping

Pillar	Live (actual state)	Code (declared state)	Determinism
Security	Prowler	Checkov (tfsec/Trivy alt)	✅ Strong
Reliability	Prowler + Steampipe/Powerpipe WA mod	Checkov reliability policies	✅ Config subset
Cost Optimization	Steampipe "Thrifty" mod + cloud-custodian	Infracost (cost from code)	✅ Strong, incl. code-side
Operational Excellence	Prowler (CloudTrail/Config/monitoring) + cloud-custodian (tagging) + terraform plan drift	Checkov ops policies	⚠️ Technical subset only — process items (runbooks, game days, change mgmt) → questionnaire + LLM
Performance Efficiency	AWS Compute Optimizer (native, deterministic right-sizing) + limited config checks	limited (instance families, gp2→gp3)	⚠️ Weak — largely runtime metrics + judgment
Sustainability	Cloud Carbon Footprint (optional) + waste/idle proxies from Cost tools	—	⚠️ Thin — largely judgment

Code-side scanner

Standardise on Checkov (most widely adopted, policy-as-code, custom-policy support, SARIF output). tfsec/Trivy remain drop-in alternatives (Trivy also covers image/dependency scanning if scope grows). Infracost is additive, not an alternative — it answers Cost from code, which Checkov does not.

Finding normalisation

Scanners emit heterogeneous formats (Prowler → OCSF/ASFF, Checkov → SARIF, Steampipe → JSON, Infracost → JSON). The pipeline normalises all of them to one canonical finding record before reconciliation:

{ pillar, resource, check_id, source_tool, origin: live|code,
  severity, status, evidence }

The LLM reconciles over this uniform schema, not over each tool's native shape.

The deterministic-vs-judgment boundary (load-bearing)

• Scanner-derived answers cite a tool + check ID + evidence and are reproducible.
• Judgment answers — Performance Efficiency, the process side of Operational Excellence, and most of Sustainability — are produced by LLM reasoning over available signals plus the WA Tool questionnaire, and are labelled as such in the report and WA-Tool notes. They are never presented as scanner facts.

Consequences

• Every scanner-backed finding is auditable (tool + check ID + evidence).
• Coverage is honest — the report distinguishes "detected" from "assessed", which is itself a governance strength.
• Tooling to install/pin in the build image (ADR-0001): Prowler, Checkov, Infracost, Steampipe + Powerpipe; Compute Optimizer and Cloud Carbon Footprint are consumed via API rather than installed.
• Broad read-only IAM — Prowler/Steampipe need SecurityAudit + ViewOnlyAccess-style permissions, wider than a hand-rolled single-check reader. Still strictly read-only; recorded in the IAM ADR.
• Maintenance surface — scanner versions must be pinned and periodically bumped; check-ID→pillar mappings maintained as the tools evolve.
• Tool maturity moves fast — mod names and maintenance status (e.g. driftctl is archived; use terraform plan drift) must be re-verified at implementation.

Alternatives considered

Option	Verdict	Why
Per-pillar portfolio + explicit judgment boundary	Chosen	Deterministic where possible, honest where not; auditable evidence per finding.
Prowler + Checkov only	Rejected	Covers Security/Reliability well but silently drops Cost-from-code, Ops, Performance, Sustainability.
Single tool for everything	Rejected	No OSS tool spans all six pillars; Steampipe reaches the most but not Security depth or code-side cost.
Scanner-only (no LLM judgment for soft pillars)	Rejected	Would leave Performance/Sustainability/process-Ops unanswered or force false precision.
tfsec/Trivy instead of Checkov	Viable alt	Comparable; Checkov chosen for adoption + custom policies. Trivy revisited if image/dependency scanning enters scope.

Revisit trigger

• Add Trivy if container images or dependencies enter review scope.
• Add AWS Resilience Hub (native) if Reliability needs a formal resiliency score beyond config checks.
• Promote Sustainability from proxy-based to first-class if Cloud Carbon Footprint (or an AWS-native successor) matures enough to be authoritative.

References

• ADR-0003 (detection & orchestration — how these findings are consumed)
• ADR-0001 (compute — the build image that hosts these CLIs)
• The least-privilege IAM ADR (the read-only policy Prowler/Steampipe require)