epic: tech-debt paydown #2 — live-state / runtime (TD2-NN)

[gangster]

Inventory: docs/tech-debt-audit-2026-06-26-second-pass.md

Second tech-debt audit pass — live-state / runtime, complementing the static first pass (epic #769). 7 parallel adversarial agents covered live AWS + cluster state, DR/backup, secrets lifecycle, policy coverage, resilience/SPOF, cost/FinOps, and regulated-tier readiness. Every Tier-1/2 headline was spot-verified against live state before filing.

Top-line: live state is exceptionally clean (measurable cost-waste < ~$1/mo, no orphaned resources, no circular deps). The real debt is missing resilience/recovery machinery plus a few live faults — not cruft.

Tier 1 — act soon (verified)

• TD2-01: Keycloak CNPG DB has zero backup (data-loss risk) #810 — TD2-01 (HIGH) CNPG Backstage + Keycloak DBs: single-instance, reclaimPolicy: Delete, zero backup (Keycloak = identity SoT)
• TD2-02: No rotation on any Secrets Manager secret #811 — TD2-02 (HIGH) No rotation on any of the 26 Secrets Manager secrets
• TD2-03/04/18: preprod Kyverno admission hardening (HA + CPU limit + failed release) #812 — TD2-03 (MED-HIGH) preprod Kyverno admission single-replica, fail-closed, no PDB (scale to 2–3 + PDB). [TD2-18 failed release self-healed; TD2-04 CPU split to Kyverno: skip Crossplane reconcile SA via a GLOBAL webhook matchCondition (TD2-04 CPU) #833]
• Kyverno: skip Crossplane reconcile SA via a GLOBAL webhook matchCondition (TD2-04 CPU) #833 — TD2-04 (MED) preprod Kyverno ~1.1-core admission churn — global webhook matchCondition to skip the Crossplane reconcile SA (the per-policy attempt fix(policy): stop envelope webhook firing on Crossplane reconciles (TD2-04) #830 was reverted Revert #830 — envelope matchCondition is ineffective (aggregated webhook) #831). Unblocks TD2-13: preprod right-sizing — consolidate to one node #818.
• TD2-05: Backstage portal down — backend can't reconnect to rescheduled CNPG DB #813 — TD2-05 (MED) Backstage portal down (can't reconnect to rescheduled DB)

Tier 2 — resilience / DR / cost

• TD2-09/10: DR — backup tooling + state-backend protection (ADR-054) #814 — TD2-09/10 Backup tooling (Velero or CNPG Barman/PITR) + state Object-Lock & lock-table protection (ADR-054)
• TD2-07: Keycloak HA + CNPG instance counts not wired to cost_profile #815 — TD2-07 Wire Keycloak HA + CNPG instance counts to cost_profile (stay single even at prod) or document
• TD2-08: EKS control-plane audit logging off on both clusters #816 — TD2-08 EKS control-plane audit-logging path (off on both; hipaa/pci blocker)
• TD2-14: Tenant/product ECR repos have no lifecycle policy #817 — TD2-14 ECR lifecycle policy on the per-product provisioning path

Tier 3 — hygiene / coverage

• TD2-13: preprod right-sizing — consolidate to one node #818 — TD2-13 preprod right-sizing → likely consolidate to one node (~50% preprod node cost)
• TD2-15/16: platform node disk pressure + OTel Instrumentation never installed #819 — TD2-15/16 Platform node image-fs 85% + agent revisionHistoryLimit; OTel Instrumentation never installed
• TD2-17: agent-demo/checkout crashloop fixture is hand-applied drift #820 — TD2-17 agent-demo/checkout hand-applied drift — move to git or delete
• TD2-20: Kyverno coverage gaps (platform namespaces, NetworkPolicy, RBAC escalation) #821 — TD2-20 Kyverno coverage gaps (platform namespaces, NetworkPolicy, RBAC escalation)
• TD2-11: Regulated-tier (hipaa/pci) pod policies are untested #822 — TD2-11 Exercise the regulated-tier pod policies in the test harness

Tier 4 — deferred re-confirmed + leftovers (low risk)

• TD2-21/22: re-confirmed deferred items + rename leftovers #823 — TD2-21/22 Still-open first-pass items (TD-103/105/107/206) + rename leftovers (tenant-api label, app-alpha SA)

Corrections logged (skeptic discipline)

• An agent's "Kyverno is Audit-only on both clusters" was false — it read the deprecated spec.validationFailureAction and missed per-rule validate.failureAction: Enforce. Kyverno is enforcing.
• First-pass TD-208 ("prod VPC costs NAT for nothing") is false in live state — prod has no NAT gateway ($0).