P11 (pt2) — True cloud cost: CUR → Athena → Grafana (ADR-079 D4)

[gangster]

P11 part 2 of the observability epic #102. Decided in ADR-079 D4. Brings true AWS spend (the actual bill, incl. the non-cluster cost OpenCost can't see — NAT data processing, data transfer, EKS control-plane, EBS, NLB hours, TGW, S3/ECR, KMS) into Grafana, attributed by the Team tag, next to in-cluster OpenCost allocation.

Why (the gap)

OpenCost runs cloudCost.enabled = false today: it estimates in-cluster compute as node list-price × pod allocation. It cannot see the real invoice (after Savings Plans/discounts) or any non-cluster service. This issue closes that with the authoritative source — the CUR.

---

Architecture

mgmt account (851725353202, payer)            platform account (829808296602)
┌──────────────────────────────────┐          ┌─────────────────────────────┐
│ CUR / Data Export (Parquet) → S3  │          │ EKS hub: OpenCost + Grafana │
│ Glue crawler → Glue table         │  cross-  │                             │
│ Athena workgroup                  │◀─account─│ OpenCost cloudCost (or       │
│ Team activated as cost-alloc tag  │  assume  │ Grafana Athena datasource)  │
└──────────────────────────────────┘          └─────────────────────────────┘

Two decisions baked in:

1. CUR lives in mgmt (payer) — a member account can't generate an org CUR. Glue + Athena co-locate in mgmt (us-east-1; CUR is us-east-1 only). The platform consumer assumes a mgmt-side read role. (This cross-account hop is the main complexity.)
2. Consume via OpenCost cloudCost (recommended), not the Grafana Athena plugin. The Athena datasource is a plugin and the Grafana here has no plugin-install mechanism wired (only allow_loading_unsigned_plugins). OpenCost's cloudCost reuses the existing Prometheus/Mimir + platform-cost dashboard with zero new Grafana surface. The standalone Athena datasource stays an optional Phase 2b for ad-hoc CUR SQL. Either way the AWS plumbing (CUR + Glue + Athena) is identical — only the consumer differs.

---

Plan (phased; stop-early friendly)

Phase 0 — Prerequisites & decisions (~0.5d)

• Activate the Team (and team) cost-allocation tag in mgmt via aws_ce_cost_allocation_tag. ⚠️ Forward-only + ~24h lag — costs only break down by team from activation onward, not retroactively. Do this first so data accrues while the rest is built.
• Confirm PlatformDeployer in mgmt can manage cur:/bcm-data-exports:/glue:/athena:/ce: (the deny-regions SCP already exempts these globally — organizations/scps.tf:302 — but verify the deployer role policy allows them; may need an IAM add).
• Decide scope: org-wide CUR (all accounts, the real per-team story) vs platform-account-only (smaller). Recommend org-wide.

Phase 1 — AWS CUR + Athena infra (new aws/cost-export module, applied in mgmt) (~1.5–2d)

• New module infra/modules/aws/cost-export/ (follow aws/cloudtrail bucket conventions: create toggle, versioning, SSE-KMS, public-access-block, lifecycle, tags).
• CUR: aws_cur_report_definition (or aws_bcmdataexports_export for CUR 2.0) — Parquet, OVERWRITE_REPORT, hourly, → the S3 bucket.
• Glue: aws_glue_catalog_database + aws_glue_crawler (daily) over the CUR Parquet prefix — auto-discovers schema + partitions (avoids hand-defining ~100 CUR columns and handles schema drift).
• Athena: aws_athena_workgroup with a results-output S3 location (lifecycle-expired).
• Cross-account read role (cost-reader, in mgmt): trust the platform Grafana/OpenCost Pod-Identity role; allow athena:* (scoped to the workgroup), glue:Get* (db/table/partitions), s3:GetObject/ListBucket on the CUR bucket + results location.
• Live unit infra/live/aws/mgmt/global/cost-export/terragrunt.hcl (alongside state-bootstrap).
• Verify: an Athena query in the AWS console returns CUR rows partitioned by month; Team column populated (post-activation lag).

Phase 2a — OpenCost cloudCost (recommended consumer) (~0.5–1d)

• Extend observability-opencost: flip cloudCost.enabled = true, wire aws athena inputs (bucket/database/table/workgroup/region) + the cross-account role ARN (OpenCost assumes the mgmt cost-reader).
• Grant the OpenCost SA a platform-side Pod-Identity role allowed to sts:AssumeRole the mgmt cost-reader (mirror the grafana_cloudwatch role pattern, observability/main.tf:588-672).
• Gate on enable_cost_metrics (already true on platform) + a new enable_cloud_cost sub-flag (default off) so it's opt-in.
• Verify: OpenCost exposes cloud_cost_* / node+service cloud-cost metrics in Mimir; spend by service/account/Team queryable.

Phase 3 — Dashboards (~0.5d)

• Extend the existing Platform Cost dashboard (now Mimir-backed) with true-cloud-cost rows: spend by service, by account, by Team, and an OpenCost-estimate-vs-CUR-actual reconciliation panel.
• Verify: in-cluster allocation AND real cloud spend by team in one pane (the P11 verify bar from Observability stack (epic) — metrics, logs, traces, profiles, cost #102).

Phase 2b — Grafana Athena datasource (OPTIONAL, ~1d)

• Only if ad-hoc CUR SQL in Grafana is wanted. Requires solving plugin install (GF_INSTALL_PLUGINS=grafana-athena-datasource or chart extraInitContainers) + an additionalDataSources Athena entry + the same cross-account role. Defer unless asked.

---

Cost & effort

• Run cost ≈ $1–5/mo (CUR delivery is free; S3 pennies; Athena $5/TB scanned but CUR is small; Glue crawler pennies).
• Build ≈ 3–4 days for Phase 0→1→2a→3 (the per-team-true-cost MVP). +1d for the optional Athena datasource.

Risks / caveats

• Cross-account is the real work — CUR in mgmt, consumer in platform; the assume-role chain + bucket policy must be exact.
• Cost-allocation tag is forward-only + ~24h lag — activate in Phase 0 or the first month has no team breakdown.
• CUR schema drift — the Glue crawler handles it; a hand-defined table would not.
• mgmt-account blast radius — this adds billing infra to the payer account; keep it least-privilege and create-gated.

Open questions

1. Org-wide CUR (recommended) or platform-account-only for v1?
2. OpenCost cloudCost only, or also the Grafana Athena datasource (Phase 2b)?
3. Activate cost-allocation tags now (start the 24h clock) even before building Phase 1?

[gangster]

Phase 1 applied + live (2026-06-30)

The CUR → S3 → Glue → Athena pipeline is built and applied to the mgmt/payer account (PR #1064). Live resources:

• CUR platform-cur (legacy, Parquet, hourly, RESOURCES, ATHENA artifact)
• S3 bucket platform-cost-export-48f6bc2bb22f043dd3d8b2c84f (AES256)
• Glue db platform_cur + daily crawler platform-cost-export-cur
• Athena workgroup platform-cost-export
• Cross-account read role arn:aws:iam::851725353202:role/platform-cost-reader (for Phase 2a)

Decisions from the recon spike: legacy CUR (not CUR 2.0 — OpenCost cloudCost schema compatibility), AES256 (the CUR service + cross-account reader can't use the managed KMS key), org-wide CUR, Team tag already active (#673).

⏰ Pending verification (do ~2026-07-01, after the ~24h CUR first-delivery lag)

The Glue table + Athena queries are empty until the CUR's first delivery lands (~24h after apply). Then verify per docs/runbooks/cost-true-spend.md:

• aws s3 ls s3://platform-cost-export-48f6bc2bb22f043dd3d8b2c84f/cur/ --recursive shows CUR data
• run the Glue crawler → aws glue get-tables --database-name platform_cur shows a table
• Athena spend-by-team query returns rows with resource_tags_user_team populated

Remaining

• Phase 2a — flip observability-opencost cloudCost.enabled = true, assume platform-cost-reader, wire Athena inputs.
• Phase 3 — dashboard rows (true spend by team/service/account; OpenCost-estimate-vs-CUR-actual reconciliation).