OpenCost allocations report zero usage against Coroot Prometheus

[devantler]

🤖 Generated by the Daily AI Engineer

Reproduction and evidence

Read-only production verification on 2026-07-18 found the OpenCost release Ready with two healthy primary replicas, but its Coroot-backed metrics input is incomplete:

• Coroot Prometheus returns 0 series for node_cpu_seconds_total, container_cpu_usage_seconds_total, and container_memory_working_set_bytes.
• It does return kube-state data (kube_pod_container_resource_requests: 677 current samples; kube_node_status_capacity: 54).
• OpenCost GET /allocation/compute?window=24h&aggregate=namespace returns HTTP 200 and 34 namespaces with non-zero requests/costs, but 0/34 namespaces have non-zero cpuCoreUsageAverage or ramByteUsageAverage.

The release being healthy therefore masks a functional degradation: allocation cost is derived from requested resources, while actual usage and efficiency are absent.

Audience and impact

Operators using OpenCost for utilization, idle-spend, or request-versus-usage decisions receive plausible-looking costs with zero actual CPU/RAM usage. This can misstate efficiency and makes the current OpenCost view unsafe as a usage-aware source of record.

The same latent design was about to be copied into the Platform template; platform-template#82 stopped that rollout after exact-head review prompted this live check.

Expected behavior

Choose and prove one coherent cost path:

1. keep OpenCost and provide a Prometheus source with the standard node/cAdvisor/kube-state series its allocation model needs; or
2. retire OpenCost and use the independently live-proven Coroot node/application cost rollups instead.

Do not treat Helm readiness or a successful allocation HTTP response as sufficient proof.

Acceptance criteria

• Record the chosen OpenCost-versus-Coroot cost architecture and its trade-off.
• If OpenCost stays, its Prometheus query surface contains the required standard usage series and a 24-hour allocation reports non-zero CPU/RAM usage for active namespaces.
• If OpenCost retires, remove its release, route, policy, and related ownership cleanly, and update user-facing cost guidance to Coroot.
• Add an executable or operational check that detects the current zero-usage false-green state.
• Validate statically and verify the resulting cost view read-only in production before closing.

Rough size: M

Related: #1784, #2043, devantler-tech/platform-template#65

[coderabbitai]

🔗 Related PRs

#2507 - fix(apps): make actual-budget openid schema offline-resolvable [merged]
#2527 - feat: install the OpenFeature Operator (runtime feature-flag layer) [merged]
#2554 - chore(deps): update helm release opencost to v2.5.26 [merged]
#2555 - fix(opencost): patch ui probe timeouts [merged]
#2578 - fix(policy-reporter): run all components at the HA replica floor [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[devantler]

🤖 Generated by the Agentic Engineer

Read-only revalidation on 2026-07-22 narrows the failure to an incompatible metrics input, not OpenCost-to-Prometheus reachability:

• The 3-day allocation still returns 34 namespaces, but cpuCoreUsageAverage sums to zero across all of them.
• Coroot Prometheus returns 339 kube_pod_container_resource_requests{resource="cpu"} series, but zero container_cpu_usage_seconds_total series and no 5-minute container CPU rate.
• The live coroot-prometheus deployment config only self-scrapes 127.0.0.1:9090; its other data arrives through remote write. The required cAdvisor/container-usage series are not present in that TSDB.
• OpenCost is healthy and explicitly points PROMETHEUS_SERVER_ENDPOINT at this Coroot Prometheus service.

So the current root cause is the metrics contract: the configured Prometheus source has kube-state request data but does not ingest the container usage series OpenCost needs. Rightsizing remains correctly fail-closed until a compatible usage source is supplied and the proposed regression check proves non-zero usage for active namespaces.