fix(kubernetes-cost): make cost.enabled=true work end to end

[simlarsen]

Found by deploying the kubernetes-agent chart with --set cost.enabled=true against a real 3-node DigitalOcean cluster and following the data. It produced zero cost rows. Three independent defects, each fatal on its own.

1. The poller never got a single allocation

Date.toISOString() always emits milliseconds, and neither OpenCost nor Kubecost accepts a fractional RFC3339 bound in the Allocation API. Every fetch, on every candidate path, failed:

HTTP 400 Invalid 'window' parameter: illegal window:
  2026-07-25T16:00:00.000Z,2026-07-25T17:00:00.000Z

Confirmed by hand against the bundled OpenCost 1.121.0:

window bound	/allocation/compute	/allocation
2026-07-25T17:00:00.000Z,...	400 illegal window	400 illegal window
2026-07-25T17:00:00Z,...	200	200

Truncated to second precision. The poller floors windows to the window grid before this point, so the millisecond field is always zero — nothing is lost.

The existing test asserted the millisecond form, which is how this shipped; it now pins the format the engines actually parse.

2. The cost agent image was unpullable

cost.agent.image.tag fell back to .Chart.AppVersion ("1.0.0"). Nothing bumps the chart's appVersion at release time, and the pipeline publishes <major>.<minor> plus release — so the documented one-command install lands in ImagePullBackOff. Now defaults to release, the moving tag that always exists (same convention as the main oneuptime chart).

logs.api.image.tag had the identical latent bug, reachable via preset=gke-autopilot / eks-fargate.

3. Ingest was unroutable through the standard ingress

Every other ingest family has its own nginx location (/otlp, /telemetry, /pyroscope, /probe-ingest, /server-monitor, ...). /kubernetes-cost had none, so with BILLING_ENABLED=true the catch-all location / proxies to the marketing Home service and the POST 404s there. On a live billing-enabled instance:

POST /kubernetes-cost/ingest            -> 404 {"message":"Page not found - /kubernetes-cost/ingest"}
POST /telemetry/kubernetes-cost/ingest  -> 200

Both are the same Express route (TELEMETRY_PREFIXES = ["/telemetry", "/"]), which is what pins this on the proxy rather than the app.

Added the missing location (validated with nginx -t), and pointed the poller at the /telemetry prefix so a new agent also reaches instances whose reverse proxy predates this change.

Also

Cost was the one enabled collector NOTES.txt never mentioned — added, distinguishing the bundled engine from an external cost.engine.url.

Verification

Deployed against a 3-node DigitalOcean cluster (s-4vcpu-8gb, k8s 1.35.1) with the bundled engine.

• OpenCost detects DigitalOcean and prices nodes at $0.0714/hr — the real list price — and volumes at $0.000137/GiB/hr, with no credentials.
• Bundled Prometheus: both scrape targets (kubernetes-nodes-cadvisor via the API-server node proxy, and opencost) up.
• Path auto-detection resolves correctly: /model/allocation 404s, /allocation/compute wins.
• After the fix the poller ships its first closed window: shipped cost window windowStart=18:00 windowEnd=19:00 rows=39.
• Running the agent's own mapAllocationToRow over that window: 39 rows, all with valid second-precision windows, namespaces __idle__ / kube-system / oneuptime / wordpress, __idle__ sentinel mapped correctly, per-component costs and efficiency populated. Totals are internally consistent — $0.137/hr against 3 x $0.0714/hr of capacity scaled by the ~64% of the window Prometheus had history for.
• 11 of the 14 allowlisted cost metrics are exported by the bundled engine; the 3 absent ones (kubecost_network_*_egress_cost) are Kubecost-only and no dashboard panel references them.
• KubernetesCostAgent: 28/28 tests pass. eslint and tsc clean.

Not verified here: ClickHouse persistence and the Costs UI rendering. The test instance runs with DISABLE_TELEMETRY_INGESTION=true, so every ingest endpoint returns 200 and drops the payload (a malformed body with a valid key returns 200 instead of 400). The 39 rows above were accepted and dropped. That is an instance setting, not a code path in this diff.

[simlarsen]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[simlarsen]

End-to-end verification now complete

The test instance had DISABLE_TELEMETRY_INGESTION=true when the PR was opened, so the ingest half was unverified. Ingestion has since been enabled (confirmed: a malformed body with a valid key now answers 400 instead of 200), and the full path works.

Cluster auto-registration — the cost ingest alone created the cluster record via findOrCreateByClusterIdentifier; do-sfo2-cost-test shows as Connected on the Clusters page with no other collector running.

Cluster Costs page (Kubernetes → cluster → Costs):

Tile	Value
Total spend	$0.23
Workload spend	$0.08
Idle spend	$0.15
Idle %	66%

• Spend by namespace with cpu/memory/storage split and efficiency: Idle $0.15, wordpress $0.03 (33%), oneuptime $0.03 (35%), kube-system $0.02 (68%).
• Spend by workload: 18 controllers with kind, namespace, total and efficiency — wordpress deployment $0.03 (28%), cilium daemonset $0.01 (85%), wordpress-mariadb statefulset $0.0041 (38%), down to doks-telemetry-config-reloader at $0.0001.

Project Costs page (Kubernetes → Costs): same tiles plus Spend by Cluster — do-sfo2-cost-test, workload $0.08 / idle $0.15 / total $0.23, 56% efficiency.

Numbers reconcile at every layer: $0.08 workload + $0.15 idle = $0.23 total; idle 66% = 0.15/0.23; and the total matches the two shipped windows (39 rows + 38 rows) summed from the raw Allocation API responses. Sentinel __idle__ rows render as "Idle (unused capacity)" and are correctly excluded from the workload table.

So the verified chain is: OpenCost pricing → bundled Prometheus → cost poller → HTTP POST → telemetry queue → ClickHouse → aggregate queries → both Costs pages.

One operational note for the docs, not a defect in this diff: a full agent install (logs + eBPF traces + metrics from a 3-node cluster) produced ~55k spans and ~117k metric points in under two hours, which was enough to overwhelm a small single-instance OneUptime deployment once ingestion was on. Worth a sizing note in the agent guide.