Add Prometheus metrics endpoint for browser VM observability

[sjmiller609]

Summary

Adds a GET /metrics endpoint (Prometheus text format, dedicated listener on METRICS_PORT, default 10002) so an external collector can scrape per-VM browser and system metrics and aggregate them across the fleet. Metrics carry no per-session or per-user labels — instance identity is attached (and aggregated away) by the scraper.

Chrome metrics — lib/metrics/chrome.go

Read browser-level over CDP (Browser.getHistograms, Browser.getVersion, Target.getTargets): no page attach, no script injection, no navigation — invisible to automation running in the browser.

Metric	Meaning
kernel_chromium_uma{histogram=...}	Prometheus histogram (_bucket/_sum/_count) per curated UMA histogram: FCP, LCP, DOMContentLoaded, load event, parse start (~TTFB), INP, FID, interactions per page, CLS, per-page CPU ms and browser-start-to-first-paint. Cumulative since browser start, so rate()/increase() work naturally.
kernel_chromium_up	CDP endpoint reachable and responsive
kernel_chromium_info{product}	Browser version (fleet version distribution)
kernel_chromium_pages	Open page targets (tabs)

Chrome's sparse [low, high) UMA buckets map to cumulative le bounds; the overflow bucket folds into +Inf. The curated list lives in DefaultUMAHistograms; histograms with no samples yet are simply absent.

VM / process metrics — lib/metrics/system.go

kernel_vm_cpu_seconds_total{mode}, kernel_vm_memory_{total,available}_bytes, kernel_vm_load1, kernel_vm_uptime_seconds (monotonic — excludes suspended time), kernel_vm_network_{receive,transmit}_bytes_total, kernel_vm_disk_{total,free}_bytes{mount="/"}, kernel_chromium_processes, kernel_chromium_memory_rss_bytes.

Design notes

• Separate listener, no scale-to-zero middleware: periodic scrapes must not count as session activity or hold VMs awake. Also skips per-request logging so scrapes don't drown the logs.
• Not in the OpenAPI spec: this is an internal scrape surface, following the /internal/fork-identity precedent.
• Browser-process histograms only: Browser.getHistograms cannot see renderer/child-process histograms (EventLatency.*, Graphics.Smoothness.*, Blink.*) — those only merge into the browser's recorder when a UMA log is staged or chrome://histograms is opened, neither of which happens on these images. Verified empirically against a live browser; the curated set is restricted to histograms that populate without a sync.
• No new dependencies: the exposition format is rendered directly; scrape cost is a few small CDP round-trips + /proc reads (~50ms).
• New cdpclient methods GetHistograms and CountPageTargets follow the existing minimal-client patterns.

Test plan

• Unit tests for exposition rendering/escaping, UMA→Prometheus bucket conversion (incl. exact-name filtering and overflow fold), /proc parsers, and the chrome collector against a fake CDP server
• Full unit suite (go vet + go test -race, non-e2e) passes
• Validated the real cmd/api binary inside two live browser VMs (both image variants) on alternate ports: all metric families present with sane values; page-load histogram sums matched values read independently via raw CDP
• e2e suite not run (requires pre-built Docker images); the endpoint has no interaction with existing routes

Sample output (from a live VM)

kernel_chromium_up 1
kernel_chromium_info{product="Chrome/145.0.7632.75"} 1
kernel_chromium_pages 1
kernel_chromium_uma_bucket{histogram="PageLoad.PaintTiming.NavigationToFirstContentfulPaint",le="885"} 1
kernel_chromium_uma_bucket{histogram="PageLoad.PaintTiming.NavigationToFirstContentfulPaint",le="1240"} 2
kernel_chromium_uma_bucket{histogram="PageLoad.PaintTiming.NavigationToFirstContentfulPaint",le="+Inf"} 2
kernel_chromium_uma_sum{histogram="PageLoad.PaintTiming.NavigationToFirstContentfulPaint"} 1977
kernel_chromium_uma_count{histogram="PageLoad.PaintTiming.NavigationToFirstContentfulPaint"} 2
kernel_vm_cpu_seconds_total{mode="user"} 75.1
kernel_chromium_processes 8
kernel_chromium_memory_rss_bytes 1.277575168e+09

🤖 Generated with Claude Code

---

Note

Medium Risk
New exposed listener and CDP/nvidia-smi work on each scrape add operational surface and load; metrics path is intentionally isolated from scale-to-zero and the public API.

Overview
Adds a dedicated Prometheus scrape surface on METRICS_PORT (default 10002) with GET /metrics, wired in cmd/api on its own HTTP listener without scale-to-zero middleware or request logging so periodic scrapes do not keep VMs awake or flood logs.

Introduces lib/metrics with a custom text exposition handler (serialized scrapes, per-collector timeouts, failed collectors dropped without partial families) and three collectors: Chrome (browser-level CDP: kernel_chromium_up, version, page count, curated UMA histograms re-bucketed for fleet aggregation), system (/proc CPU/memory/load/PSI/network/disk plus Chromium process count/RSS), and GPU (nvidia-smi when present, else kernel_gpu_present 0).

Extends cdpclient with GetHistograms and CountPageTargets. Config and README document METRICS_PORT; config tests cover the new default and override.

^{Reviewed by Cursor Bugbot for commit f993190. Bugbot is set up for automated code reviews on this repo. Configure here.}

[rgarcia]

reviewed the full diff — overall this is in good shape: right transport boundaries (separate listener, no scale-to-zero coupling), curated allowlist, no per-session labels, solid test coverage. two structural findings worth addressing before merge, one suggestion, and a handful of nits.

main findings

• server/lib/metrics/chrome.go:119-146 — sparse UMA buckets break cross-VM aggregation and inflate cardinality. chrome only returns buckets that have samples, and writeUMAHistogram maps them directly to le bounds — so two VMs emit different le sets for the same histogram depending on what pages they loaded. sum by (le) at the collector then produces non-monotonic garbage (histogram aggregation requires identical bucket sets), and worst case is ~50-100 le values per histogram per VM. consider re-bucketing into a small fixed boundary set (10-15 hardcoded bounds per histogram) before exposition — chrome's [low, high) counts re-map cleanly onto wider fixed bounds, every VM emits identical le sets, and cardinality drops ~10x.
• server/lib/metrics/metrics.go:89-93 — partial-family output on collector error. collectors write into the shared buffer as they go, and e.g. the return err in the UMA loop (chrome.go:91) exits after the kernel_chromium_uma header (and possibly some histograms) are already written. strict parsers (prometheus, otel collector) reject the whole scrape body on a malformed family, so one flaky collector can zero out all metrics. consider gather-then-write per collector: collect into a local Writer, append only on success. one wrinkle: the chrome collector intentionally writes kernel_chromium_up 0 then returns an error (chrome.go:70-73) — that sample is load-bearing for browser-down alerting, so with gather-then-write the collector should instead log the dial failure and return nil after writing up 0, so "returned error" can safely mean "discard the buffer". TestHandlerServesAllCollectorsDespiteError and TestChromeCollectorDown assert the current behavior and would need updating, and a test for a GetHistograms failure mid-loop would pin the fix.

suggestion

• server/lib/metrics/system.go — add PSI gauges from /proc/pressure/{cpu,memory,io} (some avg10 per resource, 3 gauges). it's the most workload-independent "guest is starved" signal — the thing steal time only approximates — and this collector already reads everything adjacent to it.

nits

• server/lib/metrics/chrome.go:129-133 — if chrome's snapshot ever yields count below the cumulative bucket sum, the +Inf sample lands below the last bucket (non-monotonic histogram). count = max(h.Count, bucketSum) is strictly safer.
• server/lib/metrics/chrome.go:42 — Memory.GPU.PeakMemoryUsage2.Scroll is MB while the rest of the family is ms; the shared HELP string is honest but dashboards will trip on it. maybe call out the exceptions in the HELP text.
• server/lib/metrics/system.go:174-203 — summing per-process RSS double-counts shared pages, so the total scales with renderer count more than real memory pressure. smaps_rollup PSS is the accurate version; otherwise worth a comment that this is an upper bound for trends, not absolute usage.
• server/lib/metrics/system.go:63 — /proc/uptime is CLOCK_BOOTTIME-based on modern kernels, not CLOCK_MONOTONIC. the conclusion holds (external pause stops guest clocks entirely) but the stated mechanism is off.
• server/lib/metrics/gpu.go:154 — time.Parse with a named-zone abbreviation silently parses unknown zones with offset 0. fine while nvidia-smi always emits GMT, but asserting that (check the suffix + parse in UTC) or commenting the assumption keeps a driver change from silently skewing expiry.
• server/lib/metrics/gpu.go:137 — the now param of parseNvidiaSMI is unused; Collect computes the delta itself.
• server/lib/metrics/gpu.go:145 — only GPUs[0] is read; a one-line comment that single-vGPU-per-VM is assumed would help.

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit f993190. Configure here.}