Expose proxy errors as first-class browser telemetry events

[chruffins]

Summary

The metro egress host-proxy serves branded 5xx error pages carrying an X-Kernel-Proxy-Error header with a typed code when a proxy-layer failure occurs. This change makes those failures first-class in browser telemetry from the image side: the CDP collector now emits a dedicated low-volume proxy_error event carrying the code, instead of burying them in the raw network_response stream.

What changed

• openapi.yaml + regenerated oapi.go / category_gen.go: new BrowserProxyErrorEvent/BrowserProxyErrorEventData schema (proxy_error, category network) registered in the KnownBrowserTelemetryEvent discriminated union. code is a typed enum mirroring the proxy's wire values (destination_blocked, provider_blacklisted, provider_unreachable, proxy_unavailable, upstream_timeout, upstream_dns_failure, upstream_connect_failed); status is a required int (502).
• cdpmonitor/handlers.go: on Network.responseReceived, when a 5xx response carries X-Kernel-Proxy-Error, emit proxy_error with the header value as code, plus status, url, method, request_id, nav_seq, and target/frame context. Header lookup is case-insensitive; origin 5xx pass-through is untouched (no masking). Requests in flight at CDP attach get their context from the CDP params.
• Enum validation + rate limit: code is validated against the generated enum before use, so unknown header values are dropped and the rate-limit map stays bounded; emission is deduplicated per (session, code) with a 1s min interval so volume tracks an outage without flooding the ring.
• lib/events/otlpconvert.go: proxy_error maps to ERROR severity only for the top-level Document and WARN for subresources; status promotes to http.response.status_code.
• Tests: classifier/rate-limit unit tests, response-path unit tests (branded, untracked), and a real-Chromium e2e that serves a branded 502 and asserts the emitted event.

Notes / trade-offs

• The event is header-driven: it is emitted only when the proxy's branded page is observed, so the code is always the real header value. No derived/synthetic codes.
• It rides the network telemetry category (CDP-derived and opt-in). Its value is per-session/per-URL attribution for sessions already capturing the network stream — not a default-on alerting signal (proxy failures are only observable while the CDP collector runs).
• No Chromium patch and no change to the browser fork: the image-side observation point is the CDP collector. The oapi.go diff is large because it is regenerated (embedded spec + union accessors).
• Deliberately not in scope: making the event default-on (would require the CDP collector always running), server-side emission from metro, and a provenance gate (remote addresses can't distinguish proxy-generated from origin responses; the header is signal-integrity advisory).

Related

• kernel/kernel #3283 mirrors the schema into the API's BrowserTelemetryEvent union + Stainless SDK models so consumers can type against proxy_error.

Tests

• go test ./lib/cdpmonitor/ ./lib/events/ — green (unit + real-Chromium e2e).

---

Note

Medium Risk
Adds new telemetry on the hot CDP response path (gated to 502 + header) and changes OTLP alerting semantics for document vs subresource proxy failures; scope is bounded by enum validation and sampling.

Overview
Introduces proxy_error as a first-class network telemetry event when the CDP monitor classifies a 502 response that carries X-Kernel-Proxy-Error, surfacing metro egress proxy failures with a typed code instead of only generic network_response traffic.

Schema & pipeline: OpenAPI adds BrowserProxyErrorEvent (enum codes aligned with metro), regenerated oapi.go / category_gen.go, and union wiring. cdpmonitor detects the header on Network.responseReceived (502-only gate), fills request/nav context from pending state or CDP params, validates codes against the enum, and rate-limits to at most one emit per session+code+resource type per second.

OTLP: Promotes code to kernel.proxy_error_code; proxy_error severity is ERROR for Document and WARN for subresources.

Unit tests, a Chromium e2e, and README taxonomy updates cover the new path. Events remain opt-in via the network CDP collector category.

^{Reviewed by Cursor Bugbot for commit ed5697e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[chruffins]

@cursor review

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 30fe6f1. Configure here.}

[Sayan-]

Reviewed with a focus on correctness. Leaving this as a comment rather than a formal review.

Needs changing

1. The code vocabulary doesn't match what the proxy emits.

provider_blocklisted should be provider_blacklisted. metro-api returns that literal today, and #3217 renames only the Go identifier (providerBlocklisted = "provider_blacklisted"), so the wire value is unchanged. Production only emits four reason values and this isn't one of them.

Three more documented values aren't reachable as header values:

• source_auth_error doesn't appear anywhere.
• source_auth_failed is a metric label only. There's a comment on that path noting the header deliberately exposes the generic proxy_unavailable instead, so a Kernel dependency outage isn't misread as a customer auth problem.
• destination_blocked only exists in #3217, which is still open. It's also the canonical example in the new e2e test.

Detection still fires either way since the value is passed through verbatim, but the spec is the published contract and it's already mirrored downstream, so a consumer matching the documented list silently drops a large share of real proxy errors.

2. proxy_error exports at severity INFO.

otlpSeverity in lib/events/otlpconvert.go keys off console_error, an explicit crash list, and a _failed suffix. proxy_error matches none of them, so it falls through to INFO in the customer's log backend, below the filters most teams run. page_crashed was added to the ERROR case for this reason.

3. status: 0 leaks into OTLP as http.response.status_code = 0.

promotedAttributes promotes data["status"] unconditionally for the network category, so the tunnel path exports a zero status code. The schema is inconsistent here too: method and resource_type are optional-when-unknown, but status is required with a sentinel for the same unknown case. Suggest a pointer, omitted on the tunnel path, with the promotion guarded.

4. The event is only deliverable to sessions that opted into the full network stream.

DefaultCategories excludes network specifically because it's the high-volume stream that starts the CDP collector. So the low-volume typed signal is only reachable by callers who already enabled the firehose it summarizes, which rules out the alerting use case in the description. Either move it to a default-on category, or reframe the value as per-session and per-URL attribution.

Worth a look

5. The header is origin-controlled and nothing checks provenance. Any site can serve a 502 carrying X-Kernel-Proxy-Error and forge a proxy-layer failure in telemetry. No consumer exists yet so nothing is fooled today, but cdpNetworkResponse already parses RemoteIPAddress and RemotePort and neither is used, so gating on the local egress proxy is cheap now and awkward later.

6. nav_seq is 0 on the responseReceived path when the request wasn't tracked, while handleLoadingFailed falls back to currentNavSeq() for exactly that case. A request in flight at CDP attach lands in epoch 0 while everything around it is in epoch N. p.LoaderID / p.FrameID / p.Type are also present in the params and dropped in that branch.

7. No dedup or rate limit. One event per failed request means volume tracks the outage, into a fixed-capacity ring, at the moment the pipeline is most contended. bindingCalled has a min-interval for this; worth deciding deliberately either way.

Checked out

Regenerated oapi.go and category_gen.go from the spec with the pinned generator: byte-identical to what's committed, no drift. The large diff is the embedded base64 spec blob plus gofmt realignment. Build, vet, and -race all pass. The handleResponseReceived refactor preserves the prior non-proxy path exactly and doesn't reintroduce the pendReqMu to sessionsMu ordering cycle. The union addition isn't breaking for existing consumers.

One thing I couldn't confirm: that bumped HTTPS responses surface the header normally through Network.responseReceived and nothing strips it in between. Worth a second opinion from whoever owns the egress proxy.

Minor

code is Kernel-controlled end to end, so an enum would match this repo's convention better than a prose list. optPtr and ptrOf are used with opposite semantics in the same payload builder (method and resource_type omit when empty, url / frame_id / loader_id emit ""). lib/cdpmonitor/README.md enumerates the CDP event types and wasn't updated.

[Sayan-]

Second pass. Vocabulary, OTLP severity, status pointer, and nav_seq all look correctly fixed; codegen still clean. Two blockers left.

1. The provenance gate is inverted. Envoy pools launch Chromium with --proxy-server=https://127.0.0.1:3128 (browser_pool_activities.go:354), exactly the gate's default; proxy-v3 pools use 3129. The header is only produced on proxy-v3, so the gate drops every genuine event and admits every forged one. Correcting the address won't help: I probed real Chromium in three proxy shapes and remoteIPAddress is always the socket peer, so it can't distinguish proxy-generated from origin-generated responses. Needs a proxy-minted token, or drop the gate. My suggestion last round pointed you here, so that one's on me.

2. code is unbounded remote text used as a never-swept map key. No length cap, and a 200 KB header value arrives intact; 200 distinct codes added 130 permanent proxyLastEmit entries in under a second. Clamping with the generated Valid() (precedent: sysmon.go:137) fixes this and the invalid-enum cast together.

Smaller: the limiter keys on code rather than sessionID + code, so rotating codes evades it while one noisy tab suppresses genuine failures in other tabs. And tunnel_connect_failed relabels the proxy's destination_blocked, telling consumers to retry something that won't recover.

Withdrawing my egressProxyAddr race note from last round: Start establishes happens-before and -race is clean.

Last thing, I'd stop calling this anti-spoofing. POST /telemetry/events already accepts a caller-supplied proxy_error, so the VM owner isn't the adversary. This is signal integrity, not a trust boundary.

[Sayan-]

Ask: pls update the pr desc since it's stale.

Final pass, focused on correctness and parity.

Follow-ups (not blocking)

Rate limiter tuning. 1s per sessionID + code means at most one URL per code per
second. In a 6h prod sample 54% of branded errors shared a (second, code) bucket. The
schema and README promise per-URL attribution, and the unsampled aggregate already exists
server-side, so per-URL detail is the whole point of this event. Worth tightening the
interval or adding resource_type to the key, and documenting the sampling either way.

Severity. proxy_error maps to OTLP ERROR next to service_crashed, while
network_loading_failed is WARN. In practice ~99% of branded proxy errors are subresources
(blocked scripts, images, fetches). Worth revisiting to WARN, or ERROR only for Document.

WebSocket handshakes. The producer's comment says the header exists for WebSocket and
subresource requests. Subresources are covered; WebSockets aren't (dispatchEvent has no
Network.webSocket* case). Either handle it or call it a documented non-goal.

Smaller:

• Unknown codes drop silently with no log, inside proxyErrorRateLimited. Worth a log line
  and a comment pinning the enum to the producer. Note TestProxyErrorRateLimit uses a nil
  logger so adding one panics it.
• code isn't promoted to an OTLP attribute. promotedAttributes does this for console
  level for the same reason; no collision.
• proxyLastEmit comment says entries stay bounded, but they aren't pruned on detach.
• Spec says status is 502; the gate is >= 500.
• README got request_id but not frame_id, loader_id, nav_seq, and the network field
  table has no proxy_error row.
• Case-insensitivity test uses source_auth_error, which nothing emits and Valid() drops.
  No test covers the < 500 gate, unknown codes through the handler, or the untracked
  nav_seq fallback. The two handler subtests only pass because they use different codes.
• Stray blank line at handlers.go:666.
• Origin-supplied X-Kernel-Proxy-Error isn't stripped by the producer, so any site can
  forge an event. This PR's own e2e test demonstrates it. Fix belongs on the producer side.

[chruffins]

thanks for the pass. on the follow-ups — quick status of what was decided / left:

deferred to the producer (not in this PR)

• origin-supplied X-Kernel-Proxy-Error isn't stripped, so a site can forge an event. that fix belongs on the metro egressproxy side (strip the header on forward), out of this repo's scope. agreed the image treats it as signal-integrity advisory, not a trust boundary.

documented non-goals

• websocket handshakes aren't classified (collector has no Network.webSocket* case). noted as a non-goal in the README; can be a follow-up if we want WS coverage.

decisions

• severity: proxy_error is now ERROR only when resource_type == "Document", WARN for subresources (the ~99% volume). untracked/empty resource_type defaults to WARN — flagging that choice in case a mid-attach top-level nav should be ERROR.
• rate limiter keys on session+code+resource_type now (1s interval); sampling documented in the code + README.

everything else from the final pass (otlp kernel.proxy_error_code attr, unknown-code logging, == 502 gate, field-table/README rows, test coverage) landed in the commits.