Skip to content

PostHog error tracking for the ORB self-host server (swap the src/selfhost/sentry.ts sink; cron-monitor heartbeat design) #8287

Description

@JSONbored

Part of epic #8286 (Phase 1 — Errors). The largest surface, and the pattern-setter for the other Node-side swaps.

Current state

src/selfhost/sentry.ts (#1468) is the self-host ORB server's opt-in error tracking — a complete no-op unless the operator sets SENTRY_DSN. It carries five responsibilities that all need a PostHog answer, not just captureException:

  1. Capture + redactioncaptureException behind a deep scrub pipeline (SECRET_KEY/PAYLOAD_KEY/SECRET_VALUE regexes, PUBLIC_LOCAL_PATH_SCRUB_PATTERN, PUBLIC_UNSAFE_TERMS), operational tags (SENTRY_OPERATIONAL_TAG_KEYS), subsystem taxonomy (SENTRY_OPERATIONAL_SUBSYSTEMS), hashed installation ids.
  2. Structured-log forwardingsrc/server.ts wraps console sinks so every level:"error"/"fatal" JSON log line becomes an event without per-site wiring.
  3. Severity thresholdsSENTRY_MIN_SEVERITY / SENTRY_REPO_MIN_SEVERITY (src/services/severity-threshold.ts) gate what gets reported.
  4. Cron monitorscaptureCheckIn against five monitors (SENTRY_MONITOR_NAMES: scheduled-loop, orb-export, orb-relay-drain, orb-relay-register, queue-dead-letter-revive) from sqlite-queue.ts/pg-queue.ts/monitored-work.ts/orb-collector.ts. PostHog has no cron-monitor product — this needs a real design: emit a snake_case heartbeat event per check-in and alert on absence via a PostHog insight alert (no-data condition), preserving the "silent death is an alert" property.
  5. OpenTelemetry bridgebuildSentryOpenTelemetryBridge() feeds spans to Sentry when configured. Out of scope here: the bridge stays Sentry-side until the decommission issue decides its fate (tracing deferral is its own sub-issue).

Deliverables

  • A PostHog sink for (1)–(3): $exception capture via posthog-node (already a root dependency), the full scrub pipeline ported and applied to event properties before capture, severity thresholds preserved unchanged.
  • Opt-in posture preserved exactly: no-op unless the operator configures a PostHog key; lazy import so the Worker bundle and unconfigured self-hosts pay zero cost (same posture the Sentry module documents).
  • Env-var decision implemented and documented: how the existing MCP-scoped POSTHOG_API_KEY/POSTHOG_HOST (feat(mcp): add a typed PostHog wrapper module for src/mcp/server.ts (remote) #6235) relate to error tracking (recommend: same project key activates both, with the MCP allowlist untouched) — src/env.d.ts docs + npm run selfhost:env-reference regenerated.
  • Cron-monitor replacement design implemented per (4), with the alert side documented for the dashboards sub-issue to wire.
  • Sentry path untouched and still working (parallel-run principle) — both sinks active simultaneously when both are configured.
  • Tests: capture redaction parity (reuse the existing sentry test fixtures' worst-case payloads), threshold gating, heartbeat emission, no-op-when-unconfigured.

Boundaries

Resources

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions