feat(replay): emit replay and replay vision metrics via otlp - #70535
Merged
Conversation
Dual-emits the headline session replay and replay vision operational metrics into the PostHog Metrics product alongside the existing Prometheus instrumentation, and adds new metrics for quota exhaustion, credit burn, sweep outcomes, observation end-to-end latency, side effect failures, Gemini cleanup backlog, and ingestion end-to-end lag. Emission is gated off unless OTEL_METRICS_EXPORT_URL and OTEL_METRICS_EXPORT_TOKEN are set, so nothing changes for deployments that don't opt in. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
arnohillen
requested review from
a team,
TueHaulund,
fasyy612 and
ksvat
and removed request for
a team
July 13, 2026 17:37
|
|
Applies the findings from an 8-angle review pass: - Histogram twins now derive bucket boundaries from the prom instrument (the playback twins previously fell back to OTel's default advisory buckets, breaking 1:1 dashboard parity), and twin metadata is memoized per instrument instead of re-derived per record. - Provider invalidation is unified on an epoch counter covering both fork and test reset (reset previously left factory caches bound to a dead provider). - The sweep headroom formula is defined once in constants and shared by the workflow decision and the throttled metric. - Signal side-effect failures count per activity attempt, comparable to the other effects. - Node: shared createOtlpMeterProvider, exemplar-wrapping instrument factories, and the swallowing wrapper move to common/metrics, used by the logs, session replay ingestion, and rasterizer twin modules (the rasterizer twins gain exemplar support). - The e2e lag value is sanitized once at the compute site. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Note 🤖 stamphog reviewed Gates denied for size (over the 800L ceiling), this is a cross-team change lacking a landed review, and greptile-apps and hex-security-app both have in-flight reviews (👀) that haven't completed — should not approve over active review.
Gate mechanics and policy version
|
Contributor
|
Reviews (1): Last reviewed commit: "refactor(replay): harden and dedupe the ..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| url, | ||
| token, | ||
| serviceName: process.env.OTEL_SERVICE_NAME ?? 'recording-rasterizer', | ||
| exportIntervalMillis: parseInt(process.env.OTEL_METRICS_EXPORT_INTERVAL_MS ?? '60000', 10), |
Contributor
There was a problem hiding this comment.
When OTEL_METRICS_EXPORT_INTERVAL_MS is set to an empty or non-numeric value, parseInt returns NaN and this value is passed into the metric reader during rasterizer module startup. Because initMetrics() runs before the worker's startup error handler, a bad optional metrics env var can prevent the rasterizer worker from booting.
Suggested change
| exportIntervalMillis: parseInt(process.env.OTEL_METRICS_EXPORT_INTERVAL_MS ?? '60000', 10), | |
| exportIntervalMillis: Number.isFinite(parseInt(process.env.OTEL_METRICS_EXPORT_INTERVAL_MS ?? '60000', 10)) | |
| ? parseInt(process.env.OTEL_METRICS_EXPORT_INTERVAL_MS ?? '60000', 10) | |
| : 60000, |
Comment on lines
+152
to
+154
| if receipt_created: | ||
| # Gate on the receipt so a lost-result retry can't double count the burn rate. | ||
| record_credits_consumed(inputs.scanner_type, model, credits) |
Contributor
There was a problem hiding this comment.
If this activity commits the usage receipt and then the worker crashes or Temporal loses the activity result before these metric calls run, the retry finds the observation already terminal and returns before emitting credit burn. Billing state is durable, but the new replay_vision_credits_consumed_total signal can permanently miss that usage.
Contributor
🤖 CI report
|
| File | Patch | Uncovered changed lines |
|---|---|---|
products/replay_vision/backend/temporal/activities/count_in_flight_applies.py |
75.0% | 50 |
products/replay_vision/backend/temporal/activities/call_scanner_provider.py |
80.0% | 344 |
products/replay_vision/backend/temporal/activities/emit_observation_signal.py |
85.7% | 113 |
posthog/otel_metrics.py |
92.3% | 163–164, 172–173, 176–180 |
posthog/session_recordings/session_recording_api.py |
92.3% | 795 |
posthog/test/test_otel_metrics.py |
98.4% | 29 |
🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 29345452673 -n patch-coverage), or the coverage-data block at the end of this comment.
Per-product line coverage (touched products)
| Product | Coverage | Lines |
|---|---|---|
platform_features |
██░░░░░░░░░░░░░░░░░░ 12.1% |
7 / 58 |
batch_exports |
████████░░░░░░░░░░░░ 39.6% |
8,347 / 21,104 |
demo |
███████████░░░░░░░░░ 55.2% |
1,436 / 2,601 |
warehouse_sources_queue |
████████████░░░░░░░░ 59.2% |
148 / 250 |
tasks |
█████████████░░░░░░░ 66.7% |
24,376 / 36,570 |
data_tools |
██████████████░░░░░░ 70.0% |
63 / 90 |
ai_gateway |
███████████████░░░░░ 75.0% |
9 / 12 |
data_modeling |
████████████████░░░░ 78.4% |
4,696 / 5,987 |
signals |
████████████████░░░░ 78.5% |
18,072 / 23,028 |
cdp |
████████████████░░░░ 80.6% |
3,105 / 3,851 |
wizard |
████████████████░░░░ 82.5% |
772 / 936 |
cohorts |
█████████████████░░░ 82.9% |
3,048 / 3,675 |
notebooks |
█████████████████░░░ 83.8% |
6,086 / 7,259 |
agent_platform |
█████████████████░░░ 84.1% |
3,095 / 3,678 |
actions |
█████████████████░░░ 86.6% |
717 / 828 |
engineering_analytics |
█████████████████░░░ 87.3% |
4,269 / 4,889 |
product_tours |
█████████████████░░░ 87.5% |
1,266 / 1,447 |
exports |
██████████████████░░ 88.3% |
6,857 / 7,763 |
visual_review |
██████████████████░░ 88.5% |
5,565 / 6,287 |
business_knowledge |
██████████████████░░ 88.5% |
4,400 / 4,969 |
conversations |
██████████████████░░ 88.9% |
15,924 / 17,921 |
mcp_analytics |
██████████████████░░ 89.1% |
2,485 / 2,790 |
dashboards |
██████████████████░░ 89.1% |
5,650 / 6,342 |
error_tracking |
██████████████████░░ 89.2% |
9,473 / 10,615 |
streamlit_apps |
██████████████████░░ 90.4% |
2,499 / 2,764 |
slack_app |
██████████████████░░ 90.6% |
9,460 / 10,444 |
links |
██████████████████░░ 90.6% |
183 / 202 |
marketing_analytics |
██████████████████░░ 90.7% |
11,476 / 12,646 |
product_analytics |
██████████████████░░ 91.2% |
5,652 / 6,195 |
managed_migrations |
██████████████████░░ 91.9% |
908 / 988 |
workflows |
██████████████████░░ 92.0% |
4,795 / 5,210 |
mcp_store |
██████████████████░░ 92.1% |
3,665 / 3,981 |
data_warehouse |
██████████████████░░ 92.1% |
17,282 / 18,765 |
alerts |
██████████████████░░ 92.1% |
3,389 / 3,678 |
web_analytics |
███████████████████░ 92.7% |
13,702 / 14,787 |
notifications |
███████████████████░ 92.7% |
1,026 / 1,107 |
ai_observability |
███████████████████░ 92.7% |
14,670 / 15,822 |
surveys |
███████████████████░ 92.9% |
5,660 / 6,094 |
posthog_ai |
███████████████████░ 93.2% |
1,311 / 1,407 |
tracing |
███████████████████░ 93.2% |
2,423 / 2,599 |
approvals |
███████████████████░ 93.3% |
3,395 / 3,640 |
reminders |
███████████████████░ 93.4% |
468 / 501 |
early_access_features |
███████████████████░ 93.8% |
848 / 904 |
legal_documents |
███████████████████░ 94.1% |
1,568 / 1,667 |
endpoints |
███████████████████░ 94.1% |
8,606 / 9,143 |
messaging |
███████████████████░ 94.3% |
2,366 / 2,508 |
skills |
███████████████████░ 94.4% |
2,819 / 2,987 |
revenue_analytics |
███████████████████░ 94.4% |
3,586 / 3,797 |
growth |
███████████████████░ 94.9% |
2,393 / 2,522 |
logs |
███████████████████░ 95.3% |
9,429 / 9,895 |
experiments |
███████████████████░ 95.6% |
24,057 / 25,166 |
replay_vision |
███████████████████░ 95.6% |
12,785 / 13,369 |
feature_flags |
███████████████████░ 96.1% |
14,533 / 15,127 |
warehouse_sources |
███████████████████░ 96.1% |
212,352 / 221,024 |
annotations |
███████████████████░ 96.2% |
732 / 761 |
user_interviews |
███████████████████░ 96.4% |
2,242 / 2,325 |
access_control |
███████████████████░ 96.8% |
849 / 877 |
customer_analytics |
███████████████████░ 97.3% |
7,277 / 7,481 |
data_catalog |
███████████████████░ 97.3% |
1,288 / 1,324 |
analytics_platform |
████████████████████ 98.2% |
2,098 / 2,137 |
metrics |
████████████████████ 98.3% |
2,363 / 2,403 |
live_debugger |
████████████████████ 99.2% |
613 / 618 |
field_notes |
████████████████████ 99.4% |
158 / 159 |
Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.
DanielVisca
self-requested a review
July 14, 2026 15:57
DanielVisca
approved these changes
Jul 14, 2026
DanielVisca
left a comment
DanielVisca
left a comment
Contributor
There was a problem hiding this comment.
I'm adding to the sdk for node so it will be easier to instrument in the future. This looks good for OTLP
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Session replay and replay vision are well instrumented in Prometheus/Grafana, but none of that signal reaches the PostHog Metrics product. We can't watch our own pipelines' health inside PostHog, can't correlate operational metrics with logs and traces there, and we're not dogfooding Metrics from the replay side at all. A few high-value signals also don't exist anywhere yet, notably quota exhaustion and credit burn for replay vision (both launch-relevant for billing) and ingestion end-to-end lag for session replay.
Changes
Every mirrored metric keeps its exact Prometheus name and label set, following the convention the logs team established with
logs_ingestion_*twins, so dashboards translate 1:1 between sinks. Emission is off unlessOTEL_METRICS_EXPORT_URLandOTEL_METRICS_EXPORT_TOKENare set (same env pair the Node exporter already uses), so nothing changes for deployments that don't opt in.flowchart LR RV[replay vision temporal activities] --> H[record_* helpers] PB[playback API] --> H H --> PROM[Prometheus registry / Grafana] H --> PY[posthog/otel_metrics.py] ING[blob ingestion v2 + rasterizer] --> NPROM[prom-client / Grafana] ING --> NOTEL[OTel twins] PY --> CL[capture-logs /i/v1/metrics] NOTEL --> CL CL --> MP[PostHog Metrics product] classDef phBlue fill:#1d4aff,stroke:#1d4aff,color:#fff; classDef phYellow fill:#f9bd2b,stroke:#f9bd2b,color:#000; classDef phGray fill:#e5e7eb,stroke:#c7ccd1,color:#000; class H,PY,NOTEL phBlue; class MP phYellow; class PROM,NPROM,CL phGray;New shared infrastructure
posthog/otel_metrics.py: a Python twin ofnodejs/src/common/metrics/otel-metrics.ts. Lazy, per-process (PID-keyed) provider init so preforking servers (gunicorn, celery) get a live exporter thread per worker, and aOtelInstrumentFactorywithrecord_*_twinmethods that derive the OTLP metric's name and description from the Prometheus instrument itself, so the two sinks can't drift. Twin recording swallows all errors.OTEL_METRICS_EXPORT_URL/OTEL_METRICS_EXPORT_TOKEN/OTEL_METRICS_EXPORT_INTERVAL_MS, matching the Node names so one env pair configures both runtimes.Replay vision (
products/replay_vision/backend/temporal/metrics.pynow exposes dual-emitrecord_*helpers used by all call sites)Mirrored:
replay_vision_observations_total,replay_vision_failure_kinds_total,replay_vision_ineligible_kinds_total,replay_vision_activity_duration_seconds,replay_vision_provider_call_seconds.New (Prometheus + OTLP):
replay_vision_quota_exhausted_skips_total{scanner_type}replay_vision_credits_consumed_total{scanner_type, model}replay_vision_sweep_outcomes_total{outcome}+replay_vision_sweep_candidates_totalreplay_vision_observation_e2e_seconds{scanner_type}replay_vision_side_effect_failures_total{effect}replay_vision_gemini_cleanup_backlogSession replay ingestion (Node,
otel-metrics.tstwin module wired insideSessionBatchMetricsso no call sites changed)Mirrored: sessions/events flushed, bytes written, all drop counters (missing retention, restrictions, rate limits, blocklist), S3 upload errors/timeouts/latency. New:
recording_blob_ingestion_v2_e2e_lag_seconds, per-session staleness at flush (wall clock minus the newest flushed event timestamp), which is what "why can't I see my recording yet" tickets are about.Rasterizer: mirrors
recording_rasterizer_activities_total,recording_rasterizer_activity_duration_seconds,recording_rasterizer_errors_total. It gets its own env-driveninitMetricsbecause the shared one evaluatesdefaultConfigat import, which throws without Postgres env vars the rasterizer deployment doesn't have.Playback API (Python): the snapshot stage histograms (gather sources, fetch blocks, stream to client) and the throttle counter now dual-emit through two small helpers.
Note
No behavior changes when the env vars are unset: all Prometheus metrics keep identical names and labels, and OTLP recording is a no-op against a noop meter.
How did you test this code?
Automated tests, all run locally:
posthog/test/test_otel_metrics.py(new): catches gating inversion (unconfigured must be a safe no-op), provider construction breakage against the real SDK signatures, and the_totalsuffix restoration in counter twins, which prometheus_client strips internally and would silently rename every mirrored counter.products/replay_vision/backend/tests/test_metrics.py(new): one parameterized call perrecord_*helper. These run inside activity error handlers where a prom label mismatch raises at runtime, and no existing test exercises the new helpers.posthog/session_recordings/test/test_snapshot_stage_metrics.py(new): the timed-stage context manager must still observe when the body raises, and must propagate the exception.nodejs/src/ingestion/pipelines/sessionreplay/otel-metrics.test.ts(new): mirrors the logs twin test. Locks in the lazy-instrument startup order, name/attribute emission, zero-count skipping, negative-lag clamping, prom bucket parity, and that a throwing OTel SDK is swallowed (in production those errors are invisible by design, so a test is the only place they surface).observeE2eLag), full nodejs typecheck and lint pass. Two pre-existing local failures (*.integration.test.tsneeding a localtest_personsDB, and astorage.test.tslogger flake) reproduce identically on master without this change.Not done: no end-to-end verification against a live capture-logs ingest; the export path is exercised through the SDK pipeline with a capturing exporter instead.
Automatic notifications
Docs update
Internal operational instrumentation only, no user-facing docs affected. The new env vars are deployment config wired via charts.
🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Authored with Claude Code (Claude Fable 5) in an interactive session directed by the assignee. Skills invoked:
/writing-tests,/query-clickhouse-via-metabase(used to confirm no replay metrics exist in the Metrics product yet in either region before building).Decisions along the way:
initMetrics; review caught that it evaluatesdefaultConfigat import and throws without Postgres env vars, which would have crashed the rasterizer at boot, so it got a self-contained env-driven init instead.record_*_twinmethods that derive both from the Prometheus instrument, with the counter_totalsuffix restoration handled once and locked by a test.🤖 Generated with Claude Code