feat(eng-analytics): emit CI signals to Signals - #67117
Conversation
|
Size Change: +13.8 kB (+0.02%) Total Size: 70.2 MB 📦 View Changed
ℹ️ View Unchanged
|
Query snapshots: Backend query snapshots updatedChanges: 3 snapshots (3 modified, 0 added, 0 deleted) What this means:
Next steps:
|
|
Reviews (1): Last reviewed commit: "test(mcp): update unit test snapshots" | Re-trigger Greptile |
There was a problem hiding this comment.
Huh, looks like fields with choices are a bad choice here when they required every integration to do this migration)
There was a problem hiding this comment.
Oh yeah... That's not ideal 😢
Adapts the PR to the signals taxonomy refactor (backend-owned enums/contracts, schema-signals.ts removed), the mandatory workflow_health run_scope kwarg, and the 0054 migration collision (regenerated as 0055). Generated-By: PostHog Code Task-Id: 063a5989-b314-423a-9f9e-c9235173a37b
🤖 CI report
|
| File | Size | Δ vs base |
|---|---|---|
posthog-app/src/scenes/inbox/InboxScene.js |
689.8 KiB | 🔺 +6.3 KiB (+0.9%) |
Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report
✅ Eager graph — within budget
How much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy import() / React.lazy chunks are not counted.
| Root | Eager (shipped) | Δ vs base | Budget |
|---|---|---|---|
entry (logged-out pages, app bootstrap)src/index.tsx |
1.24 MiB · 22 files | no change | ███░░░░░░░ 27.5% of 4.51 MiB |
authenticated shell (every logged-in page)src/scenes/AuthenticatedShell.tsx |
8.25 MiB · 3,014 files | 🔺 +663 B (+0.0%) | ████████░░ 84.9% of 9.71 MiB |
🟢 node_modules/monaco-editor/ stays out of src/index.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 node_modules/monaco-editor/ stays out of src/scenes/AuthenticatedShell.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
Largest files eagerly shipped from src/index.tsx
| Size | File |
|---|---|
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 24.6 KiB | ../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js |
| 6.3 KiB | ../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js |
| 4.5 KiB | ../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js |
| 3.9 KiB | ../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js |
| 1.4 KiB | ../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js |
| 1.3 KiB | src/RootErrorBoundary.tsx |
| 912 B | ../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js |
| 789 B | src/scenes/ChunkLoadErrorBoundary.tsx |
| 762 B | src/index.tsx |
Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx
| Size | File |
|---|---|
| 281.5 KiB | ../node_modules/.pnpm/posthog-js@1.407.1/node_modules/posthog-js/dist/rrweb.js |
| 267.7 KiB | ../node_modules/.pnpm/@posthog+icons@0.38.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js |
| 236.0 KiB | src/taxonomy/core-filter-definitions-by-group.json |
| 226.1 KiB | ../node_modules/.pnpm/posthog-js@1.407.1/node_modules/posthog-js/dist/module.js |
| 167.1 KiB | src/queries/validators.js |
| 154.3 KiB | ../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js |
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 105.8 KiB | src/lib/api.ts |
| 94.0 KiB | ../packages/quill/packages/quill/dist/index.js |
| 93.3 KiB | ../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js |
Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479
✅ Toolbar bundle — eager 2.18 MiB within budget
What the toolbar ships to customer pages, measured from the esbuild output (minified, post-tree-shake). The eager set is the entry plus everything statically imported from it — fetched before any feature runs; deferred chunks load lazily. The eager guardrail is 5.72 MiB. Each output file must also stay below 10 MB, where CloudFront stops compressing it. The module boundary is enforced separately by check-toolbar-graph.
| Metric | Size | Δ vs base | Budget |
|---|---|---|---|
| Eager (shipped) entry + static imports |
2.18 MiB · 17 files | no change | ████░░░░░░ 38.1% of 5.72 MiB |
| Deferred (lazy) | 2.07 MiB · 33 files | no change | n/a — loads on demand |
Loader dist/toolbar.js |
1.1 KiB | no change | █░░░░░░░░░ 5.8% of 19.5 KiB |
Largest eagerly-shipped chunks
| Size | File |
|---|---|
| 714.7 KiB | dist/toolbar/toolbar-app-MKBF64XP.css |
| 545.3 KiB | dist/toolbar/chunk-chunk-HYYFIAXK.js |
| 484.2 KiB | dist/toolbar/chunk-chunk-QS5AHYGW.js |
| 133.6 KiB | dist/toolbar/chunk-chunk-3HAV52CR.js |
| 131.8 KiB | dist/toolbar/chunk-chunk-T5KY5WYR.js |
| 71.0 KiB | dist/toolbar/toolbar-app-4WXEV7ZF.js |
| 69.0 KiB | dist/toolbar/chunk-chunk-27JL52RE.js |
| 35.6 KiB | dist/toolbar/chunk-chunk-EL3T2QEA.js |
| 20.9 KiB | dist/toolbar/chunk-chunk-HSIJKX5O.js |
| 12.2 KiB | dist/toolbar/chunk-chunk-PIK3PADE.js |
Posted automatically by check-toolbar-size · sizes are toolbar output bytes (shipped, post-tree-shake) from the esbuild metafile
✅ Dist folder size — 🟢 -8.1 KiB (-0.0%)
Total size of the built frontend/dist folder (all assets), compared against the base branch.
Total: 1359.28 MiB · 🟢 -8.1 KiB (-0.0%)
ℹ️ MCP UI apps size — 32 app(s), 17065.5 KB JS
Built size of each MCP UI app (main.js + styles.css).
| App | JS | CSS |
|---|---|---|
| debug | 599.5 KB | 187.7 KB |
| action | 457.8 KB | 187.7 KB |
| action-list | 564.3 KB | 187.7 KB |
| cohort | 456.8 KB | 187.7 KB |
| cohort-list | 563.3 KB | 187.7 KB |
| email-template | 456.6 KB | 187.7 KB |
| error-details | 472.4 KB | 187.7 KB |
| error-issue | 457.5 KB | 187.7 KB |
| error-issue-list | 564.2 KB | 187.7 KB |
| experiment | 561.5 KB | 187.7 KB |
| experiment-list | 565.1 KB | 187.7 KB |
| experiment-results | 563.2 KB | 187.7 KB |
| feature-flag | 567.1 KB | 187.7 KB |
| feature-flag-list | 570.9 KB | 187.7 KB |
| feature-flag-testing | 461.0 KB | 187.7 KB |
| insight-actors | 562.1 KB | 187.7 KB |
| invite-email-preview | 456.0 KB | 187.7 KB |
| llm-costs | 559.5 KB | 187.7 KB |
| session-recording | 458.6 KB | 187.7 KB |
| session-summary | 463.9 KB | 187.7 KB |
| survey | 458.4 KB | 187.7 KB |
| survey-global-stats | 562.2 KB | 187.7 KB |
| survey-list | 565.0 KB | 187.7 KB |
| survey-stats | 562.2 KB | 187.7 KB |
| trace-span | 457.2 KB | 187.7 KB |
| trace-span-list | 564.2 KB | 187.7 KB |
| workflow | 457.1 KB | 187.7 KB |
| workflow-list | 563.7 KB | 187.7 KB |
| loops-review | 461.2 KB | 187.7 KB |
| query-results | 745.5 KB | 187.7 KB |
| render-ui | 826.2 KB | 187.7 KB |
| visual-review-snapshots | 461.6 KB | 187.7 KB |
⚠️ Backend snapshots — 11 updated (11 modified, 0 added, 0 deleted)
Query snapshots: Backend query snapshots updated
Changes: 11 snapshots (11 modified, 0 added, 0 deleted)
What this means:
- Query snapshots have been automatically updated to match current output
- These changes reflect modifications to database queries or schema
Next steps:
- Review the query changes to ensure they're intentional
- If unexpected, investigate what caused the query to change
⚠️ Backend coverage — 88.0% of changed backend lines covered — 89 uncovered
🧪 Backend test coverage
Patch coverage — changed backend lines (products + core): ██████████████████░░ 88.0% (696 / 785)
| File | Patch | Uncovered changed lines |
|---|---|---|
products/signals/backend/facade/api.py |
0.0% | 72, 78–82 |
products/engineering_analytics/backend/facade/temporal.py |
0.0% | 14–15 |
products/engineering_analytics/backend/logic/signals/temporal.py |
0.0% | 7, 14–15 |
products/engineering_analytics/backend/logic/signals/schedule.py |
0.0% | 8, 10, 12, 22, 24–25, 28–29, 41–42, 44 |
products/engineering_analytics/backend/logic/signals/coordinator.py |
57.1% | 75–83, 88, 94, 99–100, 107, 138–139, 145–146, 149, 153–154, 160–162, 168, 174, 183, 193, 200, 209, 224–225, 237–238, 245–247, 260, 264, 269–274, 276–284 |
products/engineering_analytics/backend/facade/api.py |
75.0% | 83 |
products/engineering_analytics/backend/presentation/views/ci_signals.py |
92.0% | 36–37 |
products/engineering_analytics/backend/logic/signals/detect.py |
92.9% | 39–40 |
products/engineering_analytics/backend/logic/queries/workflow_flakiness.py |
95.8% | 64 |
products/engineering_analytics/backend/logic/signals/detectors.py |
96.2% | 109, 176, 178, 268 |
products/engineering_analytics/backend/logic/ci_signals_config.py |
98.7% | 124 |
products/engineering_analytics/backend/tests/test_ci_signals.py |
99.3% | 556–557 |
🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 30021834006 -n patch-coverage), or the coverage-data block at the end of this comment.
Per-product line coverage (touched products)
| Product | Coverage | Lines |
|---|---|---|
platform_features |
██░░░░░░░░░░░░░░░░░░ 12.1% |
7 / 58 |
batch_exports |
████████░░░░░░░░░░░░ 39.6% |
8,412 / 21,249 |
demo |
███████████░░░░░░░░░ 56.2% |
1,497 / 2,663 |
warehouse_sources_queue |
████████████░░░░░░░░ 59.2% |
148 / 250 |
tasks |
██████████████░░░░░░ 69.2% |
29,524 / 42,687 |
data_tools |
██████████████░░░░░░ 70.0% |
63 / 90 |
ai_gateway |
███████████████░░░░░ 75.0% |
9 / 12 |
signals |
████████████████░░░░ 79.8% |
21,630 / 27,098 |
cdp |
████████████████░░░░ 81.0% |
3,144 / 3,881 |
data_modeling |
█████████████████░░░ 82.7% |
5,547 / 6,708 |
wizard |
█████████████████░░░ 84.8% |
935 / 1,102 |
notebooks |
█████████████████░░░ 85.3% |
7,275 / 8,531 |
agent_platform |
█████████████████░░░ 86.4% |
3,807 / 4,405 |
actions |
█████████████████░░░ 86.6% |
717 / 828 |
cohorts |
██████████████████░░ 87.8% |
4,488 / 5,114 |
product_tours |
██████████████████░░ 87.9% |
1,303 / 1,482 |
exports |
██████████████████░░ 88.4% |
6,949 / 7,861 |
data_warehouse |
██████████████████░░ 88.9% |
11,887 / 13,378 |
business_knowledge |
██████████████████░░ 89.0% |
4,391 / 4,936 |
conversations |
██████████████████░░ 89.3% |
16,820 / 18,834 |
engineering_analytics |
██████████████████░░ 89.4% |
6,371 / 7,130 |
dashboards |
██████████████████░░ 89.4% |
5,983 / 6,693 |
visual_review |
██████████████████░░ 89.4% |
5,818 / 6,505 |
error_tracking |
██████████████████░░ 89.7% |
10,174 / 11,342 |
alerts |
██████████████████░░ 89.9% |
4,054 / 4,508 |
early_access_features |
██████████████████░░ 90.1% |
1,031 / 1,144 |
mcp_analytics |
██████████████████░░ 90.1% |
2,763 / 3,065 |
streamlit_apps |
██████████████████░░ 90.4% |
2,501 / 2,767 |
links |
██████████████████░░ 90.6% |
183 / 202 |
slack_app |
██████████████████░░ 90.7% |
9,028 / 9,951 |
marketing_analytics |
██████████████████░░ 91.0% |
11,792 / 12,964 |
stamphog |
██████████████████░░ 91.1% |
4,056 / 4,450 |
mcp_store |
██████████████████░░ 91.9% |
4,257 / 4,634 |
notifications |
███████████████████░ 92.6% |
1,011 / 1,092 |
product_analytics |
███████████████████░ 92.7% |
5,811 / 6,268 |
ai_observability |
███████████████████░ 92.7% |
14,995 / 16,168 |
surveys |
███████████████████░ 93.0% |
5,730 / 6,163 |
web_analytics |
███████████████████░ 93.1% |
14,197 / 15,257 |
managed_migrations |
███████████████████░ 93.1% |
1,424 / 1,530 |
posthog_ai |
███████████████████░ 93.2% |
1,326 / 1,422 |
approvals |
███████████████████░ 93.3% |
3,437 / 3,682 |
reminders |
███████████████████░ 93.4% |
468 / 501 |
workflows |
███████████████████░ 93.6% |
6,390 / 6,826 |
legal_documents |
███████████████████░ 94.1% |
1,568 / 1,667 |
endpoints |
███████████████████░ 94.1% |
8,640 / 9,177 |
messaging |
███████████████████░ 94.3% |
2,682 / 2,845 |
tracing |
███████████████████░ 94.5% |
2,670 / 2,826 |
skills |
███████████████████░ 94.5% |
2,893 / 3,061 |
revenue_analytics |
███████████████████░ 94.6% |
3,651 / 3,858 |
review_hog |
███████████████████░ 94.7% |
6,814 / 7,199 |
logs |
███████████████████░ 95.4% |
9,935 / 10,416 |
experiments |
███████████████████░ 95.7% |
25,036 / 26,159 |
growth |
███████████████████░ 96.1% |
3,245 / 3,376 |
annotations |
███████████████████░ 96.2% |
732 / 761 |
replay_vision |
███████████████████░ 96.2% |
15,094 / 15,691 |
feature_flags |
███████████████████░ 96.5% |
17,110 / 17,734 |
user_interviews |
███████████████████░ 96.5% |
2,638 / 2,734 |
access_control |
███████████████████░ 96.8% |
851 / 879 |
customer_analytics |
███████████████████░ 97.1% |
9,047 / 9,317 |
warehouse_sources |
███████████████████░ 97.1% |
329,131 / 338,870 |
data_catalog |
███████████████████░ 97.5% |
2,368 / 2,429 |
analytics_platform |
████████████████████ 98.0% |
2,153 / 2,197 |
metrics |
████████████████████ 98.2% |
2,491 / 2,536 |
pulse |
████████████████████ 98.4% |
2,017 / 2,049 |
live_debugger |
████████████████████ 99.2% |
613 / 618 |
field_notes |
████████████████████ 99.4% |
158 / 159 |
Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.
✅ Django migration risk — no migrations to analyze
No Django migrations need risk analysis.
|
Generated-By: PostHog Code Task-Id: 063a5989-b314-423a-9f9e-c9235173a37b
Generated-By: PostHog Code Task-Id: 063a5989-b314-423a-9f9e-c9235173a37b
# Conflicts: # products/signals/backend/migrations/max_migration.txt # products/signals/frontend/generated/api.zod.ts # services/mcp/src/api/generated.ts # services/mcp/src/generated/signals/api.ts # services/mcp/tests/unit/__snapshots__/tool-schemas/inbox-source-configs-create.json # services/mcp/tests/unit/__snapshots__/tool-schemas/inbox-source-configs-partial-update.json # services/mcp/tests/unit/__snapshots__/tool-schemas/inbox-source-configs-update.json
# Conflicts: # tach.toml
…gnals # Conflicts: # products/engineering_analytics/backend/facade/api.py # products/engineering_analytics/backend/logic/queries/workflow_health.py # products/engineering_analytics/backend/presentation/views.py # products/engineering_analytics/frontend/generated/api.ts # products/engineering_analytics/mcp/tools.yaml # products/signals/backend/migrations/max_migration.txt
…gnals-base # Conflicts: # products/signals/backend/contracts.py # products/signals/backend/enums.py # products/signals/backend/migrations/max_migration.txt # products/signals/frontend/generated/api.schemas.ts # products/signals/frontend/generated/api.zod.ts # services/mcp/src/api/generated.ts # services/mcp/src/generated/signals/api.ts # services/mcp/tests/unit/__snapshots__/tool-schemas/inbox-source-configs-create.json # services/mcp/tests/unit/__snapshots__/tool-schemas/inbox-source-configs-partial-update.json # services/mcp/tests/unit/__snapshots__/tool-schemas/inbox-source-configs-update.json
…gnals # Conflicts: # frontend/src/scenes/inbox/components/config/AgentsRoster.tsx # frontend/src/scenes/inbox/signalSourcesLogic.ts
…gnals # Conflicts: # products/engineering_analytics/backend/presentation/serializers.py # products/engineering_analytics/backend/presentation/views.py # products/engineering_analytics/backend/tests/test_logic.py # products/engineering_analytics/backend/tests/test_presentation.py
…gnals # Conflicts: # products/engineering_analytics/backend/presentation/views/__init__.py # products/engineering_analytics/backend/tests/test_presentation.py
Note
Stacked on feat(signals): ci signal taxonomy and emit plumbing — merge that first; this PR then auto-retargets to master.
Problem
PostHog Code can't see what happens to a PR between opened and deployed. Engineering analytics owns the curated CI/PR read layer; this turns its highest-value CI conditions into Signals the self-driving inbox can group, research, and autonomously fix.
Changes
flowchart LR W[GitHub warehouse tables] --> D{{CI detectors}} S([Hourly Temporal coordinator]) --> D D --> E[Signals facade emit_signal] E --> I[Inbox: group, research, auto-start PR] classDef phBlue fill:#1d4aff,stroke:#1d4aff,color:#fff; classDef phYellow fill:#f9bd2b,stroke:#f9bd2b,color:#000; classDef phGray fill:#e5e7eb,stroke:#c7ccd1,color:#000; class D,E phBlue; class W phGray; class S,I phYellow;ci_flaky_check(job failed then passed on a rerun of the same commit),ci_broken_default_branch(default branch red),ci_duration_regression(p95 up both relative and absolute)WAREHOUSE_SOURCE_SETUPregistration per source) — GitHub now backs two sources, so the old product-keyed maps and the duplicatedDataSourceSetupcopies collapse into one registry and one shared componentCaution
Known caveat to the fail-closed claims above: only the bundle enable path re-anchors
created_by. A laterconfigPATCHthrough the generic signals source-config endpoint keeps the original enabler as authorizer, so an editor's snapshot is read under the enabler's access until the bundle is re-enabled. Changing sharedPATCHsemantics is a Signals-owned decision — see the caution on the base PR.Inbox report with the three CI signal cards:
"GitHub CI" card in the signal sources roster (flag-gated, armed):
How did you test this code?
dry_runshadow (logged, wrote nothing) → real emit → one ledger row → idempotent re-sweep (no re-emission) → roster card rendered. Local verification stops at theemit_signalboundary — grouping needs an LLM path the dev stack lacksReproduce locally (runbook)
python manage.py seed_engineering_analytics --team-id 1is_ai_data_processing_approved, anengineering-analyticsflag at 100%, thenupdate_ci_signals_config(team=..., enabled=True, created_by_id=...)(snapshots sources); setdry_runin the flaky-check row's config to shadow firstfailureclones of recent attempt-2successrows for one(workflow, job)into the seeded jobs CSV in object storage — same repo/workflow/job, run present in the runs table, duration >10s, within 7dtemporal --address temporal:7233 workflow start --type engineering-analytics-ci-signals-coordinator --task-queue development-task-queue --input '{}'(from the temporal admin-tools container)ci_signals_dry_runin temporal-worker logs; flipdry_runoff, re-trigger, checkSignalEmissionRecord; trigger once more to watch the ledger suppress re-emissionBacktest against production CI data
Replicated each detector's predicate against the real
github.eng_analytics.*tables in project 2 before review. It found three semantic bugs, now fixed:ci_flaky_checkfirst sweepVisual regression tests pass(140),Semgrep Checks Pass(110) — required-check aggregators, not testsci_broken_default_branchrate guardcancelled/skippedpinned E2E (0.20), Dagster (0.13), Storybook (0.17) under the 0.5 floor foreversource_idis now week-keyed per job (matchingci_duration_regression); 536 sightings fold in asflaky_count+ one worked example.* Passgate fails only because a job it gates failed. Measured: aggregators settle in 3-5s, real jobs 60s+. Filter atNO_OP_JOB_MAX_SECONDS = 10, mirroringNO_OP_RUN_MAX_SECONDS.cancelledis notfailure. Newconclusive_run_countonWorkflowHealthItem; also makessuccess_rateinterpretable on the read surface.description.split("\n")[0]).Emission mechanics (follows conversations + scout)
source_idinSignalEmissionRecord(unique, no expiry) and skips it next sweep — the conversations pattern; replaces the retention-boundidempotency_key(removed in the base PR). Recorded only after a successful emit, and the sweep gates org AI-approval up front, so a skipped or errored emit is retried next sweep rather than suppressed for its window.source_ids — one signal per standing condition per week, so an hourly sweep never mints a new signal (or ledger row) per completed run. Keys percent-encode GitHub-controlled name components and cap at the ledger column length, so crafted workflow/job names can neither collide two conditions into one key nor overflow the ledger and re-emit forever.dry_runconfig flag detects and logs but emits nothing — the shadow step before a team's findings reach an inbox, mirroring the scout'semitflag.emit_signalfacade, the shape anomaly investigation and replay_vision use for internal products;products/signals/backend/emission/registers per-row warehouse-sync emitters (github/linear/jira/zendesk records), which these scheduled detectors are not. If Signals would rather route scheduled sources through a different surface (an external-emitter API has been floated), happy to adapt.Known limitation:
ci_duration_regressionneeds 14d of history; the GitHub sync holds ~4d, so it emits nothing until history accumulates.Automatic notifications
Docs update
None, alpha and flag-gated.
🤖 Agent context
Autonomy: Human-driven (agent-assisted)
query_workflow_health, so detection and the MCP read surface can't drift