Skip to content

feat(data-warehouse): attribute posthog code-created sources as created_via=self_driving - #70661

Merged
Gilbert09 merged 6 commits into
masterfrom
tom/dwh-self-driving-created-via
Jul 14, 2026
Merged

feat(data-warehouse): attribute posthog code-created sources as created_via=self_driving#70661
Gilbert09 merged 6 commits into
masterfrom
tom/dwh-self-driving-created-via

Conversation

@Gilbert09

Copy link
Copy Markdown
Member

Problem

#70242 made wizard-created warehouse sources distinguishable via created_via=wizard. PostHog Code is the next surface getting source onboarding in its UI (the self-driving area of the app), and those sources would currently land as plain mcp - indistinguishable from any other MCP client.

Changes

Extends the same server-side derivation to PostHog Code:

  • New SELF_DRIVING = "self_driving" value on ExternalDataSource.CreatedVia (+ choices-only migration, no DDL).
  • The create path's single wizard upgrade becomes a transport map: a machine-injected created_via=mcp is upgraded to wizard for wizard transports and to self_driving when get_event_source resolves EventSource.POSTHOG_CODE (the existing posthog/code / *.hog.dev UA detection, which also matches the ; for posthog/code suffix the MCP server appends when proxying).
  • Same anti-spoof posture as wizard: self_driving is not accepted from callers (not in the create serializer's choices - a 400), and explicit web/api values are never rewritten. The MCP tool still injects mcp itself; agents can't set the field.
  • Help texts updated on both serializers.

Note

Generated OpenAPI types are not in this PR - local codegen isn't available in this environment, so the check-openapi-types CI job will regenerate and auto-commit them to this branch (same-repo PR path).

How did you test this code?

  • Extended the existing parameterized transport test (test_create_external_data_source_transport_user_agent_upgrades_mcp_created_via) rather than adding a new one: new cases cover posthog/code UA, the MCP-server-proxied ; for posthog/code UA shape, a plain MCP UA staying mcp, and explicit web/api staying untouched under a PostHog Code UA. The regression this catches: PostHog Code-driven creates silently reverting to mcp, or the proxied UA shape not matching.
  • Added self_driving to the reject-invalid-created_via cases - a caller must not be able to self-label as PostHog Code-created.
  • Ran the 18 created_via API tests locally (Python 3.13 venv): all pass.
  • Not run locally: OpenAPI codegen (see note above) and the wider backend suite (CI covers both).

Automatic notifications

  • Publish to changelog?
  • Alert Sales and Marketing teams?

Docs update

No user-facing docs affected - the field's API help text is the documentation surface and is updated in this PR.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Claude Code implemented this by mirroring the merged wizard attribution PR (#70242). Checked prod first: no posthog_code-sourced warehouse creations exist yet, so this is preparatory for the app's onboarding flow and there is no backfill concern. Decisions: kept the upgrade rule scoped to machine-injected mcp values only (consistent with the wizard's anti-spoof rationale) rather than also upgrading defaulted api values, so a future direct-REST path in PostHog Code would need to opt in explicitly; converted the single wizard conditional into a transport map rather than chained ifs. Skills invoked: /django-migrations (choices-only AlterField, no DDL), /writing-tests (extended existing parameterized tests instead of new functions). The migration was hand-written mirroring the wizard's 0065 because the local default venv can't run master's manage.py; it is byte-equivalent to what makemigrations produces for a choices change.

@Gilbert09 Gilbert09 self-assigned this Jul 14, 2026
@pr-assigner-resolver-posthog
pr-assigner-resolver-posthog Bot requested review from a team July 14, 2026 09:57
@Gilbert09 Gilbert09 added the stamphog Request AI approval (no full review) label Jul 14, 2026
@stamphog

stamphog Bot commented Jul 14, 2026

Copy link
Copy Markdown

Note

🤖 stamphog reviewed 7f22bd150cdab6bf13e7bd7941355abb4d4991b5 — verdict: REFUSED

Gates denied: this PR includes a migration file and was classified above the auto-approval tier; it also touches a data model field (created_via) across team boundaries with no reviews or independent assurance yet.

  • 👍 on the PR from hex-security-app[bot].
  • Gate denial: matches deny-list for migrations and is classified T2-never (102L/5F)
  • Cross-team change (touches team-data-stack, team-managed-warehouse, team-warehouse-sources) with no reviews or comments for independent assurance
  • Data model change (new CreatedVia choice + migration) is risky territory requiring human sign-off
Gate mechanics and policy version
Gate Result
prerequisites all clear
deny-list matches: migrations
size 65L, 3F substantive, 102L/5F incl. docs/generated/snapshots — within ceiling
tier classified as T2-never: T2-never (102L, 5F, single-area, feat)
stamphog 2.0.0b3 .stamphog/policy.yml @ f20ba64 · reviewed head 7f22bd1

@stamphog stamphog Bot removed the stamphog Request AI approval (no full review) label Jul 14, 2026
@trunk-io

trunk-io Bot commented Jul 14, 2026

Copy link
Copy Markdown

Static BadgeStatic BadgeStatic BadgeStatic Badge

View Full Report ↗︎Docs

@github-actions

github-actions Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Migration SQL Changes

Hey 👋, we've detected some migrations on this PR. Here's the SQL output for each migration, make sure they make sense:

products/warehouse_sources/backend/migrations/0069_alter_externaldatasource_created_via.py

BEGIN;
--
-- Alter field created_via on externaldatasource
--
-- (no-op)
COMMIT;

Last updated: 2026-07-14 12:32 UTC (0a27e88)

@github-actions

github-actions Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

🔍 Migration Risk Analysis

We've analyzed your migrations for potential risks.

Summary: 0 Safe | 1 Needs Review | 0 Blocked

⚠️ Needs Review

May have performance impact

warehouse_sources.0069_alter_externaldatasource_created_via
  └─ #1 ⚠️ AlterField
     Field alteration may cause table locks or data loss (check if changing type or constraints)
     model: externaldatasource, field: created_via, field_type: CharField

Last updated: 2026-07-14 12:32 UTC (0a27e88)

@github-actions
github-actions Bot requested a deployment to preview-pr-70661 July 14, 2026 10:32 In progress
@github-actions

github-actions Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

🦔 Hogbox preview · ✅ ready

▶ Open the preview

🔑 Login test@posthog.com / 12345678 (demo data)
🧩 Running this PR's backend and frontend, on the PostHog :master base
🔗 Link stable across rebuilds — a re-push swaps the box underneath, the URL stays
🔒 Access tailnet only (PostHog VPN)
🛠️ Admin inspect & debug state in hogland
💤 Idle sleeps after ~30 min idle (snapshot to S3, zero node cost) and wakes on your next visit in ~30s, behind a brief "waking up" screen

commit 0a27e88 · box box-07a15d9b80b5 · ready in 905s (push → usable) · build log · rebuilds on every push, torn down on close

@github-actions

github-actions Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

🤖 CI report

Bundle size — no change

Uncompressed size of every built .js bundle, compared against the base branch.

Total: 64.43 MiB · no change

No file changed by more than 1000 B.

Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report

Eager graph — within budget

How much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy import() / React.lazy chunks are not counted.

Root Eager (shipped) Δ vs base Budget
entry (logged-out pages, app bootstrap)
src/index.tsx
1.21 MiB · 22 files no change ███░░░░░░░ 28.2% of 4.29 MiB
authenticated shell (every logged-in page)
src/scenes/AuthenticatedShell.tsx
8.11 MiB · 2,974 files no change █████████░ 87.7% of 9.25 MiB

🟢 node_modules/monaco-editor/ stays out of src/index.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 node_modules/monaco-editor/ stays out of src/scenes/AuthenticatedShell.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx

Largest files eagerly shipped from src/index.tsx
Size File
126.8 KiB ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js
24.6 KiB ../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js
6.3 KiB ../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js
4.5 KiB ../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js
3.9 KiB ../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js
1.4 KiB ../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js
1.3 KiB src/RootErrorBoundary.tsx
912 B ../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js
789 B src/scenes/ChunkLoadErrorBoundary.tsx
668 B src/index.tsx
Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx
Size File
278.6 KiB ../node_modules/.pnpm/posthog-js@1.399.5/node_modules/posthog-js/dist/rrweb.js
267.7 KiB ../node_modules/.pnpm/@posthog+icons@0.38.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js
234.9 KiB src/taxonomy/core-filter-definitions-by-group.json
222.1 KiB ../node_modules/.pnpm/posthog-js@1.399.5/node_modules/posthog-js/dist/module.js
164.0 KiB src/queries/validators.js
154.3 KiB ../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js
126.8 KiB ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js
105.9 KiB src/lib/api.ts
93.3 KiB ../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js
90.6 KiB ../node_modules/.pnpm/@tiptap+core@3.20.6_@tiptap+pm@3.20.6/node_modules/@tiptap/core/dist/index.js

Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479

Dist folder size — no change

Total size of the built frontend/dist folder (all assets), compared against the base branch.

Total: 1284.43 MiB · no change

Generated-By: PostHog Code
Task-Id: b229f6d2-9134-4a6c-a9fc-09ddc24e6e71
@github-actions
github-actions Bot requested a deployment to preview-pr-70661 July 14, 2026 10:44 In progress
Generated-By: PostHog Code
Task-Id: b229f6d2-9134-4a6c-a9fc-09ddc24e6e71

Copy link
Copy Markdown
Member Author

CI status — one failing check is a master-wide infra flake, not this PR

The only red check is Product tests (warehouse-sources (4/4)) (and the Django Tests Pass gate that aggregates it). Everything else on the latest commit is green (146 passing).

This shard is failing for an infrastructure reason, not a test in this diff:

  • Every failure ends with the annotation "The self-hosted runner lost communication with the server… Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error" and uploads no logs (the runner dies before flushing them).
  • It reproduces on clean master — shard warehouse-sources (4/4) has been red while shards 1–3 stay green across multiple recent master runs (e.g. 29329064920, 29329033116, 29328190837, 29328058805, 29327356460). It was also red on this PR's pre-merge head, so merging master in did not introduce it.
  • I re-ran the failed job twice; it died the same way each time. The consistency (always the heaviest shard, 4/4) points to the shard OOMing / the runner being reclaimed rather than a flaky test.

The change here is a small, self-contained addition (self_driving created_via enum value + migration + serializer/test updates); I verified the new test cases match the server-side derivation logic in get_event_source, and the MCP tool-schema snapshots have been regenerated so Unit Tests is green.

This needs a fleet-side fix (shard rebalancing or a larger runner for warehouse-sources) rather than a code change in this PR. Flagging for a maintainer — happy to re-run once the shard is healthy on master.

🦉 via talyn.dev

Generated-By: PostHog Code
Task-Id: b229f6d2-9134-4a6c-a9fc-09ddc24e6e71
@github-actions
github-actions Bot requested a deployment to preview-pr-70661 July 14, 2026 12:03 In progress
@Gilbert09
Gilbert09 merged commit 04a79ad into master Jul 14, 2026
230 checks passed
@Gilbert09
Gilbert09 deleted the tom/dwh-self-driving-created-via branch July 14, 2026 12:54
@deployment-status-posthog

deployment-status-posthog Bot commented Jul 14, 2026

Copy link
Copy Markdown

Deploy status

Environment Status Deployed At Workflow
dev ✅ Deployed 2026-07-14 13:21 UTC Run
prod-us ✅ Deployed 2026-07-14 13:33 UTC Run
prod-eu ✅ Deployed 2026-07-14 13:33 UTC Run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants