feat(data-warehouse): add Codemagic source - #73186
Conversation
|
😎 This pull request was merged. |
🦔 Hogbox preview · ❌ build failedThe preview didn't come up for commit Previews are optional and never block merging. A failure here is often a hogland or tailnet hiccup rather than anything in your PR, so the check stays green and this comment is the status. |
|
Hey @Gilbert09! 👋 It looks like your git author email on this PR isn't your
You can fix it for this repo with: git config user.email "you@posthog.com"Or set it globally with |
![hex-security-app[bot]](https://avatars.githubusercontent.com/in/2665233?s=60&v=4){kind=small}
🤖 CI report✅ Bundle size — no changeUncompressed size of every built Total: 64.39 MiB · no change No file changed by more than 1000 B. Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report ✅ Eager graph — within budgetHow much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy
🟢 Largest files eagerly shipped from
|
| Size | File |
|---|---|
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 24.6 KiB | ../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js |
| 6.3 KiB | ../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js |
| 4.5 KiB | ../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js |
| 3.9 KiB | ../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js |
| 1.4 KiB | ../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js |
| 1.3 KiB | src/RootErrorBoundary.tsx |
| 912 B | ../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js |
| 789 B | src/scenes/ChunkLoadErrorBoundary.tsx |
| 762 B | src/index.tsx |
Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx
| Size | File |
|---|---|
| 281.5 KiB | ../node_modules/.pnpm/posthog-js@1.407.2/node_modules/posthog-js/dist/rrweb.js |
| 267.7 KiB | ../node_modules/.pnpm/@posthog+icons@0.38.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js |
| 236.0 KiB | src/taxonomy/core-filter-definitions-by-group.json |
| 226.1 KiB | ../node_modules/.pnpm/posthog-js@1.407.2/node_modules/posthog-js/dist/module.js |
| 154.3 KiB | ../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js |
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 106.2 KiB | src/lib/api.ts |
| 94.0 KiB | ../packages/quill/packages/quill/dist/index.js |
| 93.3 KiB | ../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js |
| 90.6 KiB | ../node_modules/.pnpm/@tiptap+core@3.20.6_@tiptap+pm@3.20.6/node_modules/@tiptap/core/dist/index.js |
Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479
✅ Toolbar bundle — eager 2.18 MiB within budget
What the toolbar ships to customer pages, measured from the esbuild output (minified, post-tree-shake). The eager set is the entry plus everything statically imported from it — fetched before any feature runs; deferred chunks load lazily. The eager guardrail is 5.72 MiB. Each output file must also stay below 10 MB, where CloudFront stops compressing it. The module boundary is enforced separately by check-toolbar-graph.
| Metric | Size | Δ vs base | Budget |
|---|---|---|---|
| Eager (shipped) entry + static imports |
2.18 MiB · 17 files | no change | ████░░░░░░ 38.1% of 5.72 MiB |
| Deferred (lazy) | 2.07 MiB · 33 files | no change | n/a — loads on demand |
Loader dist/toolbar.js |
1.1 KiB | no change | █░░░░░░░░░ 5.8% of 19.5 KiB |
Largest eagerly-shipped chunks
| Size | File |
|---|---|
| 713.8 KiB | dist/toolbar/toolbar-app-HLTHZTTF.css |
| 545.1 KiB | dist/toolbar/chunk-chunk-JXZAQF2A.js |
| 484.2 KiB | dist/toolbar/chunk-chunk-BQIRV2TJ.js |
| 133.6 KiB | dist/toolbar/chunk-chunk-YBAFBB2V.js |
| 131.8 KiB | dist/toolbar/chunk-chunk-T5KY5WYR.js |
| 71.0 KiB | dist/toolbar/toolbar-app-HJ3A3JN3.js |
| 69.0 KiB | dist/toolbar/chunk-chunk-27JL52RE.js |
| 35.6 KiB | dist/toolbar/chunk-chunk-YSBEFDVO.js |
| 20.9 KiB | dist/toolbar/chunk-chunk-2H2TQVQG.js |
| 12.2 KiB | dist/toolbar/chunk-chunk-PIK3PADE.js |
Posted automatically by check-toolbar-size · sizes are toolbar output bytes (shipped, post-tree-shake) from the esbuild metafile
✅ Dist folder size — 🔺 +49.0 KiB (+0.0%)
Total size of the built frontend/dist folder (all assets), compared against the base branch.
Total: 1353.46 MiB · 🔺 +49.0 KiB (+0.0%)
⚠️ Backend coverage — 99.0% of changed backend lines covered — 2 uncovered
🧪 Backend test coverage
Patch coverage — changed backend lines (products + core): ████████████████████ 99.0% (357 / 359)
| File | Patch | Uncovered changed lines |
|---|---|---|
products/warehouse_sources/backend/temporal/data_imports/sources/codemagic/codemagic.py |
96.7% | 42, 54 |
🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 30095505089 -n patch-coverage), or the coverage-data block at the end of this comment.
Per-product line coverage (touched products)
| Product | Coverage | Lines |
|---|---|---|
demo |
███████████░░░░░░░░░ 56.2% |
1,497 / 2,663 |
tasks |
██████████████░░░░░░ 69.4% |
30,026 / 43,270 |
signals |
████████████████░░░░ 80.0% |
22,088 / 27,597 |
cdp |
████████████████░░░░ 81.0% |
3,146 / 3,883 |
data_modeling |
█████████████████░░░ 82.7% |
5,547 / 6,708 |
notebooks |
█████████████████░░░ 85.6% |
7,478 / 8,736 |
agent_platform |
█████████████████░░░ 86.4% |
3,807 / 4,405 |
actions |
█████████████████░░░ 86.6% |
717 / 828 |
cohorts |
██████████████████░░ 87.8% |
4,488 / 5,114 |
product_tours |
██████████████████░░ 87.9% |
1,303 / 1,482 |
exports |
██████████████████░░ 88.4% |
6,951 / 7,863 |
data_warehouse |
██████████████████░░ 88.9% |
11,887 / 13,378 |
conversations |
██████████████████░░ 89.3% |
16,875 / 18,894 |
engineering_analytics |
██████████████████░░ 89.4% |
6,386 / 7,145 |
dashboards |
██████████████████░░ 89.5% |
5,989 / 6,693 |
error_tracking |
██████████████████░░ 89.7% |
10,174 / 11,343 |
alerts |
██████████████████░░ 90.0% |
4,056 / 4,508 |
early_access_features |
██████████████████░░ 90.1% |
1,031 / 1,144 |
mcp_analytics |
██████████████████░░ 90.3% |
2,815 / 3,116 |
streamlit_apps |
██████████████████░░ 90.4% |
2,501 / 2,767 |
slack_app |
██████████████████░░ 90.7% |
9,028 / 9,951 |
marketing_analytics |
██████████████████░░ 91.0% |
11,792 / 12,964 |
stamphog |
██████████████████░░ 91.1% |
4,056 / 4,450 |
product_analytics |
███████████████████░ 92.5% |
5,848 / 6,319 |
ai_observability |
███████████████████░ 92.8% |
15,043 / 16,217 |
surveys |
███████████████████░ 93.0% |
5,734 / 6,167 |
web_analytics |
███████████████████░ 93.1% |
14,426 / 15,492 |
posthog_ai |
███████████████████░ 93.2% |
1,326 / 1,422 |
approvals |
███████████████████░ 93.3% |
3,437 / 3,682 |
reminders |
███████████████████░ 93.4% |
468 / 501 |
workflows |
███████████████████░ 93.6% |
6,434 / 6,877 |
endpoints |
███████████████████░ 94.1% |
8,640 / 9,177 |
skills |
███████████████████░ 94.6% |
3,158 / 3,337 |
review_hog |
███████████████████░ 94.7% |
6,814 / 7,199 |
logs |
███████████████████░ 95.4% |
10,012 / 10,498 |
experiments |
███████████████████░ 95.7% |
25,208 / 26,332 |
annotations |
███████████████████░ 96.2% |
732 / 761 |
replay_vision |
███████████████████░ 96.2% |
15,465 / 16,069 |
revenue_analytics |
███████████████████░ 96.3% |
1,887 / 1,960 |
feature_flags |
███████████████████░ 96.5% |
17,144 / 17,768 |
user_interviews |
███████████████████░ 96.5% |
2,638 / 2,734 |
customer_analytics |
███████████████████░ 97.1% |
9,093 / 9,364 |
warehouse_sources |
███████████████████░ 97.2% |
339,511 / 349,344 |
data_catalog |
███████████████████░ 97.4% |
2,367 / 2,429 |
pulse |
████████████████████ 98.4% |
2,017 / 2,049 |
Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.
The Codemagic API token rides in a custom x-auth-token header that requests preserves across cross-origin redirects, so following a 3xx could replay it off-host. Pin allow_redirects=False on both the sync REST client and the credential-validation session. Generated-By: PostHog Code Task-Id: 86f43bb4-2867-42d1-9e3e-822654cdd202
b02470d to
d160a4f
Compare
3 participants
Problem
PostHog customers who use Codemagic (Nevercode Ltd) (Mobile CI/CD) can't currently pull that data into the data warehouse. This adds Codemagic as a first-class warehouse import source so they can sync it alongside their other data and join it in queries, insights, and revenue/analytics models.
Changes
New Codemagic warehouse import source under
products/warehouse_sources/backend/temporal/data_imports/sources/codemagic/, built on the sharedrest_sourceframework following theimplementing-warehouse-sourcesskill.ResumableSource.releaseStatus=ReleaseStatus.ALPHA(new, lightly tested), visible and connectable (nounreleasedSourceflag).canonical_descriptions.pyfor semantic enrichment, the vendor icon, and moves the row intoSOURCES.md's implemented table.How did you test this code?
Automated tests only, run locally:
test_codemagic.py(transport: pagination, incremental vs full-refresh resource generation, primary-key mapping, credential-status mapping) andtest_codemagic_source.py(source class:source_type,get_source_config,get_schemas,validate_credentials, pipeline handoff) — 31 tests, all passing. The repo-widetest_source_categories,test_source_versions, andtest_generated_configsguards pass for this source.mypyandruffare clean.I did not exercise this against a live Codemagic (Nevercode Ltd) account (no test credentials in this environment) — endpoint shapes, pagination, auth, and field names were verified against the vendor's official API docs/OpenAPI spec, and HTTP is mocked in the tests. That's what
ALPHAreflects.Note
The user-facing posthog.com source doc isn't included here (it lives in a separate repo).
docsUrlpoints at the conventional/docs/cdp/sources/codemagicpath for when that page is authored.