feat(data-warehouse): implement vantage import source - #70016
Conversation
|
Hey @Gilbert09! 👋 It looks like your git author email on this PR isn't your
You can fix it for this repo with: git config user.email "you@posthog.com"Or set it globally with |
🦔 Hogbox preview · ✅ ready▶ Open the preview
commit |
|
Reviews (1): Last reviewed commit: "feat(data-warehouse): implement vantage ..." | Re-trigger Greptile |
🤖 CI report✅ Bundle size — no changeUncompressed size of every built Total: 64.35 MiB · no change No file changed by more than 1000 B. Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report ✅ Eager graph — within budgetHow much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy
🟢 Largest files eagerly shipped from
|
| Size | File |
|---|---|
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 24.6 KiB | ../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js |
| 6.3 KiB | ../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js |
| 4.5 KiB | ../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js |
| 3.9 KiB | ../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js |
| 1.4 KiB | ../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js |
| 1.3 KiB | src/RootErrorBoundary.tsx |
| 912 B | ../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js |
| 789 B | src/scenes/ChunkLoadErrorBoundary.tsx |
| 668 B | src/index.tsx |
Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx
| Size | File |
|---|---|
| 278.6 KiB | ../node_modules/.pnpm/posthog-js@1.399.4/node_modules/posthog-js/dist/rrweb.js |
| 267.7 KiB | ../node_modules/.pnpm/@posthog+icons@0.38.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js |
| 234.9 KiB | src/taxonomy/core-filter-definitions-by-group.json |
| 221.8 KiB | ../node_modules/.pnpm/posthog-js@1.399.4/node_modules/posthog-js/dist/module.js |
| 164.0 KiB | src/queries/validators.js |
| 154.3 KiB | ../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js |
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 105.9 KiB | src/lib/api.ts |
| 93.3 KiB | ../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js |
| 90.6 KiB | ../node_modules/.pnpm/@tiptap+core@3.20.6_@tiptap+pm@3.20.6/node_modules/@tiptap/core/dist/index.js |
Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479
✅ Dist folder size — 🟢 -39.2 KiB (-0.0%)
Total size of the built frontend/dist folder (all assets), compared against the base branch.
Total: 1281.24 MiB · 🟢 -39.2 KiB (-0.0%)
⚠️ Backend coverage — 99.0% of changed backend lines covered — 3 uncovered
🧪 Backend test coverage
Patch coverage — changed backend lines (products + core): ████████████████████ 99.0% (352 / 355)
| File | Patch | Uncovered changed lines |
|---|---|---|
products/warehouse_sources/backend/temporal/data_imports/sources/vantage/vantage.py |
97.4% | 42–43 |
products/warehouse_sources/backend/temporal/data_imports/sources/vantage/tests/test_vantage.py |
99.3% | 45 |
🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 29300522499 -n patch-coverage), or the coverage-data block at the end of this comment.
Per-product line coverage (touched products)
| Product | Coverage | Lines |
|---|---|---|
warehouse_sources |
███████████████████░ 96.1% |
211,729 / 220,375 |
Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.
PR overviewAll previously flagged issues have been addressed. No open security concerns remain on this pull request. Security reviewNo open security issues remain on this pull request. Fixed/addressed: 1 · PR risk: 0/10 |
estefaniarabadan
left a comment
There was a problem hiding this comment.
Isolated new data-warehouse source scaffold — self-contained under its own source directory with only the expected shared-registry additions (SOURCES.md, generated_configs.py, icon), tests included, CI green. Reviewed as part of a batch. LGTM 👍
Fill in the scaffolded Vantage FinOps source: page-number pagination over the v2 REST API, resumable full-refresh sync for 20 config/report endpoints, credential validation, canonical table/column descriptions, and tests. Replaces the placeholder icon with the real brand mark. Generated-By: PostHog Code Task-Id: 40698480-5dce-441b-99ab-1240f2426318
Redact the Vantage API token in tracked HTTP sessions and pin every fetched URL to Vantage's own HTTPS host and /v2 path, so a spoofed `links.next` (or a resumed cursor derived from one) can't forward the bearer token off-host. Also disable redirect following on the paginating session as defense-in-depth. Fix the mypy call-arg error in the test by passing `response=` to `requests.HTTPError`. Generated-By: PostHog Code Task-Id: cc101280-a1ed-4d43-ae9f-02a204c6746b
e6a4a99 to
4295080
Compare
Adds the user-facing setup page for the new Vantage import source introduced in PostHog/posthog#70016. Covers prerequisites, API access token setup, sync behavior, and troubleshooting. Generated-By: PostHog Code Task-Id: 3a2c2e12-78d1-4b0e-9278-a1c2754a90ae
Problem
We didn't have a data warehouse connector for Vantage, a FinOps and cloud cost management platform. Teams that use Vantage to track cloud spend across AWS, Azure, GCP, and other providers couldn't pull that data into PostHog to join spend against product usage and revenue. The source was scaffolded on
masterbut shipped as an empty stub with no sync logic.Changes
Fills in the scaffolded
vantagesource end to end, following theimplementing-warehouse-sourcesskill. The enum, schema entry, and generated-config wiring already landed with the scaffold, so this is purely the implementation.vantage.py) - talks to the v2 REST API (https://api.vantage.sh/v2) over the tracked session, Bearer-token auth, page-number pagination that follows the responselinks.nextURL. Retries429/5xxand transient network errors with bounded exponential backoff viatenacity.settings.py) - 20 top-level list endpoints a FinOps user actually wants: cost reports, budgets, folders, dashboards, cost/anomaly alerts, resource/financial-commitment/network-flow/kubernetes-efficiency reports, segments, saved filters, recommendations, billing rules, access grants, integrations, workspaces, report notifications, teams, users. Each merges on the globally-uniquetoken; objects with a stablecreated_atpartition by month.source.py) -ResumableSourceso large paginated backfills survive Temporal heartbeat timeouts (resume URL persisted to Redis after each yielded batch). Implementsvalidate_credentials(cheap/pingprobe),get_schemas,source_for_pipeline,get_resumable_source_manager, andget_non_retryable_errors(401/403).vantage.pngwas a generic placeholder; replaced it with the real Vantage brand mark fetched from vantage.sh.Note
Every table is full refresh only. Vantage's config/report objects expose no server-side
updated_aftercursor. Only the cost/usage data endpoints accept date-range windows, and those (plus their per-report fan-out and stricter 5 req/5s rate limit) are intentionally left for a follow-up. I couldn't verify server-side incremental filtering with a live curl smoke test since I don't have a Vantage token, so I took the conservative route the skill prescribes: ship full refresh, note the uncertainty in code.The source stays behind
unreleasedSource=TruewithreleaseStatus=ALPHA.How did you test this code?
Automated only (I'm an agent and did not manually connect a live Vantage account):
test_vantage.py(transport) - pagination followslinks.nextand terminates, reads the endpoint-specific data key (not a hardcodeddata), resume starts from saved URL vs initial URL, resume state saved after yield only while pages remain, credential status mapping, retry classification (429/5xx/network retried, other4xxraised), and partition/primary-key wiring per endpoint.test_vantage_source.py(source class) - source type, secret field shape, alpha/unreleased config, full-refresh schemas + names filter, credential delegation, resumable-manager binding,source_for_pipelineplumbing, non-retryable error matching, public-docs catalog, and canonical-key/endpoint agreement.test_source_categoriessuite (1506 tests) still passes.ruff check --fixandruff formatclean.I verified the API shape against the live published OpenAPI spec (
oas_v3.json) and confirmed the base URL returns401unauthenticated, but could not exercise authenticated endpoints without a token.Docs update
Wrote the user-facing doc per the
documenting-warehouse-sourcesskill, but there's no posthog.com checkout in this environment so it isn't committed here. It needs to land atposthog.com/contents/docs/cdp/sources/vantage.md(matchingdocsUrl). Content ready to drop in:vantage.md
🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Authored by Claude (Claude Code). Skills invoked:
/implementing-warehouse-sources,/writing-tests,/documenting-warehouse-sources.Key decisions:
ResumableSourceoverSimpleSource- the API paginates via alinks.nextURL, which is a clean resume cursor. Modeled the transport on the existingklaviyosource (same links-based pagination shape).costs,anomaly_alerts, andrecommendationsaccept date-range params, and none is a genericupdated_aftercursor on config resources. Without a token to curl-verify that a future-date cutoff actually filters server-side, declaring incremental would risk silently skipping rows, so I shipped full refresh and documented why.{"links", "<key>": [...]}list endpoints that map to real FinOps objects; deferred the cost/usage data endpoints (per-report fan-out + 5 req/5s limit) to keep the alpha scope safe.vantage.pngwas a placeholder (an unrelated eye/"i" glyph). Replaced it with the real brand mark from vantage.sh. Kept.pngbecause I couldn't render-verify a hand-authored SVG in this environment and correct branding beat format preference.Created with PostHog Code