Skip to content

feat(data-warehouse): implement vantage import source - #70016

Merged
talyn-app[bot] merged 2 commits into
masterfrom
posthog-code/implement-vantage-source
Jul 14, 2026
Merged

feat(data-warehouse): implement vantage import source#70016
talyn-app[bot] merged 2 commits into
masterfrom
posthog-code/implement-vantage-source

Conversation

@Gilbert09

Copy link
Copy Markdown
Member

Problem

We didn't have a data warehouse connector for Vantage, a FinOps and cloud cost management platform. Teams that use Vantage to track cloud spend across AWS, Azure, GCP, and other providers couldn't pull that data into PostHog to join spend against product usage and revenue. The source was scaffolded on master but shipped as an empty stub with no sync logic.

Changes

Fills in the scaffolded vantage source end to end, following the implementing-warehouse-sources skill. The enum, schema entry, and generated-config wiring already landed with the scaffold, so this is purely the implementation.

  • Transport (vantage.py) - talks to the v2 REST API (https://api.vantage.sh/v2) over the tracked session, Bearer-token auth, page-number pagination that follows the response links.next URL. Retries 429/5xx and transient network errors with bounded exponential backoff via tenacity.
  • Endpoints (settings.py) - 20 top-level list endpoints a FinOps user actually wants: cost reports, budgets, folders, dashboards, cost/anomaly alerts, resource/financial-commitment/network-flow/kubernetes-efficiency reports, segments, saved filters, recommendations, billing rules, access grants, integrations, workspaces, report notifications, teams, users. Each merges on the globally-unique token; objects with a stable created_at partition by month.
  • Source (source.py) - ResumableSource so large paginated backfills survive Temporal heartbeat timeouts (resume URL persisted to Redis after each yielded batch). Implements validate_credentials (cheap /ping probe), get_schemas, source_for_pipeline, get_resumable_source_manager, and get_non_retryable_errors (401/403).
  • Canonical descriptions - curated table/column docs sourced from the Vantage API reference, so semantic enrichment and the public docs table catalog don't need an LLM pass.
  • Icon - the committed vantage.png was a generic placeholder; replaced it with the real Vantage brand mark fetched from vantage.sh.

Note

Every table is full refresh only. Vantage's config/report objects expose no server-side updated_after cursor. Only the cost/usage data endpoints accept date-range windows, and those (plus their per-report fan-out and stricter 5 req/5s rate limit) are intentionally left for a follow-up. I couldn't verify server-side incremental filtering with a live curl smoke test since I don't have a Vantage token, so I took the conservative route the skill prescribes: ship full refresh, note the uncertainty in code.

The source stays behind unreleasedSource=True with releaseStatus=ALPHA.

How did you test this code?

Automated only (I'm an agent and did not manually connect a live Vantage account):

  • test_vantage.py (transport) - pagination follows links.next and terminates, reads the endpoint-specific data key (not a hardcoded data), resume starts from saved URL vs initial URL, resume state saved after yield only while pages remain, credential status mapping, retry classification (429/5xx/network retried, other 4xx raised), and partition/primary-key wiring per endpoint.
  • test_vantage_source.py (source class) - source type, secret field shape, alpha/unreleased config, full-refresh schemas + names filter, credential delegation, resumable-manager binding, source_for_pipeline plumbing, non-retryable error matching, public-docs catalog, and canonical-key/endpoint agreement.
  • 39 vantage tests pass; the shared test_source_categories suite (1506 tests) still passes. ruff check --fix and ruff format clean.

I verified the API shape against the live published OpenAPI spec (oas_v3.json) and confirmed the base URL returns 401 unauthenticated, but could not exercise authenticated endpoints without a token.

Docs update

Wrote the user-facing doc per the documenting-warehouse-sources skill, but there's no posthog.com checkout in this environment so it isn't committed here. It needs to land at posthog.com/contents/docs/cdp/sources/vantage.md (matching docsUrl). Content ready to drop in:

vantage.md
---
title: Linking Vantage as a source
sidebar: Docs
showTitle: true
availability: { free: full, selfServe: full, enterprise: full }
sourceId: Vantage
beta: true
---

import SourceSetupIntro from "../_snippets/source-setup-intro.mdx"
import SyncModes from "../_snippets/sync-modes.mdx"
import TroubleshootingLink from "../_snippets/dw-troubleshooting-link.mdx"
import AlphaRelease from "../_snippets/alpha-release.mdx"

<AlphaRelease />

[Vantage](https://www.vantage.sh) is a FinOps and cloud cost management platform that aggregates and analyzes spend across AWS, Azure, GCP, and other providers. This connector syncs your Vantage cost reports, budgets, dashboards, alerts, recommendations, and related configuration into the PostHog data warehouse so you can join cloud spend with your product and revenue data.

## Prerequisites

To connect Vantage, you need a Vantage account and an API access token with the `read` scope. Any account member can create a personal access token; account owners can also create account-scoped service tokens.

## Adding a data source

<SourceSetupIntro />

You'll need a Vantage **API access token**:

1. Sign in to Vantage and open your [access tokens settings](https://console.vantage.sh/settings/access_tokens).
2. Create a new access token (or service token) and grant it the **`read`** scope. The write scope is not required for imports.
3. Copy the token and paste it into the **API access token** field in PostHog.

## Sync modes

<SyncModes />

Vantage's configuration and report objects don't expose a server-side "updated after" filter, so every table syncs as a full refresh. Each sync re-reads the current state and merges on the object's `token`, so rows stay up to date without creating duplicates.

## Configuration

<SourceParameters />

## Supported tables

<SourceTables />

## Troubleshooting

- **Invalid or expired token** - if syncs fail with an authorization error, your token may have been revoked or lack the `read` scope. Create a new read-scoped token in your Vantage access tokens settings and reconnect the source.
- **Rate limits** - Vantage enforces per-account and per-key rate limits (and stricter limits on cost data endpoints). PostHog automatically backs off and retries when it's throttled, so transient rate-limit errors resolve on their own.

<TroubleshootingLink />

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Authored by Claude (Claude Code). Skills invoked: /implementing-warehouse-sources, /writing-tests, /documenting-warehouse-sources.

Key decisions:

  • ResumableSource over SimpleSource - the API paginates via a links.next URL, which is a clean resume cursor. Modeled the transport on the existing klaviyo source (same links-based pagination shape).
  • Full refresh everywhere - I inspected the live OpenAPI spec: only costs, anomaly_alerts, and recommendations accept date-range params, and none is a generic updated_after cursor on config resources. Without a token to curl-verify that a future-date cutoff actually filters server-side, declaring incremental would risk silently skipping rows, so I shipped full refresh and documented why.
  • Endpoint selection - picked the 20 uniform {"links", "<key>": [...]} list endpoints that map to real FinOps objects; deferred the cost/usage data endpoints (per-report fan-out + 5 req/5s limit) to keep the alpha scope safe.
  • Icon - the scaffold's vantage.png was a placeholder (an unrelated eye/"i" glyph). Replaced it with the real brand mark from vantage.sh. Kept .png because I couldn't render-verify a hand-authored SVG in this environment and correct branding beat format preference.

Created with PostHog Code

@Gilbert09 Gilbert09 self-assigned this Jul 10, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Hey @Gilbert09! 👋

It looks like your git author email on this PR isn't your @posthog.com address (owerstom@gmail.com). Since you're on the PostHog team, it's worth pointing your local git author email at your @posthog.com address. Why it matters:

  • Consistent work identity in git history — internal tooling that attributes commits to team members keys off your @posthog.com address.
  • Keeps team contributions easy to tell apart from external community ones when scanning history.

You can fix it for this repo with:

git config user.email "you@posthog.com"

Or set it globally with git config --global user.email "you@posthog.com". No need to redo this PR — just a nudge for next time. 🙂

@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

🦔 Hogbox preview · ✅ ready

▶ Open the preview

🔑 Login test@posthog.com / 12345678 (demo data)
🧩 Running this PR's backend and frontend, on the PostHog :master base
🔗 Link stable across rebuilds — a re-push swaps the box underneath, the URL stays
🔒 Access tailnet only (PostHog VPN)
🛠️ Admin inspect & debug state in hogland
💤 Idle sleeps after ~30 min idle (snapshot to S3, zero node cost) and wakes on your next visit in ~30s, behind a brief "waking up" screen

commit 4295080 · box box-693b65ac2faa · ready in 637s (push → usable) · build log · rebuilds on every push, torn down on close

@greptile-apps

greptile-apps Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Reviews (1): Last reviewed commit: "feat(data-warehouse): implement vantage ..." | Re-trigger Greptile

@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

🤖 CI report

Bundle size — no change

Uncompressed size of every built .js bundle, compared against the base branch.

Total: 64.35 MiB · no change

No file changed by more than 1000 B.

Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report

Eager graph — within budget

How much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy import() / React.lazy chunks are not counted.

Root Eager (shipped) Δ vs base Budget
entry (logged-out pages, app bootstrap)
src/index.tsx
1.21 MiB · 22 files no change ███░░░░░░░ 28.1% of 4.29 MiB
authenticated shell (every logged-in page)
src/scenes/AuthenticatedShell.tsx
8.11 MiB · 2,973 files no change █████████░ 87.7% of 9.25 MiB

🟢 node_modules/monaco-editor/ stays out of src/index.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 node_modules/monaco-editor/ stays out of src/scenes/AuthenticatedShell.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx

Largest files eagerly shipped from src/index.tsx
Size File
126.8 KiB ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js
24.6 KiB ../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js
6.3 KiB ../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js
4.5 KiB ../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js
3.9 KiB ../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js
1.4 KiB ../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js
1.3 KiB src/RootErrorBoundary.tsx
912 B ../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js
789 B src/scenes/ChunkLoadErrorBoundary.tsx
668 B src/index.tsx
Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx
Size File
278.6 KiB ../node_modules/.pnpm/posthog-js@1.399.4/node_modules/posthog-js/dist/rrweb.js
267.7 KiB ../node_modules/.pnpm/@posthog+icons@0.38.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js
234.9 KiB src/taxonomy/core-filter-definitions-by-group.json
221.8 KiB ../node_modules/.pnpm/posthog-js@1.399.4/node_modules/posthog-js/dist/module.js
164.0 KiB src/queries/validators.js
154.3 KiB ../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js
126.8 KiB ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js
105.9 KiB src/lib/api.ts
93.3 KiB ../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js
90.6 KiB ../node_modules/.pnpm/@tiptap+core@3.20.6_@tiptap+pm@3.20.6/node_modules/@tiptap/core/dist/index.js

Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479

Dist folder size — 🟢 -39.2 KiB (-0.0%)

Total size of the built frontend/dist folder (all assets), compared against the base branch.

Total: 1281.24 MiB · 🟢 -39.2 KiB (-0.0%)

Playwright — all passed

All tests passed.

View test results →

⚠️ Backend coverage — 99.0% of changed backend lines covered — 3 uncovered

🧪 Backend test coverage

Patch coverage — changed backend lines (products + core): ████████████████████ 99.0% (352 / 355)

File Patch Uncovered changed lines
products/warehouse_sources/backend/temporal/data_imports/sources/vantage/vantage.py 97.4% 42–43
products/warehouse_sources/backend/temporal/data_imports/sources/vantage/tests/test_vantage.py 99.3% 45

🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 29300522499 -n patch-coverage), or the coverage-data block at the end of this comment.

Per-product line coverage (touched products)
Product Coverage Lines
warehouse_sources ███████████████████░ 96.1% 211,729 / 220,375

Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.

@veria-ai

veria-ai Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

@trunk-io

trunk-io Bot commented Jul 10, 2026

Copy link
Copy Markdown

Static BadgeStatic BadgeStatic BadgeStatic Badge

View Full Report ↗︎Docs

@estefaniarabadan estefaniarabadan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isolated new data-warehouse source scaffold — self-contained under its own source directory with only the expected shared-registry additions (SOURCES.md, generated_configs.py, icon), tests included, CI green. Reviewed as part of a batch. LGTM 👍

Fill in the scaffolded Vantage FinOps source: page-number pagination over the v2 REST API,
resumable full-refresh sync for 20 config/report endpoints, credential validation, canonical
table/column descriptions, and tests. Replaces the placeholder icon with the real brand mark.

Generated-By: PostHog Code
Task-Id: 40698480-5dce-441b-99ab-1240f2426318
Redact the Vantage API token in tracked HTTP sessions and pin every
fetched URL to Vantage's own HTTPS host and /v2 path, so a spoofed
`links.next` (or a resumed cursor derived from one) can't forward the
bearer token off-host. Also disable redirect following on the paginating
session as defense-in-depth.

Fix the mypy call-arg error in the test by passing `response=` to
`requests.HTTPError`.

Generated-By: PostHog Code
Task-Id: cc101280-a1ed-4d43-ae9f-02a204c6746b
@Gilbert09
Gilbert09 force-pushed the posthog-code/implement-vantage-source branch from e6a4a99 to 4295080 Compare July 14, 2026 02:13
@talyn-app
talyn-app Bot merged commit 14c26ac into master Jul 14, 2026
235 checks passed
@talyn-app
talyn-app Bot deleted the posthog-code/implement-vantage-source branch July 14, 2026 02:44
@deployment-status-posthog

deployment-status-posthog Bot commented Jul 14, 2026

Copy link
Copy Markdown

Deploy status

Environment Status Deployed At Workflow
dev ✅ Deployed 2026-07-14 03:13 UTC Run
prod-us ✅ Deployed 2026-07-14 03:28 UTC Run
prod-eu ✅ Deployed 2026-07-14 03:29 UTC Run

posthog Bot added a commit to PostHog/posthog.com that referenced this pull request Jul 14, 2026
Adds the user-facing setup page for the new Vantage import source
introduced in PostHog/posthog#70016. Covers prerequisites, API access
token setup, sync behavior, and troubleshooting.

Generated-By: PostHog Code
Task-Id: 3a2c2e12-78d1-4b0e-9278-a1c2754a90ae
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants