feat(data-warehouse): implement vellum import source - #70028
Conversation
Fill in the scaffolded Vellum warehouse source end-to-end: a ResumableSource over Vellum's DRF-style limit/offset REST API (X-API-KEY auth) covering workflow deployments, prompt deployments, document indexes, documents, and per-deployment workflow execution events (a fan-out over deployments). All outbound HTTP goes through the tracked transport. Every table is full refresh (the list endpoints expose no server-side timestamp filter, verified against the live API), with offset-based resumable pagination. Adds canonical descriptions, the brand icon, source + transport tests, and updates SOURCES.md. Generated-By: PostHog Code Task-Id: db20f000-5c98-4ee7-a2af-e4ddb1000fc0
|
Hey @Gilbert09! 👋 It looks like your git author email on this PR isn't your
You can fix it for this repo with: git config user.email "you@posthog.com"Or set it globally with |
🦔 Hogbox preview · ✅ ready▶ Open the preview
commit |
![hex-security-app[bot]](https://avatars.githubusercontent.com/in/2665233?s=60&v=4){kind=small}
|
Reviews (1): Last reviewed commit: "feat(data-warehouse): implement vellum i..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
🤖 CI report✅ Bundle size — no changeUncompressed size of every built Total: 64.35 MiB · no change No file changed by more than 1000 B. Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report ✅ Eager graph — within budgetHow much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy
🟢 Largest files eagerly shipped from
|
| Size | File |
|---|---|
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 24.6 KiB | ../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js |
| 6.3 KiB | ../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js |
| 4.5 KiB | ../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js |
| 3.9 KiB | ../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js |
| 1.4 KiB | ../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js |
| 1.3 KiB | src/RootErrorBoundary.tsx |
| 912 B | ../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js |
| 789 B | src/scenes/ChunkLoadErrorBoundary.tsx |
| 668 B | src/index.tsx |
Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx
| Size | File |
|---|---|
| 278.6 KiB | ../node_modules/.pnpm/posthog-js@1.399.4/node_modules/posthog-js/dist/rrweb.js |
| 267.7 KiB | ../node_modules/.pnpm/@posthog+icons@0.38.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js |
| 234.9 KiB | src/taxonomy/core-filter-definitions-by-group.json |
| 221.8 KiB | ../node_modules/.pnpm/posthog-js@1.399.4/node_modules/posthog-js/dist/module.js |
| 164.0 KiB | src/queries/validators.js |
| 154.3 KiB | ../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js |
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 105.9 KiB | src/lib/api.ts |
| 93.3 KiB | ../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js |
| 90.6 KiB | ../node_modules/.pnpm/@tiptap+core@3.20.6_@tiptap+pm@3.20.6/node_modules/@tiptap/core/dist/index.js |
Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479
⚠️ Dist folder size — 🔺 +1.6 KiB (+0.0%)
Total size of the built frontend/dist folder (all assets), compared against the base branch.
Total: 1278.78 MiB · 🔺 +1.6 KiB (+0.0%)
⚠️ Playwright — 1 failed
🎭 Playwright report · View test results →
❌ 1 failed test:
- Creating a SQL insight with a variable and overriding it on a dashboard (chromium)
These issues are not necessarily caused by your changes.
Annoyed by this section? Help fix flakies and failures and it will go green!
⚠️ Backend coverage — 97.0% of changed backend lines covered — 8 uncovered
🧪 Backend test coverage
Patch coverage — changed backend lines (products + core): ███████████████████░ 97.0% (374 / 382)
| File | Patch | Uncovered changed lines |
|---|---|---|
products/warehouse_sources/backend/temporal/data_imports/sources/vellum/vellum.py |
92.9% | 46–47, 50–54, 145 |
🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 29282678386 -n patch-coverage), or the coverage-data block at the end of this comment.
Per-product line coverage (touched products)
| Product | Coverage | Lines |
|---|---|---|
warehouse_sources |
███████████████████░ 96.1% |
210,001 / 218,601 |
Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.
…rom logs Address review feedback on the Vellum source: - Register the API key for value-based redaction on the tracked sessions used for both credential validation and the sync path, so the custom X-API-KEY header value is masked from outbound request logs and captured HTTP samples. - Stop logging Vellum error response bodies, which can echo synced execution data (workflow inputs/outputs) into operational logs. Status code and URL are retained for debugging. - Type the fan-out parent-id injection cleanly instead of a type: ignore. Generated-By: PostHog Code Task-Id: b8bffd38-92f9-499e-8be7-85389be0b8a4
Fix two static-typing errors surfaced now that ty passes and mypy runs: - Pass response= when constructing requests.HTTPError in the fetch-page test. - Narrow the credential field to SourceFieldInputConfig before asserting on its required/secret attributes. Generated-By: PostHog Code Task-Id: b8bffd38-92f9-499e-8be7-85389be0b8a4
estefaniarabadan
left a comment
estefaniarabadan
left a comment
There was a problem hiding this comment.
Isolated new data-warehouse source scaffold — self-contained under its own source directory with only the expected shared-registry additions (SOURCES.md, generated_configs.py, icon), tests included, CI green. Reviewed as part of a batch. LGTM 👍
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
PR overviewAll previously flagged issues have been addressed. No open security concerns remain on this pull request. Security reviewNo open security issues remain on this pull request. Fixed/addressed: 1 · PR risk: 0/10 |
Vellum API responses can echo user-authored content (workflow inputs/outputs, document metadata, descriptions) that the generic name-based scrubbers can't recognise. Set capture=False on both the sync session and the credential-probe session so response bodies are never stored in the HTTP sample bucket, while keeping redact_values for request telemetry. Generated-By: PostHog Code Task-Id: 75cc1a5f-d51a-4f5a-895f-77a4b6a2e13c
Generated-By: PostHog Code Task-Id: 75cc1a5f-d51a-4f5a-895f-77a4b6a2e13c
2 participants
Problem
Vellum (a platform for building, deploying, and monitoring production LLM apps) was scaffolded as a
Data warehouse source but had no sync logic. Users who run their LLM apps on Vellum could not pull
their deployments, documents, or execution history into the PostHog Data warehouse to join it with the
rest of their product data.
Changes
Fills in the scaffolded
vellumsource end-to-end, following the standard source architecture split:settings.py— declarative endpoint catalog for five tables:workflow_deployments,prompt_deployments,document_indexes,documents, andworkflow_execution_events(a fan-outthat pulls each workflow deployment's execution history). Primary keys and stable
created-stylepartition keys per endpoint; the execution-events child carries the parent deployment id in its
composite key so rows stay unique table-wide.
vellum.py—X-API-KEYREST client with offset/limit pagination, tenacity retries, and thefan-out transport. All HTTP goes through
make_tracked_session().source.py—ResumableSourcewiring: credential validation, schema list, non-retryableauth-error mapping, and resumable pagination state (offset per resource, deployment-id bookmark for
the fan-out; saved after each yielded batch).
updated
SOURCES.mdentry.Every table is full refresh: Vellum's list endpoints expose no server-side timestamp filter (a
future-dated
created__gtecutoff was confirmed against the live API to be silently ignored), sodeclaring incremental sync would be misleading. Rows dedupe on their primary key at merge time. The
source is kept behind
unreleasedSource=TruewithreleaseStatus=alpha.A user-facing doc for
posthog.com/docs/cdp/sources/vellumhas been written but could not be committedhere (no posthog.com checkout in this environment) — it needs to land in the posthog.com repo.
How did you test this code?
live
curlcalls againstapi.vellum.aiand its public OpenAPI spec (no credentials required for thepublic endpoints; response schemas for the auth-gated ones came from the spec).
tests/test_vellum_source.py,tests/test_vellum.py) — 40 tests coveringschema/full-refresh guarantees, credential-status mapping, non-retryable auth errors, offset
pagination termination and resume, the fan-out's parent-id injection / 404-skip / resume-from-bookmark
behavior, retry policy, and per-endpoint primary-key/partition config.
test_source_categories.py, 1506tests) — all green. Ran
ruff check --fixandruff format.generate:source-configs/schema:build(no database/JS toolchain in thisenvironment); the generated config field was hand-applied to match the identical single-key pattern
and confirmed via the passing tests.
Docs update
Doc authored for posthog.com (see note above); needs to be added to that repo.
🤖 Agent context
Autonomy: Fully autonomous
Implemented with PostHog Code. Invoked the
/implementing-warehouse-sources,/documenting-warehouse-sources,and
/writing-testsskills.Key decisions:
created__gtecutoffshowed the row count unchanged, so no endpoint got
supports_incremental=True.ordering=createdisstill sent where a stable
createdfield exists to keep offset pagination deterministic.documentsis not partitioned — its only timestamp is the mutablelast_uploaded_at; partitioningon it would rewrite partitions on every sync.
span_id+start) is documented inVellum's OpenAPI spec, so the composite key and partition key are known; it is opt-in (off by default)
because it multiplies API calls by the deployment count. Its
filters/orderingparams could not beverified without a live authenticated key, so the transport does not rely on them (noted in code).
connector icon).
Created with PostHog Code