feat(data-warehouse): implement skyvern import source

[Gilbert09]

Problem

Skyvern (an open-source AI browser-automation agent with a managed cloud) was scaffolded as a data warehouse source but had no sync logic. Users who run browser-automation tasks and workflows on Skyvern couldn't pull that history into PostHog to analyze run outcomes, failures, and credit spend alongside the rest of their data.

Changes

Fills in the scaffolded skyvern source end-to-end. It talks to Skyvern's FastAPI REST surface (https://api.skyvern.com, configurable for self-hosted) with an x-api-key header, all through the tracked HTTP transport (make_tracked_session).

Five tables:

Table	Endpoint	Sync
workflows	/v1/agents?only_workflows=true	Full refresh
runs	/v1/agents/{workflow_permanent_id}/runs	Incremental on created_at
schedules	/v1/schedules	Full refresh
browser_profiles	/v1/browser_profiles	Full refresh
credentials	/v1/credentials	Full refresh (metadata only, no secrets)

Architecture follows the source.py / settings.py / skyvern.py split. It's a ResumableSource: page-number pagination with resume state saved after each yielded page.

Key design decisions:

• Runs are the interesting table and the only one with a genuine server-side time filter. The global /v1/runs endpoint caps at page 100 (~10k newest runs) and has no timestamp filter, so it can't do complete history or incremental. The per-workflow /v1/agents/{workflow_permanent_id}/runs endpoint accepts created_at_start, so runs enumerates workflows via /v1/agents and fans out per workflow, bounding each with the incremental watermark. That gives both full history and cheap incremental syncs. workflow_run_id is globally unique, so it's a sufficient primary key.
• Incremental only where the API actually filters server-side. I verified the endpoint parameters against Skyvern's published OpenAPI 3 spec: only the per-workflow runs endpoint exposes created_at_start/created_at_end. Every other list endpoint is page/page_size with no time filter, so they ship full-refresh only rather than pretending to be incremental.
• Lookback on runs. Run records keep mutating (status, credits_used, finished_at) until terminal, but the only server-side filter is on the immutable created_at. A 3-day lookback re-pulls a trailing window each run so merge picks up late status changes for recently-created runs.
• Partition keys are stable creation timestamps (created_at), never mutable fields. credentials carries no timestamp, so it has no partition key.
• sort_mode="desc" because Skyvern list endpoints return newest-first and expose no sort param; in desc mode the pipeline checkpoints the incremental watermark at successful job end, which is correct for the fan-out.

Note

I could not curl the live API (no Skyvern key in this environment), so endpoint behavior is verified against the published OpenAPI 3 spec rather than live responses. The main uncertainty is that {workflow_id} in the runs path is documented as a bare string with no description; I pass the stable workflow_permanent_id since that's the identifier runs are grouped by. This is noted in a code comment.

The source stays behind unreleasedSource=True with releaseStatus=alpha.

Not included

• Icon. No frontend/public/services/skyvern.svg exists yet and I don't have a Logo.dev key to fetch one legitimately, so no icon was committed (no placeholder, no hardcoded key). iconPath stays as scaffolded. The source is hidden (unreleasedSource) so this doesn't surface to users yet.
• posthog.com doc. There's no posthog.com checkout in this environment, so the user-facing doc isn't committed here. It needs to land in the posthog.com repo at contents/docs/cdp/sources/skyvern.md (matching the docsUrl). Draft content, written per the /documenting-warehouse-sources skill, is in the collapsed block below.

Draft posthog.com doc — contents/docs/cdp/sources/skyvern.md

---
title: Linking Skyvern as a source
sidebar: Docs
showTitle: true
availability: { free: full, selfServe: full, enterprise: full }
sourceId: Skyvern
beta: true
---

import SourceSetupIntro from "../_snippets/source-setup-intro.mdx"
import SyncModes from "../_snippets/sync-modes.mdx"
import TroubleshootingLink from "../_snippets/dw-troubleshooting-link.mdx"
import AlphaRelease from "../_snippets/alpha-release.mdx"

<AlphaRelease />

Sync your [Skyvern](https://www.skyvern.com) browser-automation data into PostHog: workflow definitions, task and workflow run history (status, failure reasons, credits used, timing), schedules, browser profiles, and stored credential metadata. Use it to analyze run outcomes and cost alongside the rest of your product data.

## Prerequisites

A Skyvern account with an API key. Skyvern Cloud offers self-serve signup; self-hosted deployments work too.

## Adding a data source

<SourceSetupIntro />

You need your Skyvern API key, found in your [Skyvern settings](https://app.skyvern.com/settings) and sent as the `x-api-key` header.

If you self-host Skyvern, set the base URL to your deployment (for example `http://localhost:8000`). Leave it blank to use Skyvern Cloud.

## Sync modes

<SyncModes />

The `runs` table supports incremental sync, filtering on each run's immutable `created_at` with a short lookback window (runs keep mutating status until they finish). All other tables are full refresh.

## Configuration

<SourceParameters />

## Supported tables

<SourceTables />

## Troubleshooting

If a sync fails with a 401 or 403, your API key is invalid or lacks permission. Generate a new key in your Skyvern settings and reconnect.

<TroubleshootingLink />

How did you test this code?

Automated tests only (I'm an agent and could not exercise the live Skyvern API):

• tests/test_skyvern.py (transport) — created_at_start lookback + future-clamp (guards that incremental doesn't refetch all history and doesn't build a filter that drops every row), bare-array vs wrapped-response extraction (guards schedules syncing zero rows), page-number termination + resume checkpoint, the runs fan-out passing created_at_start per workflow and resuming from a workflow bookmark, credential status mapping, and SourceResponse shape per endpoint.
• tests/test_skyvern_source.py (source class) — per-endpoint incremental/append/primary-key capabilities, auth errors being non-retryable, credential plumbing, resumable-manager binding, config field requirements, and that the public-docs table catalog renders from the credential-free placeholder config.

All 36 pass. Ran ruff check/ruff format (clean) and hogli ci:preflight --fix (0 failures).

Automatic notifications

• Publish to changelog?
• Alert Sales and Marketing teams?

🤖 Agent context

Autonomy: Fully autonomous

Authored by Claude (Opus 4.8) via a PostHog Code cloud task. Skills invoked: /implementing-warehouse-sources (followed for the architecture split, base-class choice, incremental rules, and tracked-transport requirement), /writing-tests (ran the value gate before writing tests), and /documenting-warehouse-sources (shaped the draft doc above).

Notable decisions while building:

• Cross-checked the task's API notes against Skyvern's live OpenAPI 3 spec. The spec confirmed the per-workflow runs endpoint has created_at_start (the basis for incremental runs) and revealed a global /v1/schedules endpoint worth syncing, while confirming no other list endpoint has a server-side time filter. Chose full-refresh for those rather than a fake client-side "incremental".
• Considered the global /v1/runs for the runs table but rejected it: it's capped at ~10k newest runs with no time filter, so the per-workflow fan-out is the only path to complete history and incremental.
• The config generator couldn't run here (it needs a DB at Django setup), so the generated_configs.py change was produced by hand to exactly match the generator's deterministic output for the two fields, then verified the diff was limited to those lines.

---

Created with PostHog Code

[github-actions]

Hey @Gilbert09! 👋

It looks like your git author email on this PR isn't your @posthog.com address (owerstom@gmail.com). Since you're on the PostHog team, it's worth pointing your local git author email at your @posthog.com address. Why it matters:

• Consistent work identity in git history — internal tooling that attributes commits to team members keys off your @posthog.com address.
• Keeps team contributions easy to tell apart from external community ones when scanning history.

You can fix it for this repo with:

git config user.email "you@posthog.com"

Or set it globally with git config --global user.email "you@posthog.com". No need to redo this PR — just a nudge for next time. 🙂

[greptile-apps]

_{Reviews (1): Last reviewed commit: "feat(data-warehouse): implement skyvern ..." | Re-trigger Greptile}

[trunk-io]

     

View Full Report ↗︎ ⋅ Docs

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

[estefaniarabadan]

Isolated new data-warehouse source scaffold — self-contained under its own source directory with only the expected shared-registry additions (SOURCES.md, generated_configs.py, icon), tests included, CI green. Reviewed as part of a batch. LGTM 👍

[github-actions]

🤖 CI report

⚠️ Backend coverage — 92.0% of changed backend lines covered — 29 uncovered

🧪 Backend test coverage

Patch coverage — changed backend lines (products + core): ██████████████████░░ 92.0% (358 / 387)

File	Patch	Uncovered changed lines
products/warehouse_sources/backend/temporal/data_imports/sources/skyvern/skyvern.py	81.8%	62–69, 86, 112–113, 115–116, 118–120, 122, 131, 145–146, 167, 174, 194, 225, 252, 258–259, 263
products/warehouse_sources/backend/temporal/data_imports/sources/skyvern/source.py	97.4%	158

🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 29260131859 -n patch-coverage), or the coverage-data block at the end of this comment.

Per-product line coverage (touched products)

Product	Coverage	Lines
warehouse_sources	███████████████████░ 96.0%	205,295 / 213,794

Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.

[deployment-status-posthog]

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-07-13 15:59 UTC	Run
prod-us	✅ Deployed	2026-07-13 16:18 UTC	Run
prod-eu	✅ Deployed	2026-07-13 16:23 UTC	Run