feat(data-warehouse): implement phyllo import source

[Gilbert09]

Problem

Phyllo (a unified API for creator economy data across YouTube, Instagram, TikTok, Twitch, Spotify, etc.) was scaffolded as a warehouse source but had no sync logic, so users can't pull their creator, content, or income data into the Data warehouse.

Why: fill in the scaffolded Phyllo connector end to end so it can sync real data and move it from the Scaffolded to the Implemented list.

Changes

Implements the Phyllo source following the standard source.py / settings.py / phyllo.py split:

• ResumableSource over the Phyllo v1 REST API using requests via make_tracked_session(), HTTP Basic auth (client ID + secret), and limit-offset pagination with resume state saved after each yielded batch.
• Seven streams in settings.py: work_platforms, users, accounts, and profiles are top-level listings; social_contents, income_transactions, and income_payouts require an account_id, so they fan out over the connected accounts (iterated in sorted order so resume checkpoints are deterministic).
• All streams are full refresh. Phyllo documents from_date/to_date filters on the content and income endpoints, but I could not verify they filter server-side without live credentials, so per the conservative rule those ship as full refresh (noted in settings.py).
• Connection form: client ID, client secret, and a production/sandbox environment select (Phyllo credentials are environment-specific). Credential validation probes /v1/work-platforms; 401/403 on either host are registered as non-retryable errors.
• Rate limiting: Phyllo caps requests per second and returns Retry-After on 429 - the fetch path honors the header (capped) before tenacity's exponential backoff takes over.
• canonical_descriptions.py documents all seven tables and their well-known columns from the Phyllo API reference, and lists_tables_without_credentials = True so the public docs render the table catalog.
• SOURCES.md: moved phyllo from Scaffolded to the Implemented table (HTTP / requests / tracked).
• Kept behind unreleasedSource=True with releaseStatus=alpha.

Note

Endpoint verification was limited to unauthenticated probes against both hosts (reachability plus the 401 error shape) since no API credentials were available. Response envelope ({"data": [...]}), max page size, and the required account_id params come from the public API docs; the payload guard raises a retryable error if the envelope ever differs, and the uncertainty is noted in code comments.

The user-facing doc for the posthog.com repo (contents/docs/cdp/sources/phyllo.md, matching the docsUrl) still needs to land there - this workspace has no posthog.com checkout, so audit_source_docs couldn't run. Ready-to-commit content:

posthog.com doc: contents/docs/cdp/sources/phyllo.md

---
title: Linking Phyllo as a source
sidebar: Docs
showTitle: true
availability: { free: full, selfServe: full, enterprise: full }
sourceId: Phyllo
beta: true
---

import SourceSetupIntro from "../_snippets/source-setup-intro.mdx"
import SyncModes from "../_snippets/sync-modes.mdx"
import TroubleshootingLink from "../_snippets/dw-troubleshooting-link.mdx"
import AlphaRelease from "../_snippets/alpha-release.mdx"

<AlphaRelease />

The Phyllo connector syncs creator economy data into the PostHog Data warehouse: the users and platform accounts linked through Phyllo Connect, their identity profiles, published content with engagement metrics, and income transactions and payouts.

## Prerequisites

You need a Phyllo developer account with API credentials (a client ID and secret). Credentials are environment-specific, so use the pair matching the environment (production or sandbox) you want to sync from.

## Adding a data source

<SourceSetupIntro />

You'll need your Phyllo **client ID** and **client secret**, found in the [Phyllo developer dashboard](https://dashboard.getphyllo.com) under **API credentials**, plus the environment they belong to.

## Sync modes

<SyncModes />

All Phyllo tables currently sync as full refresh.

## Configuration

<SourceParameters />

## Supported tables

<SourceTables />

The content and income tables are pulled per connected account, so sync time grows with the number of accounts your creators have linked.

## Troubleshooting

If the connection fails with an authentication error, double-check that the client ID and secret match the selected environment. Sandbox credentials don't authenticate against production, and vice versa.

<TroubleshootingLink />

How did you test this code?

• pytest products/warehouse_sources/backend/temporal/data_imports/sources/phyllo/tests/ - 61 passed.
  • tests/test_phyllo.py (transport): guards offset advancement and short-page termination (a regression here loops forever or drops pages), resume-from-saved-state for both top-level and mid-account fan-out (including a disconnected saved account, where a leaked offset would silently skip rows), state saved only after yielding (saving before loses data on crash), 429 Retry-After handling, retryable vs terminal status mapping, and credential validation status mapping.
  • tests/test_phyllo_source.py (source class): schema catalog wiring, secret field config, non-retryable error matching on both hosts (and not matching transient 429/500), argument plumbing into the transport, and the public-docs table rendering path (which exercises the placeholder-config construction).
• pytest products/warehouse_sources/backend/temporal/data_imports/sources/tests/ - 1353 passed (registry, categories, config generation).
• ruff check / ruff format clean; hogli ci:preflight --fix reports 0 failures; makemigrations --check detects no pending migrations.
• I was not able to run a live sync - no Phyllo API credentials were available (see the note above).

Automatic notifications

• Publish to changelog?
• Alert Sales and Marketing teams?

Docs update

posthog.com doc content is included above and needs a companion PR in the posthog.com repo. The source stays behind unreleasedSource=True, so nothing user-visible changes yet.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

• Built with Claude Code. Skills invoked: /implementing-warehouse-sources (workflow, architecture contract, testing expectations), /documenting-warehouse-sources (doc template), /writing-tests (test value gate).
• Followed the recent secoda/huntr sources as the reference shape (ResumableSource + requests + tracked session).
• Chose plain requests over rest_source.RESTClient to match the reference sources and keep the per-account fan-out with resumable checkpoints explicit.
• Considered adding WebhookSource (Phyllo has a manageable webhooks API) but deferred it: webhook payload shapes couldn't be verified without credentials, and none of the recent reference sources ship webhook support in their initial PR. Incremental sync via from_date filters was likewise deferred until the filters can be verified server-side with real credentials.
• generated_configs.py was produced by generate_source_configs; per the known generator drift, unrelated rewrites were reverted so the diff touches only PhylloSourceConfig.

---

Created with PostHog Code

[github-actions]

Hey @Gilbert09! 👋

It looks like your git author email on this PR isn't your @posthog.com address (owerstom@gmail.com). Since you're on the PostHog team, it's worth pointing your local git author email at your @posthog.com address. Why it matters:

• Consistent work identity in git history — internal tooling that attributes commits to team members keys off your @posthog.com address.
• Keeps team contributions easy to tell apart from external community ones when scanning history.

You can fix it for this repo with:

git config user.email "you@posthog.com"

Or set it globally with git config --global user.email "you@posthog.com". No need to redo this PR — just a nudge for next time. 🙂

[greptile-apps]

_{Reviews (1): Last reviewed commit: "feat(data-warehouse): implement phyllo i..." | Re-trigger Greptile}

[trunk-io]

     

View Full Report ↗︎ ⋅ Docs

[deployment-status-posthog]

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-07-09 11:08 UTC	Run
prod-us	✅ Deployed	2026-07-09 11:31 UTC	Run
prod-eu	✅ Deployed	2026-07-09 11:40 UTC	Run