feat(data-warehouse): implement fly_io import source - #70000
Conversation
|
Hey @Gilbert09! 👋 It looks like your git author email on this PR isn't your
You can fix it for this repo with: git config user.email "you@posthog.com"Or set it globally with |
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
PR overviewAll previously flagged issues have been addressed. No open security concerns remain on this pull request. Security reviewNo open security issues remain on this pull request. Fixed/addressed: 2 · PR risk: 0/10 |
estefaniarabadan
left a comment
estefaniarabadan
left a comment
There was a problem hiding this comment.
Isolated new data-warehouse source scaffold — self-contained under its own source directory with only the expected shared-registry additions (SOURCES.md, generated_configs.py, icon), tests included, CI green. Reviewed as part of a batch. LGTM 👍
d1421aa to
0936fc5
Compare
🤖 CI report
|
| File | Patch | Uncovered changed lines |
|---|---|---|
products/warehouse_sources/backend/temporal/data_imports/sources/fly_io/fly_io.py |
96.2% | 190–191, 234, 239 |
🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 29350189059 -n patch-coverage), or the coverage-data block at the end of this comment.
Per-product line coverage (touched products)
| Product | Coverage | Lines |
|---|---|---|
warehouse_sources |
███████████████████░ 96.1% |
215,006 / 223,695 |
Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.
Fill in the scaffolded Fly.io data warehouse source end to end: syncs apps,
machines, and volumes for an organization from the Fly.io Machines API.
Uses the org-level aggregate endpoints (GET /apps, GET /orgs/{org_slug}/machines,
GET /orgs/{org_slug}/volumes) instead of per-app fan-out — each returns every
resource in the org in one cursor-paginated call and stamps app_name onto each
row, which is fewer requests and stays within Fly.io's per-action rate limits.
SimpleSource, full refresh only (no verified server-side time filter), routed
through make_tracked_session. Ships alpha and behind unreleasedSource.
Generated-By: PostHog Code
Task-Id: 2af164a9-60a3-4a93-a3a1-e1226e106277
Address review feedback on the fly_io source: - Percent-encode organization_slug in path-scoped URLs so a reserved character can't retarget requests to a different API path than the one credential validation checked. - Raise on an unexpected top-level response shape instead of silently syncing zero rows. - Add connection_host_fields = ["organization_slug"] so changing the org re-requires the token, preventing a preserved token from being retargeted at another org. Also fix two mypy errors in the tests (HTTPError needs response=, and a var-annotation for pages). Generated-By: PostHog Code Task-Id: 12e6e049-0cc3-48ca-8473-3c8e16e2c86b
0936fc5 to
4c2ed0a
Compare
A Fly machine's config can embed deployment secrets (env vars, per-process env, and inline file contents in config.files[].raw_value). Reduce each machine row to a safe operational config allowlist before yielding, and exclude the machines stream from HTTP sample capture, so secrets reach neither the warehouse nor the sample-capture pipeline. Generated-By: PostHog Code Task-Id: b5ea24fe-da19-47b2-ba5c-be66bcb7579f
![hex-security-app[bot]](https://avatars.githubusercontent.com/in/2665233?s=60&v=4){kind=small}
The machine config allowlist kept `metadata`, `services`, and `checks` verbatim. Fly metadata is a free-form user key/value map, and service and health-check definitions can carry request headers, so a value like `metadata.api_token` or a check `Authorization` header still reached the warehouse where any project member could query it. Keep only Fly's platform-set metadata keys and recursively drop every nested `headers` map from the config before a machine row is yielded. Update the canonical `config` description and add tests for both vectors. Generated-By: PostHog Code Task-Id: 18ff2c05-99a0-40c6-8bf3-33c01102248c
Remove unreleasedSource=True so the finished source is visible in the connector catalog, and drop any test asserting the hidden state.
2 participants
Problem
Fly.io was scaffolded as a data warehouse source but had no sync logic (empty
source.py,unreleasedSource=True). Fly.io is a popular developer cloud, and teams want to pull their infrastructure inventory (apps, machines, volumes) into the warehouse to query alongside product data and for fleet auditing.Changes
Fills in the Fly.io source end to end against the Fly.io Machines API (
https://api.machines.dev/v1, Bearer auth):apps,machines,volumes.source.py/settings.py/fly_io.pysplit.SimpleSource, full-refresh only.GET /apps?org_slug=,GET /orgs/{org_slug}/machines,GET /orgs/{org_slug}/volumes) rather than a per-app fan-out for machines/volumes. Each returns every resource in the org in a single cursor-paginated call and stampsapp_nameonto each row, so it carries app context with far fewer requests and stays comfortably within Fly.io's per-action rate limits for orgs with many apps.make_tracked_session();tenacityretries on 429/5xx and transient network errors;get_non_retryable_errors()covers 401/403.machines/volumespartition on the stablecreated_at;appscarries no timestamp so it isn't partitioned. Primary keyid(globally unique per Fly.io).lists_tables_without_credentials = Trueso the public docs render the supported-tables catalog.releaseStatus=alphaand kept behindunreleasedSource=True(not yet verified against a live account end to end).SOURCES.mdupdated (moved from Scaffolded to Implemented, HTTP / requests / tracked).Full refresh, not incremental: the org machines/volumes endpoints document an
updated_afterparam, but the API statescursortakes precedence over it (so it could only bound the first page) and it could not be curl-verified to actually filter without a live token. Per the implementing-warehouse-sources guidance, all streams ship full refresh. This is noted in a code comment.Not included / follow-ups
frontend/public/services/fly_io.*asset was added — the sanctioned path is Logo.dev with a user-supplied API key, which wasn't available, so no placeholder or hardcoded key was committed.iconPathstill points atfly_io.png; the asset needs to be added separately.contents/docs/cdp/sources/fly-io.mdwithsourceId: FlyIo— matching thedocsUrlset on the source.audit_source_docscould not be run here.How did you test this code?
Automated tests only (no live Fly.io account was available for manual sync verification):
tests/test_fly_io.py(transport): org-in-path vs org-in-query URL building,validate_credentialsstatus mapping,_fetch_pageretry-vs-raise behavior, cursor pagination (followsnext_cursor, terminates, apps stays single-request), and per-endpointSourceResponsepartition/primary-key shape.tests/test_fly_io_source.py(source class): source type, config fields (token is aPASSWORD/secret), full-refresh schemas, credential/pipeline plumbing, canonical-descriptions coverage, documented-tables rendering, and non-retryable-error matching.Ran the targeted suite (
32 passed) plus the registry-widetest_source_categories/ config-generator suite.ruff checkandruff formatpass on all changed files.pnpm run generate:source-configscould not run in this sandbox (no DB + a missing optional dep during full source import), so theFlyIoSourceConfigentry ingenerated_configs.pywas written to match the generator's deterministic output for two required string fields — CI regenerates it.hogli/ci:preflightweren't available in this environment.The enum, schema, and generated-config wiring for the source type were already merged on master, so no new Django migration was needed.
Docs update
Doc needs to be added in the posthog.com repo (see follow-ups above).
🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Authored with Claude Code. Skills invoked:
/implementing-warehouse-sources(workflow, architecture contract, tracked-transport rule),/writing-tests(test value gate), and/documenting-warehouse-sources(doc template).Key decision: the original research notes assumed no org-level machines/volumes endpoints and a per-app fan-out. Inspecting the live Fly.io OpenAPI spec showed
GET /orgs/{org_slug}/machinesand.../volumesdo exist, carryapp_name+created_at/updated_at, and support cursor pagination — so the implementation uses those instead of fan-out (fewer requests, better rate-limit behavior). The org endpoints also documentupdated_after, but sincecursortakes precedence over it and it couldn't be verified against a live token, incremental sync was deliberately left off in favor of full refresh.Created with PostHog Code