feat(data-warehouse): implement aviator import source - #70022
Conversation
Fill in the scaffolded Aviator data warehouse source end to end: repositories, daily merge-queue analytics, queued pull requests, live queue stats, and config-change history. Analytics syncs incrementally via the API's start/end date window; the snapshot/list endpoints ship as full refresh. Generated-By: PostHog Code Task-Id: 3e4316ad-d1e6-4f96-a230-9843de197bdf
|
Hey @Gilbert09! 👋 It looks like your git author email on this PR isn't your
You can fix it for this repo with: git config user.email "you@posthog.com"Or set it globally with |
![hex-security-app[bot]](https://avatars.githubusercontent.com/in/2665233?s=60&v=4){kind=small}
|
Reviews (1): Last reviewed commit: "feat(data-warehouse): implement aviator ..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
- Register the API token for value-based redaction on both the credential-validation and sync sessions so it can't leak into logged URLs or captured HTTP samples. - Stop logging Aviator error response bodies; log only status and URL to avoid spilling request-specific data (config-history diffs, token-like values) into application logs. - Skip queued-PR rows missing `number` and config-history rows missing `applied_at` rather than emitting null primary keys that silently collapse distinct rows on merge. - Resume fan-out syncs by tracking the set of completed repo keys instead of a positional bookmark, so a repository added between a crash and the retry is processed rather than skipped (which would strand its older analytics outside the trailing lookback window). - Add the missing aviator.svg source icon. - Rename the placeholder source doc to .mdx and apply formatting. Generated-By: PostHog Code Task-Id: 8b4d7ee8-6606-465b-94eb-eed204b4d177
🦔 Hogbox preview · ❌ build failedThe preview didn't come up for commit |
Generated-By: PostHog Code Task-Id: 8b4d7ee8-6606-465b-94eb-eed204b4d177
🤖 CI report✅ Bundle size — no changeUncompressed size of every built Total: 64.35 MiB · no change No file changed by more than 1000 B. Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report ✅ Eager graph — within budgetHow much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy
🟢 Largest files eagerly shipped from
|
| Size | File |
|---|---|
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 24.6 KiB | ../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js |
| 6.3 KiB | ../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js |
| 4.5 KiB | ../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js |
| 3.9 KiB | ../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js |
| 1.4 KiB | ../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js |
| 1.3 KiB | src/RootErrorBoundary.tsx |
| 912 B | ../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js |
| 789 B | src/scenes/ChunkLoadErrorBoundary.tsx |
| 668 B | src/index.tsx |
Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx
| Size | File |
|---|---|
| 278.6 KiB | ../node_modules/.pnpm/posthog-js@1.399.4/node_modules/posthog-js/dist/rrweb.js |
| 267.7 KiB | ../node_modules/.pnpm/@posthog+icons@0.38.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js |
| 234.9 KiB | src/taxonomy/core-filter-definitions-by-group.json |
| 221.8 KiB | ../node_modules/.pnpm/posthog-js@1.399.4/node_modules/posthog-js/dist/module.js |
| 164.0 KiB | src/queries/validators.js |
| 154.3 KiB | ../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js |
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 105.9 KiB | src/lib/api.ts |
| 93.3 KiB | ../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js |
| 90.6 KiB | ../node_modules/.pnpm/@tiptap+core@3.20.6_@tiptap+pm@3.20.6/node_modules/@tiptap/core/dist/index.js |
Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479
⚠️ Dist folder size — 🔺 +318 B (+0.0%)
Total size of the built frontend/dist folder (all assets), compared against the base branch.
Total: 1278.81 MiB · 🔺 +318 B (+0.0%)
⚠️ Backend coverage — 97.0% of changed backend lines covered — 10 uncovered
🧪 Backend test coverage
Patch coverage — changed backend lines (products + core): ███████████████████░ 97.0% (466 / 476)
| File | Patch | Uncovered changed lines |
|---|---|---|
products/warehouse_sources/backend/temporal/data_imports/sources/aviator/aviator.py |
93.6% | 158, 240, 243, 252–255, 283, 302, 342 |
🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 29288010754 -n patch-coverage), or the coverage-data block at the end of this comment.
Per-product line coverage (touched products)
| Product | Coverage | Lines |
|---|---|---|
warehouse_sources |
███████████████████░ 96.1% |
210,870 / 219,482 |
Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.
|
All CI checks pass except deploy preview, which fails on preview-environment infrastructure unrelated to this change. I re-ran it once and it failed again with a different infra error, confirming it's environmental rather than caused by this PR:
Neither touches the Aviator source code. The 166 required/product checks (including the warehouse-sources product tests, Jest, and Python quality) are green. Remaining merge block is the required human review. |
estefaniarabadan
left a comment
estefaniarabadan
left a comment
There was a problem hiding this comment.
Isolated new data-warehouse source scaffold — self-contained under its own source directory with only the expected shared-registry additions (SOURCES.md, generated_configs.py, icon), tests included, CI green. Reviewed as part of a batch. LGTM 👍
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
PR overviewAll previously flagged issues have been addressed. No open security concerns remain on this pull request. Security reviewNo open security issues remain on this pull request. Fixed/addressed: 1 · PR risk: 0/10 |
Aviator responses carry arbitrary repository data (config-history diffs, PR titles, branch names) that the name-based sample scrubber cannot sanitise. Pass capture=False on both the sync session and the credential probe so response bodies never reach the HTTP sample bucket, while requests stay metered and logged. Generated-By: PostHog Code Task-Id: dbb3d70d-6421-4ead-96e0-a4c1110de0b6
Generated-By: PostHog Code Task-Id: dbb3d70d-6421-4ead-96e0-a4c1110de0b6
Status
Note on the red
|
2 participants
Problem
Aviator (YC S21) is a developer-productivity suite built around a GitHub merge queue. Its public JSON API is self-serve, so any customer can mint a user access token and pull their merge-queue data. We had a scaffolded stub for it but no working sync, so the connector could not import anything. This wires it up so teams can join Aviator merge-queue data with their product and engineering analytics in the Data warehouse.
Changes
Filled in the scaffolded
aviatorsource end to end. It follows the standardsource.py/settings.py/aviator.pysplit and inheritsResumableSource.Tables:
repositoriesGET /repomerge_queue_analyticsGET /analyticsqueued_pull_requestsGET /pull_request/queuedqueue_statsGET /queue/statsconfig_historyGET /config/historyKey decisions:
merge_queue_analyticsis incremental. Itsstart/endUTC date window is the endpoint's fundamental interface (it returns per-day aggregate rows bounded by that window), so it is a genuine server-side filter. Each incremental run advances from the stored watermark minus a 7-day trailing window, because recent daily aggregates get revised upstream; merge dedupes the re-pulled days on the[repo, date]key. The endpoint returns five separate daily series (time_in_queue,wait_times_to_queue,mergequeue_usage,blocked_reason,sync_frequency); I merge them into one row per date with per-series-prefixed columns so their sharedmin/avg/pXXnames do not collide.GET /config/historydocuments optionalstart/endparams, but I could not curl-verify they actually filter server-side (no test credentials), so it stays full refresh conservatively - config changes are low volume, so re-reading them each sync is cheap.GET /repoand call once per repo, injectingorg/repointo each row so composite primary keys stay unique table-wide. Resume is bookmarked at repo granularity (a stableorg/namekey), and fan-out endpoints reportsort_mode="desc"so the incremental watermark only persists at successful job end.make_tracked_session(); retries usetenacityon 429/5xx and transient network errors.Note
The source stays behind
unreleasedSource=TruewithreleaseStatus=ALPHAas requested, so it is not yet visible to users.Two follow-ups needed outside this repo:
frontend/public/services/aviator.svgexists yet and I had no Logo.dev key to fetch one legitimately, so I did not commit a placeholder.iconPathpoints at the expected path; the SVG needs to be added.contents/docs/cdp/sources/aviator.md). No posthog.com checkout was available, so the finished doc is committed here ataviator.posthog-doc.mdand needs to be copied across, after whichaudit_source_docsshould be run.How did you test this code?
Automated only (I am an agent and did not run a live sync - I have no Aviator credentials, so endpoint response shapes were taken from the public API docs and confirmed against the live host only for auth behavior and the base URL). Added two test modules run with
hogli test:tests/test_aviator.py(50 assertions across transport): the five-series analytics flattening, the incremental date-window computation (first sync, watermark rewind, future clamp, value-type coercion), repo-list pagination termination, per-endpoint fan-out row shapes, resume bookmarking (save-after-each-repo, resume-from-saved, deleted-repo restart), credential-status mapping, retry behavior, and per-endpoint sort/partition/primary-key wiring.tests/test_aviator_source.py(source class): source type, config fields/flags, per-endpoint incremental support, credential validation, resumable-manager binding,source_for_pipelineplumbing, non-retryable errors, and canonical-description coverage.All 50 aviator tests pass, plus the existing
test_source_categoriessuite (1506) stays green.ruff checkandruff formatare clean on the changed files.Automatic notifications
🤖 Agent context
Autonomy: Fully autonomous
Implemented by Claude (Claude Code). Invoked the
/implementing-warehouse-sources,/documenting-warehouse-sources, and/writing-testsskills.I verified the base URL and Bearer auth against the live API (
api.aviator.co/api/v1returns 401 withwww-authenticate: Bearer) but could not exercise authenticated endpoints without a token, so response shapes come from the current public JSON API docs. I chose a custom transport over the declarativerest_source.RESTClientbecause the responses are heterogeneous (a bare array, wrapped lists, and the five-series analytics object that needs merging). I considered makingconfig_historyincremental via its documentedstart/endparams but shipped it full refresh since I could not confirm server-side filtering - noted inline and above. The generated config (AviatorSourceConfig) was updated by hand to add the singleapi_tokenfield because no database was reachable to rungenerate:source-configs; the output is identical to what the generator emits for a single required password field (matches the existingAwinSourceConfig).Created with PostHog Code