feat(data-warehouse): implement llama_cloud import source - #70712
Merged
Conversation
Contributor
|
Hey @Gilbert09! 👋 It looks like your git author email on this PR isn't your
You can fix it for this repo with: git config user.email "you@posthog.com"Or set it globally with |
Contributor
🦔 Hogbox preview · ✅ ready▶ Open the preview
commit |
Contributor
🤖 CI report✅ Bundle size — no changeUncompressed size of every built Total: 64.77 MiB · no change No file changed by more than 1000 B. Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report ✅ Eager graph — within budgetHow much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy
🟢 Largest files eagerly shipped from
|
| Size | File |
|---|---|
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 24.6 KiB | ../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js |
| 6.3 KiB | ../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js |
| 4.5 KiB | ../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js |
| 3.9 KiB | ../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js |
| 1.4 KiB | ../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js |
| 1.3 KiB | src/RootErrorBoundary.tsx |
| 912 B | ../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js |
| 789 B | src/scenes/ChunkLoadErrorBoundary.tsx |
| 762 B | src/index.tsx |
Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx
| Size | File |
|---|---|
| 281.3 KiB | ../node_modules/.pnpm/posthog-js@1.402.3/node_modules/posthog-js/dist/rrweb.js |
| 267.7 KiB | ../node_modules/.pnpm/@posthog+icons@0.38.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js |
| 235.5 KiB | src/taxonomy/core-filter-definitions-by-group.json |
| 222.9 KiB | ../node_modules/.pnpm/posthog-js@1.402.3/node_modules/posthog-js/dist/module.js |
| 164.0 KiB | src/queries/validators.js |
| 154.3 KiB | ../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js |
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 105.8 KiB | src/lib/api.ts |
| 93.3 KiB | ../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js |
| 92.7 KiB | ../packages/quill/packages/quill/dist/index.js |
Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479
✅ Dist folder size — 🔺 +1.9 KiB (+0.0%)
Total size of the built frontend/dist folder (all assets), compared against the base branch.
Total: 1312.64 MiB · 🔺 +1.9 KiB (+0.0%)
✅ Backend coverage — all changed backend lines covered
🧪 Backend test coverage
Patch coverage — changed backend lines (products + core): ████████████████████ 100.0% (444 / 444)
All changed backend lines are covered ✅
Per-product line coverage (touched products)
| Product | Coverage | Lines |
|---|---|---|
demo |
███████████░░░░░░░░░ 56.2% |
1,497 / 2,663 |
tasks |
█████████████░░░░░░░ 67.4% |
25,560 / 37,895 |
signals |
████████████████░░░░ 79.1% |
19,033 / 24,073 |
data_modeling |
████████████████░░░░ 80.0% |
4,834 / 6,045 |
cdp |
████████████████░░░░ 80.7% |
3,118 / 3,864 |
notebooks |
█████████████████░░░ 84.3% |
6,343 / 7,520 |
agent_platform |
█████████████████░░░ 84.7% |
3,273 / 3,862 |
cohorts |
█████████████████░░░ 86.1% |
4,022 / 4,671 |
actions |
█████████████████░░░ 86.6% |
717 / 828 |
product_tours |
█████████████████░░░ 87.5% |
1,266 / 1,447 |
exports |
██████████████████░░ 88.3% |
6,891 / 7,800 |
conversations |
██████████████████░░ 88.9% |
16,129 / 18,133 |
dashboards |
██████████████████░░ 89.1% |
5,719 / 6,418 |
mcp_analytics |
██████████████████░░ 89.1% |
2,502 / 2,807 |
error_tracking |
██████████████████░░ 89.6% |
9,718 / 10,852 |
alerts |
██████████████████░░ 89.9% |
3,638 / 4,046 |
engineering_analytics |
██████████████████░░ 90.1% |
5,105 / 5,665 |
streamlit_apps |
██████████████████░░ 90.4% |
2,499 / 2,764 |
slack_app |
██████████████████░░ 90.6% |
9,511 / 10,503 |
marketing_analytics |
██████████████████░░ 90.8% |
11,514 / 12,684 |
product_analytics |
██████████████████░░ 91.1% |
5,599 / 6,143 |
data_warehouse |
██████████████████░░ 92.1% |
18,133 / 19,683 |
workflows |
██████████████████░░ 92.4% |
5,148 / 5,574 |
web_analytics |
███████████████████░ 92.7% |
13,624 / 14,691 |
ai_observability |
███████████████████░ 92.8% |
14,868 / 16,019 |
surveys |
███████████████████░ 92.9% |
5,687 / 6,120 |
posthog_ai |
███████████████████░ 93.2% |
1,322 / 1,418 |
approvals |
███████████████████░ 93.3% |
3,395 / 3,640 |
reminders |
███████████████████░ 93.4% |
468 / 501 |
early_access_features |
███████████████████░ 93.8% |
848 / 904 |
endpoints |
███████████████████░ 94.1% |
8,606 / 9,143 |
skills |
███████████████████░ 94.4% |
2,827 / 2,995 |
revenue_analytics |
███████████████████░ 94.5% |
3,598 / 3,809 |
review_hog |
███████████████████░ 94.6% |
6,532 / 6,905 |
logs |
███████████████████░ 95.3% |
9,528 / 9,994 |
experiments |
███████████████████░ 95.6% |
24,171 / 25,288 |
replay_vision |
███████████████████░ 95.7% |
13,354 / 13,952 |
annotations |
███████████████████░ 96.2% |
732 / 761 |
warehouse_sources |
███████████████████░ 96.3% |
233,491 / 242,583 |
feature_flags |
███████████████████░ 96.3% |
16,038 / 16,661 |
user_interviews |
███████████████████░ 96.4% |
2,242 / 2,325 |
data_catalog |
███████████████████░ 97.1% |
2,034 / 2,095 |
customer_analytics |
███████████████████░ 97.2% |
7,480 / 7,698 |
Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.
Implements the LlamaCloud (LlamaIndex) import source end to end: resumable cursor-paginated transport over the tracked HTTP session, ten endpoints (parse/extract/classify/split/sheets jobs, batches, projects, pipelines, files, usage metrics), server-side incremental sync on created_at/day, canonical table descriptions, icon, and source + transport tests. Ships behind unreleasedSource with releaseStatus=alpha. Generated-By: PostHog Code Task-Id: 0d0ea3f2-4ebc-47f9-ad3b-bab99644f935
Bind tenacity's __wrapped__ once with the established type-ignore pattern and pass the response kwarg requests' HTTPError stub requires. Generated-By: PostHog Code Task-Id: 0d0ea3f2-4ebc-47f9-ad3b-bab99644f935
Covers the _parse_error_detail fallback path where a gateway returns a non-JSON error body, so validate_credentials surfaces the raw text instead of crashing. Generated-By: PostHog Code Task-Id: 9cc974e0-19d5-49ff-a187-c3c942af5bd7
Remove unreleasedSource=True so the finished source is visible in the connector catalog, and drop any test asserting the hidden state.
Gilbert09
force-pushed
the
posthog-code/llama-cloud-source
branch
from
July 15, 2026 21:43
aa3a5c5 to
f03327e
Compare
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
Contributor
PR overviewAll previously flagged issues have been addressed. No open security concerns remain on this pull request. Security reviewNo open security issues remain on this pull request. Fixed/addressed: 4 · PR risk: 0/10 |
…ines The pipelines endpoint returns full pipeline definitions whose nested embedding and data-sink configs can carry third-party credentials (provider API keys, sink connection secrets). Yielding them unchanged would persist those secrets into the warehouse. Redact auth-bearing keys by name before yielding, reusing the same denylist the HTTP sample-capture path already scrubs by, gated behind a per-endpoint flag so only the config-style pipelines endpoint pays for it. Generated-By: PostHog Code Task-Id: 21f24636-a5ba-426b-ae64-c182fc60db03
![hex-security-app[bot]](https://avatars.githubusercontent.com/in/2665233?s=60&v=4){kind=small}
…crets Project the pipelines endpoint onto its documented, non-sensitive fields instead of relying on a key-name denylist. A denylist can't enumerate every secret-bearing shape (generic token/secret fields, credentials objects, auth headers), so an allowlist keeps them out of the warehouse structurally. Also opt the endpoint out of HTTP sample capture, since the sampler observes the upstream response before row projection runs. Generated-By: PostHog Code Task-Id: 21f24636-a5ba-426b-ae64-c182fc60db03
… webhook secrets The /api/v1/sheets/jobs listing can carry webhook credentials under nested parameters.webhook_configurations (signing secrets, auth headers). With output_fields unset the whole object was persisted, exposing those secrets to warehouse readers. Project sheets_jobs rows onto the documented, non-sensitive job metadata (matching canonical_descriptions) and disable HTTP sample capture, mirroring the pipelines endpoint fix. Generated-By: PostHog Code Task-Id: 19edced1-886c-4d07-af72-3439c70ab1bc
Generated-By: PostHog Code Task-Id: 3719b52a-2e64-4237-b7f3-5c103a41aac3
The files endpoint returns rows with a presigned download_url (and form fields) that grant access to the private source document. Project files rows onto the documented metadata fields and disable HTTP sample capture, matching the treatment already applied to the pipelines and sheets endpoints, so those credentials never reach the warehouse. Generated-By: PostHog Code Task-Id: c45c82e7-4037-4cdd-8bdd-2bb47a1a97bc
…closed Job and config endpoints can return arbitrary customer document content (e.g. parse-job user_metadata, extraction results) or embedded credentials that the name-based sample scrubbers can't remove. With capture_http_samples defaulting to True, those endpoints inherited capture-on. Flip the default to False and opt in only the metadata-only endpoints (projects, usage_metrics) whose response schema is limited to safe metadata. Add a regression test locking the fail-closed default and the capturing-endpoint set. Generated-By: PostHog Code Task-Id: 19edced1-886c-4d07-af72-3439c70ab1bc
Generated-By: PostHog Code Task-Id: 3719b52a-2e64-4237-b7f3-5c103a41aac3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
LlamaCloud (LlamaIndex's hosted document-processing and RAG platform, home of LlamaParse) was scaffolded as a warehouse source but had no sync logic. Teams running document pipelines on LlamaCloud can't analyze their parse/extract/classify jobs, pipelines, or credit consumption alongside their product data in PostHog.
Changes
Implements the
llama_cloudsource end to end, following the standardsource.py/settings.py/llama_cloud.pysplit:parse_jobs,extract_jobs,classify_jobs,batches,split_jobs,sheets_jobs(incremental oncreated_at),usage_metrics(incremental onday, with credit consumption), andprojects,pipelines,files(full refresh, the API exposes no timestamp filter for them).page_size/page_tokencursor pagination. The resume token is persisted after each yielded page, so a crashed sync re-yields the last page (merge dedupes) instead of restarting.created_at_on_or_after/day_on_or_afterfilters. The list endpoints accept no sort param and don't document ordering, so incremental endpoints reportsort_mode="desc"and the watermark persists only when a sync completes, which is correct regardless of actual server order. Job schemas declare a 24h default incremental lookback so status transitions on recently created jobs are picked up (jobs are keyed and merged onid).regionselect (NA/EU) mapping to the two regional hosts.usage_metricsrequires anorganization_idquery param, resolved at sync time from the key's project via/api/v2/projects.make_tracked_session(); retryable 429/5xx handling layered with tenacity on top of the tracked transport's urllib3 retry.validate_credentialsprobes/api/v2/projectsand surfaces the API's own error detail (which points at region mismatches, the most common failure).get_non_retryable_errorsstops retries on 401/403 for both regional hosts.lists_tables_without_credentials = True, so the public doc renders the full table catalog.frontend/public/services/llama_cloud.svg.unreleasedSource=TruewithreleaseStatus=alpha.Companion doc PR: PostHog/posthog.com#18456 (
docsUrlpoints at/docs/cdp/sources/llama-cloud).How did you test this code?
created_at_on_or_after/day_on_or_afterfilters, response shapes (items/next_page_token/total_size), and item schemas per endpoint; curl-probed both regional hosts unauthenticated and with an invalid key to confirm the 401 semantics used byvalidate_credentialsandget_non_retryable_errors.sort_mode="desc"(watermark persisted only at successful completion) rather than trusting ascending order.tests/test_llama_cloud_source.py(26 cases): schema catalog and incremental semantics per endpoint (guards a full-refresh endpoint silently advertising incremental), non-retryable error matching for both regional hosts (guards endless retries on bad keys), region plumbing into validation and the pipeline, and the watermark being dropped on full refresh.tests/test_llama_cloud.py(33 cases): page-token pagination and termination, resume-state saved after yield not before (guards data loss on crash), resume seeding, incremental filter formatting for datetime and date cursors, usage-metrics org-id resolution and failure, bare-array pipelines endpoint, retryable vs non-retryable status classification, credential-validation status mapping, and SourceResponse shape (sort mode, partitioning, primary keys).test_source_categories,test_source_config_generator, 1537 tests).ruff check/formatandhogli ci:preflight --fixclean.audit_source_docsagainst the posthog.com checkout reports no issues for this source.Automatic notifications
Docs update
User-facing doc added in PostHog/posthog.com#18456.
🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Authored with PostHog Code (Claude). Skills invoked:
/implementing-warehouse-sources,/writing-tests,/documenting-warehouse-sources. Decisions worth knowing: choseResumableSourceover the genericrest_source.RESTClientconfig path because the page-token resume, the usage-metrics org-id resolution, and per-endpoint filter formatting were simpler to express directly (klaviyo-style raw requests, yieldinglist[dict]per page, no source-level batching); chosesort_mode="desc"for incremental endpoints because the API has no sort param and ordering is undocumented; reverted an env-dependent regeneration ofgenerated_configs.py(the generator's output for the Stripe config varies with local OAuth env vars) and applied only the LlamaCloud class change, byte-identical to the generator's output for it. The source stays behindunreleasedSource=Truedeliberately since it hasn't synced against a live account yet.