feat(data-warehouse): implement campfire import source - #71469
Conversation
|
Hey @Gilbert09! 👋 It looks like your git author email on this PR isn't your
You can fix it for this repo with: git config user.email "you@posthog.com"Or set it globally with |
|
Companion posthog.com doc PR: PostHog/posthog.com#18585 |
🤖 CI report
|
| File | Patch | Uncovered changed lines |
|---|---|---|
products/warehouse_sources/backend/temporal/data_imports/sources/campfire/campfire.py |
97.8% | 165–166 |
🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 29578735469 -n patch-coverage), or the coverage-data block at the end of this comment.
Per-product line coverage (touched products)
| Product | Coverage | Lines |
|---|---|---|
demo |
███████████░░░░░░░░░ 56.2% |
1,497 / 2,663 |
tasks |
█████████████░░░░░░░ 67.2% |
25,898 / 38,558 |
signals |
████████████████░░░░ 79.1% |
19,074 / 24,116 |
data_modeling |
████████████████░░░░ 80.0% |
4,834 / 6,045 |
cdp |
████████████████░░░░ 80.7% |
3,118 / 3,864 |
notebooks |
█████████████████░░░ 84.8% |
6,903 / 8,141 |
cohorts |
█████████████████░░░ 86.2% |
4,065 / 4,717 |
agent_platform |
█████████████████░░░ 86.4% |
3,807 / 4,405 |
actions |
█████████████████░░░ 86.6% |
717 / 828 |
product_tours |
█████████████████░░░ 87.5% |
1,266 / 1,447 |
exports |
██████████████████░░ 88.4% |
6,904 / 7,813 |
conversations |
██████████████████░░ 89.0% |
16,183 / 18,186 |
mcp_analytics |
██████████████████░░ 89.2% |
2,514 / 2,819 |
engineering_analytics |
██████████████████░░ 89.3% |
5,407 / 6,053 |
dashboards |
██████████████████░░ 89.4% |
5,847 / 6,540 |
error_tracking |
██████████████████░░ 89.6% |
9,731 / 10,862 |
alerts |
██████████████████░░ 90.0% |
3,655 / 4,063 |
streamlit_apps |
██████████████████░░ 90.4% |
2,499 / 2,764 |
stamphog |
██████████████████░░ 90.4% |
3,504 / 3,875 |
slack_app |
██████████████████░░ 90.6% |
9,018 / 9,955 |
marketing_analytics |
██████████████████░░ 90.8% |
11,514 / 12,684 |
product_analytics |
██████████████████░░ 91.3% |
5,757 / 6,304 |
data_warehouse |
██████████████████░░ 92.3% |
18,345 / 19,875 |
web_analytics |
███████████████████░ 92.7% |
13,535 / 14,597 |
workflows |
███████████████████░ 92.8% |
5,482 / 5,909 |
ai_observability |
███████████████████░ 92.8% |
14,916 / 16,077 |
surveys |
███████████████████░ 92.9% |
5,689 / 6,122 |
posthog_ai |
███████████████████░ 93.2% |
1,322 / 1,418 |
approvals |
███████████████████░ 93.3% |
3,395 / 3,640 |
reminders |
███████████████████░ 93.4% |
468 / 501 |
early_access_features |
███████████████████░ 93.8% |
848 / 904 |
endpoints |
███████████████████░ 94.1% |
8,606 / 9,143 |
revenue_analytics |
███████████████████░ 94.5% |
3,598 / 3,809 |
skills |
███████████████████░ 94.5% |
2,881 / 3,049 |
review_hog |
███████████████████░ 94.6% |
6,789 / 7,174 |
logs |
███████████████████░ 95.3% |
9,572 / 10,044 |
experiments |
███████████████████░ 95.7% |
24,417 / 25,527 |
replay_vision |
███████████████████░ 95.8% |
13,776 / 14,383 |
annotations |
███████████████████░ 96.2% |
732 / 761 |
feature_flags |
███████████████████░ 96.3% |
16,172 / 16,794 |
user_interviews |
███████████████████░ 96.4% |
2,242 / 2,325 |
warehouse_sources |
███████████████████░ 96.4% |
272,434 / 282,489 |
data_catalog |
███████████████████░ 97.2% |
2,117 / 2,179 |
customer_analytics |
███████████████████░ 97.2% |
7,480 / 7,698 |
pulse |
████████████████████ 98.4% |
2,012 / 2,044 |
Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.
PR overviewAll previously flagged issues have been addressed. No open security concerns remain on this pull request. Security reviewNo open security issues remain on this pull request. Fixed/addressed: 1 · PR risk: 0/10 |
Fills in the scaffolded Campfire (meetcampfire.com) source with 15 row-level tables across the GL, AP/AR, cash management, revenue recognition, and company objects, with incremental sync on last_modified_at where the API exposes a server-side filter. Ships released with releaseStatus=ALPHA. Generated-By: PostHog Code Task-Id: 136f2535-58fa-47bf-b1ea-ebf2dc2bfe5c
Subclass ResumableSourceManager[CampfireResumeConfig] in the test fake so get_rows type-checks. Generated-By: PostHog Code Task-Id: c94455dd-8a21-4eed-ad34-646c4b2acd30
…e capture Pass capture=False on the data-fetching session so financial amounts, invoice/transaction identifiers, and free-form business fields the name-based scrubbers can't recognise are kept out of HTTP sample storage. Requests stay metered and logged. Generated-By: PostHog Code Task-Id: c94455dd-8a21-4eed-ad34-646c4b2acd30
a12827d to
bd766ee
Compare
|
Thanks for the flag. This finding is on 🦉 via talyn.dev |
Generated-By: PostHog Code Task-Id: c428af32-be76-44d0-adcc-e3afe785a609
Problem
The Campfire (meetcampfire.com) warehouse source was scaffolded in #70994 but had no sync logic, so the connector was hidden behind
unreleasedSource=Trueand users couldn't pull their Campfire accounting data into PostHog.Changes
Implements the Campfire source end-to-end and ships it visible with
releaseStatus=ALPHA:chart_transactions,journal_entries,chart_of_accounts), AP (bills,bill_payments,debit_memos), AR (invoices,invoice_payments,credit_memos), cash management (bank_accounts,bank_transactions), revenue recognition (contracts,revenue_transactions), and company objects (vendors,departments). Period-summary financial-statement endpoints are intentionally skipped in favor of the row-level lists.ResumableSourcewith the standardsource.py/settings.py/campfire.pysplit. Pagination follows the API's DRF-stylenextlinks (opting into cursor pagination where Campfire recommends it), and the resume state is the lastnextURL, saved after each yielded batch.nextlinks are validated to stay onapi.meetcampfire.comso the token header can't be replayed off-host.last_modified_atonly for the nine endpoints where the API documents a server-sidelast_modified_at__gtefilter, verified against the OpenAPI spec embedded in Campfire's docs. The payment sync endpoints document(last_modified_at, id)ascending order so they checkpoint the watermark per batch (sort_mode="asc"); every other incremental endpoint has undocumented response order, so they persist the watermark only at successful completion (sort_mode="desc"). Endpoints without a server-side filter ship full refresh only.all_time=truethere to avoid silently truncating the initial sync.make_tracked_session()with tenacity backoff on 429/5xx, and 401/403 map to non-retryable errors with actionable messages.canonical_descriptions.pyfor all 15 tables pluslists_tables_without_credentials=True, so the public docs render the table catalog.Note
Campfire's live API was probed unauthenticated (auth scheme and error shapes verified: 401
{"detail":"Invalid token."}), but endpoint behavior beyond that couldn't be exercised without an account, so pagination/filter behavior is implemented strictly from the OpenAPI spec with conservative defaults, and the source ships as alpha. Campfire also exposes manageable webhooks, but the delivery payload shapes aren't publicly documented, so webhook support is left as a follow-up rather than guessed at.The user-facing doc (
contents/docs/cdp/sources/campfire.md) is written and will land in the posthog.com repo (companion PR linked in a comment);audit_source_docsreports no campfire issues against it.How did you test this code?
tests/test_campfire.py(transport): pagination followsnextlinks and terminates, resume starts from the saved URL and state is saved only after a yield (guards skipped-batch data loss),all_time/cursor/incremental-filter URL construction (guards silent six-month truncation and watermark leakage into full refresh), off-hostnextlink rejection (guards token leakage), retry/non-retry status mapping, and per-endpointSourceResponsesort/partition behavior.tests/test_campfire_source.py(source class): schema catalog and incremental flags match the documented server-side filter availability, credential validation plumbing incl. per-schema probes, resumable manager binding, and the released-as-alpha config (guards regression tounreleasedSource).ruff check/format, andhogli ci:preflight --fix.Automatic notifications
Docs update
Doc written for posthog.com at
contents/docs/cdp/sources/campfire.md(companion PR in the posthog.com repo).🤖 Agent context
Autonomy: Fully autonomous
Implemented with Claude Code following the
implementing-warehouse-sources,writing-tests, anddocumenting-warehouse-sourcesskills. Endpoint selection and pagination/filter semantics were researched from Campfire's public docs (docs.campfire.ai, including the embedded OpenAPI spec) and the live API's unauthenticated behavior. Key decisions: preferred row-level list endpoints over period-summary statements, enabled incremental only where a server-side timestamp filter is documented, chose per-endpointsort_modebased on documented ordering guarantees (conservative end-of-sync watermarking where order is undocumented), and deferred webhook support because payload shapes aren't publicly documented.Created with PostHog Code