Skip to content

feat(data-warehouse): implement campfire import source - #71469

Merged
talyn-app[bot] merged 4 commits into
masterfrom
posthog-code/campfire-source
Jul 17, 2026
Merged

feat(data-warehouse): implement campfire import source#71469
talyn-app[bot] merged 4 commits into
masterfrom
posthog-code/campfire-source

Conversation

@Gilbert09

@Gilbert09 Gilbert09 commented Jul 16, 2026

Copy link
Copy Markdown
Member

Problem

The Campfire (meetcampfire.com) warehouse source was scaffolded in #70994 but had no sync logic, so the connector was hidden behind unreleasedSource=True and users couldn't pull their Campfire accounting data into PostHog.

Changes

Implements the Campfire source end-to-end and ships it visible with releaseStatus=ALPHA:

  • 15 row-level tables across the GL (chart_transactions, journal_entries, chart_of_accounts), AP (bills, bill_payments, debit_memos), AR (invoices, invoice_payments, credit_memos), cash management (bank_accounts, bank_transactions), revenue recognition (contracts, revenue_transactions), and company objects (vendors, departments). Period-summary financial-statement endpoints are intentionally skipped in favor of the row-level lists.
  • ResumableSource with the standard source.py / settings.py / campfire.py split. Pagination follows the API's DRF-style next links (opting into cursor pagination where Campfire recommends it), and the resume state is the last next URL, saved after each yielded batch. next links are validated to stay on api.meetcampfire.com so the token header can't be replayed off-host.
  • Incremental sync on last_modified_at only for the nine endpoints where the API documents a server-side last_modified_at__gte filter, verified against the OpenAPI spec embedded in Campfire's docs. The payment sync endpoints document (last_modified_at, id) ascending order so they checkpoint the watermark per batch (sort_mode="asc"); every other incremental endpoint has undocumented response order, so they persist the watermark only at successful completion (sort_mode="desc"). Endpoints without a server-side filter ship full refresh only.
  • The GL endpoints default to roughly a six-month window server-side, so the source passes all_time=true there to avoid silently truncating the initial sync.
  • All HTTP goes through make_tracked_session() with tenacity backoff on 429/5xx, and 401/403 map to non-retryable errors with actionable messages.
  • canonical_descriptions.py for all 15 tables plus lists_tables_without_credentials=True, so the public docs render the table catalog.
  • SOURCES.md: moved campfire into the Implemented table.

Note

Campfire's live API was probed unauthenticated (auth scheme and error shapes verified: 401 {"detail":"Invalid token."}), but endpoint behavior beyond that couldn't be exercised without an account, so pagination/filter behavior is implemented strictly from the OpenAPI spec with conservative defaults, and the source ships as alpha. Campfire also exposes manageable webhooks, but the delivery payload shapes aren't publicly documented, so webhook support is left as a follow-up rather than guessed at.

The user-facing doc (contents/docs/cdp/sources/campfire.md) is written and will land in the posthog.com repo (companion PR linked in a comment); audit_source_docs reports no campfire issues against it.

How did you test this code?

  • Added tests/test_campfire.py (transport): pagination follows next links and terminates, resume starts from the saved URL and state is saved only after a yield (guards skipped-batch data loss), all_time/cursor/incremental-filter URL construction (guards silent six-month truncation and watermark leakage into full refresh), off-host next link rejection (guards token leakage), retry/non-retry status mapping, and per-endpoint SourceResponse sort/partition behavior.
  • Added tests/test_campfire_source.py (source class): schema catalog and incremental flags match the documented server-side filter availability, credential validation plumbing incl. per-schema probes, resumable manager binding, and the released-as-alpha config (guards regression to unreleasedSource).
  • Ran the two new modules (54 passed) plus the registry-wide source test suite (2655 passed), ruff check/format, and hogli ci:preflight --fix.

Automatic notifications

  • Publish to changelog?
  • Alert Sales and Marketing teams?

Docs update

Doc written for posthog.com at contents/docs/cdp/sources/campfire.md (companion PR in the posthog.com repo).

🤖 Agent context

Autonomy: Fully autonomous

Implemented with Claude Code following the implementing-warehouse-sources, writing-tests, and documenting-warehouse-sources skills. Endpoint selection and pagination/filter semantics were researched from Campfire's public docs (docs.campfire.ai, including the embedded OpenAPI spec) and the live API's unauthenticated behavior. Key decisions: preferred row-level list endpoints over period-summary statements, enabled incremental only where a server-side timestamp filter is documented, chose per-endpoint sort_mode based on documented ordering guarantees (conservative end-of-sync watermarking where order is undocumented), and deferred webhook support because payload shapes aren't publicly documented.


Created with PostHog Code

@github-actions

Copy link
Copy Markdown
Contributor

Hey @Gilbert09! 👋

It looks like your git author email on this PR isn't your @posthog.com address (owerstom@gmail.com). Since you're on the PostHog team, it's worth pointing your local git author email at your @posthog.com address. Why it matters:

  • Consistent work identity in git history — internal tooling that attributes commits to team members keys off your @posthog.com address.
  • Keeps team contributions easy to tell apart from external community ones when scanning history.

You can fix it for this repo with:

git config user.email "you@posthog.com"

Or set it globally with git config --global user.email "you@posthog.com". No need to redo this PR — just a nudge for next time. 🙂

Copy link
Copy Markdown
Member Author

Companion posthog.com doc PR: PostHog/posthog.com#18585

@pr-assigner-resolver-posthog
pr-assigner-resolver-posthog Bot requested a review from a team July 16, 2026 10:01
@trunk-io

trunk-io Bot commented Jul 16, 2026

Copy link
Copy Markdown

Static BadgeStatic BadgeStatic BadgeStatic Badge

View Full Report ↗︎Docs

@github-actions

github-actions Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

🤖 CI report

⚠️ Backend coverage — 99.0% of changed backend lines covered — 2 uncovered

🧪 Backend test coverage

Patch coverage — changed backend lines (products + core): ████████████████████ 99.0% (386 / 388)

File Patch Uncovered changed lines
products/warehouse_sources/backend/temporal/data_imports/sources/campfire/campfire.py 97.8% 165–166

🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 29578735469 -n patch-coverage), or the coverage-data block at the end of this comment.

Per-product line coverage (touched products)
Product Coverage Lines
demo ███████████░░░░░░░░░ 56.2% 1,497 / 2,663
tasks █████████████░░░░░░░ 67.2% 25,898 / 38,558
signals ████████████████░░░░ 79.1% 19,074 / 24,116
data_modeling ████████████████░░░░ 80.0% 4,834 / 6,045
cdp ████████████████░░░░ 80.7% 3,118 / 3,864
notebooks █████████████████░░░ 84.8% 6,903 / 8,141
cohorts █████████████████░░░ 86.2% 4,065 / 4,717
agent_platform █████████████████░░░ 86.4% 3,807 / 4,405
actions █████████████████░░░ 86.6% 717 / 828
product_tours █████████████████░░░ 87.5% 1,266 / 1,447
exports ██████████████████░░ 88.4% 6,904 / 7,813
conversations ██████████████████░░ 89.0% 16,183 / 18,186
mcp_analytics ██████████████████░░ 89.2% 2,514 / 2,819
engineering_analytics ██████████████████░░ 89.3% 5,407 / 6,053
dashboards ██████████████████░░ 89.4% 5,847 / 6,540
error_tracking ██████████████████░░ 89.6% 9,731 / 10,862
alerts ██████████████████░░ 90.0% 3,655 / 4,063
streamlit_apps ██████████████████░░ 90.4% 2,499 / 2,764
stamphog ██████████████████░░ 90.4% 3,504 / 3,875
slack_app ██████████████████░░ 90.6% 9,018 / 9,955
marketing_analytics ██████████████████░░ 90.8% 11,514 / 12,684
product_analytics ██████████████████░░ 91.3% 5,757 / 6,304
data_warehouse ██████████████████░░ 92.3% 18,345 / 19,875
web_analytics ███████████████████░ 92.7% 13,535 / 14,597
workflows ███████████████████░ 92.8% 5,482 / 5,909
ai_observability ███████████████████░ 92.8% 14,916 / 16,077
surveys ███████████████████░ 92.9% 5,689 / 6,122
posthog_ai ███████████████████░ 93.2% 1,322 / 1,418
approvals ███████████████████░ 93.3% 3,395 / 3,640
reminders ███████████████████░ 93.4% 468 / 501
early_access_features ███████████████████░ 93.8% 848 / 904
endpoints ███████████████████░ 94.1% 8,606 / 9,143
revenue_analytics ███████████████████░ 94.5% 3,598 / 3,809
skills ███████████████████░ 94.5% 2,881 / 3,049
review_hog ███████████████████░ 94.6% 6,789 / 7,174
logs ███████████████████░ 95.3% 9,572 / 10,044
experiments ███████████████████░ 95.7% 24,417 / 25,527
replay_vision ███████████████████░ 95.8% 13,776 / 14,383
annotations ███████████████████░ 96.2% 732 / 761
feature_flags ███████████████████░ 96.3% 16,172 / 16,794
user_interviews ███████████████████░ 96.4% 2,242 / 2,325
warehouse_sources ███████████████████░ 96.4% 272,434 / 282,489
data_catalog ███████████████████░ 97.2% 2,117 / 2,179
customer_analytics ███████████████████░ 97.2% 7,480 / 7,698
pulse ████████████████████ 98.4% 2,012 / 2,044

Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.

@veria-ai

veria-ai Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

@hex-security-app

Copy link
Copy Markdown

Warning

Finding on products/warehouse_sources/backend/temporal/data_imports/sources/zenduty/zenduty.py:213 — could not attach an inline comment (line is not part of the diff), so reporting it here.

🟡 Disable capture of Zenduty sync responses

This session leaves HTTP sampling enabled for every imported Zenduty endpoint. When an operator enables a matching capture rule, the transport serializes the complete response body to the shared fixture bucket; its scrubber only removes a small set of credential key names and PII patterns. Incident/postmortem content, escalation configuration, and other account data therefore become accessible outside the source project's normal warehouse access boundary.

Prompt To Fix With AI
Create Zenduty tracked sessions with `redact_values=(api_key,)` and `capture=False` for data-sync requests (and credential probes if they can run with a capture context). Keep request metrics/logging enabled, but prevent raw Zenduty response bodies from being written to HTTP sample storage. Add a transport-construction test asserting both options.

Severity: medium | Confidence: 92% | React with 👍 if useful or 👎 if not

Fills in the scaffolded Campfire (meetcampfire.com) source with 15 row-level
tables across the GL, AP/AR, cash management, revenue recognition, and company
objects, with incremental sync on last_modified_at where the API exposes a
server-side filter. Ships released with releaseStatus=ALPHA.

Generated-By: PostHog Code
Task-Id: 136f2535-58fa-47bf-b1ea-ebf2dc2bfe5c
Subclass ResumableSourceManager[CampfireResumeConfig] in the test fake so get_rows type-checks.

Generated-By: PostHog Code
Task-Id: c94455dd-8a21-4eed-ad34-646c4b2acd30
…e capture

Pass capture=False on the data-fetching session so financial amounts, invoice/transaction identifiers, and free-form business fields the name-based scrubbers can't recognise are kept out of HTTP sample storage. Requests stay metered and logged.

Generated-By: PostHog Code
Task-Id: c94455dd-8a21-4eed-ad34-646c4b2acd30
@Gilbert09
Gilbert09 force-pushed the posthog-code/campfire-source branch from a12827d to bd766ee Compare July 17, 2026 11:37

Copy link
Copy Markdown
Member Author

Thanks for the flag. This finding is on zenduty/zenduty.py, which is outside this PR's diff — this PR only implements the campfire source, and campfire's sync responses are already excluded from sample capture (capture=False with redact_values). The zenduty hardening is a separate concern and should be handled in its own change rather than here.

🦉 via talyn.dev

Generated-By: PostHog Code
Task-Id: c428af32-be76-44d0-adcc-e3afe785a609
@talyn-app
talyn-app Bot enabled auto-merge (squash) July 17, 2026 12:09
@talyn-app
talyn-app Bot merged commit 9c53bc3 into master Jul 17, 2026
234 checks passed
@talyn-app
talyn-app Bot deleted the posthog-code/campfire-source branch July 17, 2026 12:32
@deployment-status-posthog

deployment-status-posthog Bot commented Jul 17, 2026

Copy link
Copy Markdown

Deploy status

Environment Status Deployed At Workflow
dev ✅ Deployed 2026-07-17 13:13 UTC Run
prod-us ✅ Deployed 2026-07-17 13:29 UTC Run
prod-eu ✅ Deployed 2026-07-17 13:33 UTC Run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants