feat(growth): real-time signup enrichment workflow - #70118
Conversation
Provider interface + Harmonic implementation, transform to a typed field registry, and a writer to OrganizationEnrichment + the enrichment_* group-property projection. No caller yet. Generated-By: PostHog Code Task-Id: aad34d9e-9758-4723-af4e-e07fe8b992cb
Cherry-picked from #69411 (matt/enrichment-snapshot-event) so this branch builds and tests standalone. These files (snapshot.py, EnrichmentSignupSnapshot model + migration, test_snapshot.py, idor coverage) belong to #69411 and collapse out on rebase once it merges. Generated-By: PostHog Code Task-Id: aad34d9e-9758-4723-af4e-e07fe8b992cb
Fire-and-forget Temporal workflow + activity started after signup, behind a kill switch (US-only v0, requires a Harmonic key). Wires the write-once at-signup snapshot and emits a launch fill-rate/failure signal. Registered on the general-purpose task queue. Generated-By: PostHog Code Task-Id: aad34d9e-9758-4723-af4e-e07fe8b992cb
…mail - headcount/country project to the existing icp_employees/icp_country group properties (same source as the incumbent waterfall, verified by a 91-domain match-rate calibration), other fields stay on the parallel enrichment_* keys - transform maps provider country names to ISO alpha-2 to match icp_country's format, via a table generated from the frontend's canonical country map - record_signup_work_email persists the work-vs-personal email signal from the request path (GenericEmails-derived, no new lists) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e/enrichment-realtime-seam
…igger Personal-domain signups skip the provider lookup but still get the work-vs-personal signal on their enrichment record, savepoint-guarded so a write failure can never surface to signup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Reviews (1): Last reviewed commit: "feat(growth): record work_email for ever..." | Re-trigger Greptile |
User.distinct_id is nullable, so the signup call site passes str | None. Widen the trigger signature and guard dispatch on a present distinct_id while still recording work_email for every signup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The activity emitted signup_enrichment_completed {success: false} on every
failed attempt, so a transient error a later retry recovers from still
counted against the launch failure-rate signal. Gate the failure event on
the final attempt via a single-sourced MAX_ENRICH_ATTEMPTS; capture_exception
still fires on every attempt for error tracking.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Run _dispatch on a daemon thread so building the Temporal client never adds latency to the signup response (on_commit fires inline once the signup transaction has committed). - Read HARMONIC_API_KEY via getattr: it only exists in ee/settings, so a non-EE deploy would AttributeError and 500 signup once the flag is on. - Catch WorkflowAlreadyStartedError explicitly (not an RPCError subclass) so a re-signup race logs instead of landing in capture_exception as noise. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
| "signup-enrichment", | ||
| inputs, | ||
| id=f"signup-enrichment-{inputs.organization_id}", | ||
| task_queue=settings.GENERAL_PURPOSE_TASK_QUEUE, |
There was a problem hiding this comment.
Medium: Untrusted signup work uses a shared task queue
A successful unauthenticated signup can enqueue an enrichment activity that holds a worker slot for up to 90 seconds, while this queue also processes unrelated operational workflows. An attacker can create accounts with arbitrary unverified work-domain emails from rotating IPs to consume shared worker capacity; route this workflow to a dedicated queue with bounded concurrency or add a separate dispatch quota before enabling it.
There was a problem hiding this comment.
This is probably fine for now, but we need a TODO to eventually move away from this and into our own specific enrichment queue
There was a problem hiding this comment.
Routed to a configurable queue in 94f98f9: dispatch reads SIGNUP_ENRICHMENT_TASK_QUEUE and the worker registers these workflows under a dedicated landing-zone queue. Default stays on the general-purpose fleet so dispatch always has a live worker; at enable time the env flips this to a dedicated, separately-scalable queue once a charts worker consumes it.
There was a problem hiding this comment.
Turned this into a config seam in 94f98f9: SIGNUP_ENRICHMENT_TASK_QUEUE env flips dispatch onto a dedicated queue once a worker exists, defaulting to general-purpose until then.
PR overviewThis pull request adds a Temporal-based real-time signup enrichment workflow for growth, triggering organization enrichment around signup and performing a delayed recheck. The changed code wires signup events into the enrichment workflow and persists or emits enrichment-related results. There are still two open security concerns. The main risk is that unauthenticated signups can dispatch enrichment work onto a shared Temporal task queue, allowing an attacker to consume worker capacity used by other operational workflows. A second issue remains around delayed rechecks writing enrichment data or analytics after an organization has been deleted. One issue has already been addressed, but the remaining items should be resolved before rollout. Open issues (2)
Fixed/addressed: 1 · PR risk: 6/10 |
🤖 CI reportℹ️ ClickHouse migration SQL — 1 migration(s)ClickHouse migration SQL per cloud environment
|
The signup work_email write and the fire-and-forget provider write both do a read/merge/save on OrganizationEnrichment.data. Concurrent writers on the same org read the same snapshot and the later save clobbers the other keys. Lock the row with select_for_update so the two serialize; extract the shared merge helper. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ment-core-activity
enrich_company_by_domain swallows every exception and returns None, making a Harmonic outage indistinguishable from a clean miss: the enrichment workflow's retries never fire and the launch failure signal cannot emit. Adds an additive strict lookup that returns None only for a genuine companyFound=false and re-raises operational errors; the growth provider uses it, the Salesforce job keeps the lenient path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Regenerating from the frontend's canonical country map is now a script run instead of a hand process, so the copy can't silently drift. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tion Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…onse Add total_raised, last_round_size, last_round_date (ability-to-pay signals) and is_ai_native to the enrichment field registry. All four are already returned by HARMONIC_COMPANY_ENRICHMENT_QUERY. They project to the parallel enrichment_* group-property namespace. is_ai_native stays unset when tagsV2 is absent — no tag data is not evidence a company is not AI-native. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…sn't indexed yet When the first enrich attempt does not match, sleep 4h on a durable timer and run the enrich activity once more with is_recheck=True. Harmonic's enrich mutation seeds async enrichment for unknown companies, so a later lookup often succeeds. The recheck writes the live stores like a first attempt but does not emit the write-once at-signup snapshot or the launch signal (that stays first-attempt-only so the launch alert keeps reading unchanged semantics). It emits a separate signup_enrichment_recheck event instead, and its failures never trip the launch failure signal. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Passthrough of funding.investors (the same list _is_yc_funded scans) as an enrichment_investors group property. Company entries carry name, Person (angel) entries carry fullName; keep both. Capped at 25 names to keep the group property bounded, and unset when the provider returns no investors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Two additions since the approval, both from review threads: 94f98f9 makes the task queue configurable (veria's shared-queue finding; dispatch defaults to general-purpose, one env var flips dispatch + worker registration to a dedicated queue), and 1e885fa adds a one-shot delayed recheck when Harmonic doesn't match at signup (from Mine's review on the core PR): the workflow sleeps 4h on a durable timer and looks once more, since the enrich mutation seeds Harmonic's async enrichment for unknown companies. The launch signal still reads only the first attempt; the recheck emits its own signup_enrichment_recheck event. Tests cover both. |
| return SignupEnrichmentInputs(**json.loads(inputs[0])) | ||
|
|
||
| @workflow.run | ||
| async def run(self, inputs: SignupEnrichmentInputs) -> dict[str, typing.Any]: |
There was a problem hiding this comment.
Medium: Recheck writes data after organization deletion
An organization owner can delete the organization during this four-hour delay, but the recheck still calls enrich_organization and emits an event without revalidating that the organization exists. Because OrganizationEnrichment.organization has db_constraint=False, a successful lookup can also create an orphaned enrichment row after deletion. Cancel this workflow as part of organization deletion, or verify that the organization exists and is not pending deletion immediately before each persistence and analytics write.
There was a problem hiding this comment.
Fixed in ca80350: the recheck verifies the org still exists before doing anything; a deleted org gets a logged skip, no writes, no events.
Add OrganizationEnrichmentFetch, an append-only archive storing each provider
response verbatim (before transform), one row per fetch. Signup and recheck are
separate rows on purpose — provider responses are time-varying, so each fetch is a
distinct observation, including a not-found (stored as {companyFound: false}).
The provider now returns a ProviderLookup (fields + raw payload); enrich_organization
archives every fetch before the live-store write and takes an is_recheck label. A
raw-archive failure is swallowed and captured so it can never break enrichment.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Thread the activity is_recheck flag through to enrich_organization so the raw-payload archive rows are labeled signup vs recheck. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…age exclusions Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ivity' into posthog-code/enrichment-realtime-seam # Conflicts: # .github/scripts/check-idor-model-coverage.py # products/growth/backend/migrations/max_migration.txt # products/growth/backend/models.py
|
Reviews (2): Last reviewed commit: "Merge remote-tracking branch 'origin/pos..." | Re-trigger Greptile |
Add the missing __init__.py so pytest imports these tests under a fully-qualified module path. Without it, test_workflow.py collided with products/batch_exports/.../redshift/test_workflow.py under the default prepend import mode, causing an import-file-mismatch collection error when both landed in the same Django temporal shard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The web fleet no longer needs the Harmonic key mounted for a presence check; a keyless worker fails into the launch alert instead of web silently never dispatching. Follow-up from charts#13143 review (greptile security P1 + langesven). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A slow or unreachable Temporal previously let signup-triggered daemon threads accumulate on web pods; dispatch now runs on a 4-worker pool with a 64-deep backlog cap and drops (logged) beyond it. From veria review on the PR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 4h recheck now verifies the org still exists before enriching; a deleted org gets a logged skip instead of orphan rows and dead group properties. From veria review. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Problem
The enrichment core (PR A) has no caller. This adds the real-time seam: a fire-and-forget Temporal workflow started right after signup, so onboarding can route on firmographic data. It must never block or fail signup.
Changes
posthog/temporal/signup_enrichment/workflow.py—SignupEnrichmentWorkflow+enrich_signup_organization_activity. The activity awaits the PR A core (enrich_organization), builds and emits the write-once snapshot, and captures asignup_enrichment_completedsignal (success + which fields filled). Hard 90s per-attempt timeout, a couple of retries then it gives up quietly.posthog/temporal/signup_enrichment/trigger.py—start_signup_enrichment_workflow, the single guarded entry point. Kill switch, US-only for v0, requires a Harmonic key, dispatches viasync_connectinsidetransaction.on_commit, and never raises. Records thework_emailsignal (via PR A'srecord_signup_work_email, savepoint-guarded) for every signup; personal-email domains skip only the provider lookup, so the work-vs-personal bit is still available to consumers for them.posthog/api/signup.py— one call afterreport_user_signed_up. No request/response or validation change.GENERAL_PURPOSE_TASK_QUEUE(where the other enrichment activities already live).GROWTH_SIGNUP_ENRICHMENT_ENABLED— the kill switch, off by default.Warning
Launch observability is a ship gate. The
signup_enrichment_completedevent (propertysuccess, andfields_filledon success) is the fill-rate/failure signal. An alert on its failure rate, routed to #alerts-growth, must exist beforeGROWTH_SIGNUP_ENRICHMENT_ENABLEDis turned on anywhere. The switch stays off in this PR. I could not create that alert from here (it's runtime config in a growth-owned project with the Slack routing), so it's an explicit pre-enable step, not code.Snapshot wiring decision (resolving #69411's two open questions)
capture(), so retries of this fire-and-forget activity never double-emit.distinct_idthrough the workflow inputs.Timing / safe default
The
work_emailsignal is written synchronously in-request, so it's always present for onboarding's first read. The Harmonic write is async and often lands after that first read, so any consumer must treat enrichment as best-effort and degrade to a safe default when the record is missing or partial.Note
No
ALTERon the organization table. The only new model isEnrichmentSignupSnapshot(aCreateModelextension table), and it comes from #69411, not this PR.How did you test this code?
Automated tests, all run locally (the full signup-enrichment + growth-enrichment suite, 201 passing including the DB-backed cases):
test_trigger.py— dispatches for a US work-email signup and recordswork_email=True; personal-email domains recordwork_email=Falsewithout a provider dispatch; never dispatches (or writes) when the kill switch is off; never dispatches when the Harmonic key is missing or the region isn't US; awork_emailwrite failure is captured and does not block dispatch.test_activity.py— success emits the signal + snapshot built from enriched fields; a provider miss falls back to the deterministic company type; a failure emits a failure signal and re-raises. UsesActivityEnvironmentwith the DB/provider boundaries mocked.ruffandtach checkpass.Automatic notifications
🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Authored by Claude via PostHog Code, driven by @MattBro from a Slack thread. Invoked the
/improving-drf-endpointsskill before touching the signup serializer (confirmed the change is a side effect, not a schema change, so no new annotations were needed).Decisions for review:
trigger.pyso the serializer stays a one-line call and the fire-and-forget contract lives in one place. Modeled onproducts/slack_app/backend/signals.pyandenrich_view_semantics.py.Note
Stacking. This PR is based on PR A and also depends on #69411 (snapshot helper). Because #69411 isn't merged yet, its files (
snapshot.py,EnrichmentSignupSnapshot+ migration0003,test_snapshot.py, idor coverage) are cherry-picked in as a separate commit so this branch builds and tests standalone. Review those in #69411, not here; they collapse out on rebase once it merges.Merge order: #69411 → PR A → PR B (this). The routing-fallback PR (former PR C) was cancelled along with the classifier PR; company type now arrives as enrichment data instead of a maintained list.
FYI @abhischekt.
Created with PostHog from a Slack thread