perf(onboarding): replace sequential update loops with bulk SQL#2647
Merged
perf(onboarding): replace sequential update loops with bulk SQL#2647
Conversation
…g init
The org initialization transaction was firing ~100 sequential queries for a
typical SOC 2 onboarding — two loops inside the transaction were the main
contributors:
1. Per-policy `policy.update` to set `currentVersionId` after creating
PolicyVersion rows (one UPDATE per new policy).
2. Per-control `control.update({ policies: { connect }, tasks: { connect } })`
to link policies/tasks to controls (one UPDATE per control).
On Prisma v6 with the native Rust engine this ran in ~5-10s. The v6→v7
migration swapped the Rust engine for `@prisma/adapter-pg` (node-postgres),
which roughly tripled per-query overhead, pushing the same workload to 30+s
and causing onboarding to trip the 30s transaction timeout (which we bumped
to 60s as a stopgap in 8d1764a).
Fix:
- Policy block: pre-generate Policy and PolicyVersion CUIDs in a single
`$queryRaw` call using `generate_prefixed_cuid()`. Create policies, then
versions, then set `currentVersionId` for all rows in a single bulk
`UPDATE ... FROM (VALUES ...)` statement. Handles the FK cycle
(Policy.currentVersionId ↔ PolicyVersion.policyId) by insert ordering
rather than schema changes. Kills N sequential updates.
- Control block: collect control↔policy and control↔task pairs in memory
during the existing requirement-map pass, then bulk-insert into the
implicit M2M join tables `_ControlToPolicy` and `_ControlToTask` with
`ON CONFLICT ("A","B") DO NOTHING`. Preserves idempotency for re-runs
(e.g. adding a framework to an existing org). Kills M sequential updates.
Result: ~100 queries → ~20, flat regardless of framework size. Expected
wall-clock 2-4s. Public API, return shape, console.warn diagnostics, and
org-scoping are all preserved.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
|
🎉 This PR is included in version 3.30.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_upsertOrgFrameworkStructureCore).@prisma/adapter-pg) roughly tripled per-query overhead, pushing the same workload to 30+s and tripping the transaction timeout (8d1764a67bumped it 30s→60s as a stopgap).What changed
Single file:
apps/app/src/actions/organization/lib/initialize-organization.ts.Policy block — eliminated N sequential
policy.updatecalls:$queryRawviagenerate_prefixed_cuid().createManyPolicy (IDs pre-set,currentVersionIdstill null).createManyPolicyVersion (IDs pre-set,policyIdpre-known).UPDATE "Policy" ... FROM (VALUES ...)to setcurrentVersionIdfor all policies at once.Policy.currentVersionId↔PolicyVersion.policyId) handled by insert ordering — no schema change.Control block — eliminated M sequential
control.update({ connect })calls:(controlId, policyId)and(controlId, taskId)pairs in memory during the existing requirement-map pass.INSERT INTO "_ControlToPolicy" ... ON CONFLICT DO NOTHING/"_ControlToTask"calls.ON CONFLICT DO NOTHINGpreserves the idempotency the oldconnectloop provided for re-runs (e.g. adding a framework to an existing org).What did NOT change
initializeOrganization) and return shape._upsertOrgFrameworkStructureCorehas no external callers (grep-verified).console.warndiagnostics preserved verbatim.if (policyTemplatesForCreation.length > 0)etc.).Test plan
currentVersionIdset, matching PolicyVersion rows,Control.policies+Control.taskspopulated, andRequirementMapentries presentinitializeOrganizationtwice against the same org (adding a framework) — confirm no unique-constraint errors and no duplicate M2M rows (idempotency)🤖 Generated with Claude Code
Summary by cubic
Speeds up org onboarding by replacing sequential update loops with bulk SQL. Query count drops from ~100 to ~20 and typical runtime falls to ~2–4s, with no API changes.
tx.$queryRawusinggenerate_prefixed_cuid(), bulkcreateManyforPolicyandPolicyVersion, then one bulkUPDATE ... FROM (VALUES ...)to setPolicy.currentVersionId(FK cycle handled by insert order).INSERTinto"_ControlToPolicy"and"_ControlToTask"withON CONFLICT DO NOTHING, replacing per-controlupdate({ connect })loops.7.6.0. 60s tx timeout kept for now.Written for commit 0c5d332. Summary will update on new commits.