fix: a full-size suite could not be ingested at all - #234
Merged
Conversation
Prisma caps an interactive transaction at five seconds. processJob wrote one statement per test inside a single transaction, so any suite past roughly a thousand tests overran the cap. The API had already answered 202, the job then failed all five attempts and went to the dead letter, and nothing surfaced anywhere the user looks — the run simply never appeared. The contract accepts 5000 executions per batch, so the ingest path promised several times what it could actually process. Resolution now happens entirely in memory, then writes go out in batches: one updateMany for identities whose fingerprint matched exactly, createMany for new identities and for stitches, and createMany for the executions. Execution ids are generated up front so retry links resolve without a second pass, and a replay reuses the ids already stored for each ordinal — RCA reports and artifacts reference them. The transaction bound is headroom for a loaded database, not a substitute for the batching. Verified against a live Postgres: 3000 executions failed on the previous code with "Transaction already closed ... timeout was 5000 ms" and now persists, as does 5000, the contract maximum.
This was referenced Aug 4, 2026
AKogut
added a commit
that referenced
this pull request
Aug 4, 2026
Partially addresses #236. ## What was slow Scoring one identity costs five reads (`computeIdentityScore`) plus a transaction that writes the score and any health events. Identities do not depend on one another, yet they ran strictly in sequence: ```ts for (const identityId of affected) { await scoreIdentity(prisma, identityId, ctx) } ``` After #234 the ingest transaction is a handful of batched statements, so this loop is where a large run's wall clock now goes — roughly 25 of the 29 seconds a 5000-test run took. ## The change A bounded worker pool. **No scoring logic changes** — same queries, same transaction per identity, only overlapped. ## Measured Against local Postgres, sequential → concurrent: | executions | before | after | |---|---|---| | 1000 | 4.3s | 2.4s | | 3000 | 14.8s | 8.9s | | 5000 | 28.9s | 17.9s | ## Why the cap is 4 and not higher Raising it pays very little: | concurrency | 5000 executions | |---|---| | 4 | 17.9s | | 8 | 15.6s | | 12 | 14.9s | It flattens because the limit is the **number** of queries — five per identity, 25 000 for a 5000-test run — not their serialisation. Meanwhile each task holds a connection, briefly two, and overrunning Prisma's pool on a small deployment trades latency for `P2024` pool timeouts, which would turn a slow run into a failed one. `FLAKEMETRY_SCORING_CONCURRENCY` is there for anyone who has measured their own pool. That flattening is the argument *for* #236, not against it: batching the per-identity reads is still the actual fix, and I am leaving the issue open. This is a mitigation that costs almost nothing and risks almost nothing. Full suite green: 62/62 tasks, 88 worker tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The defect
Prisma caps an interactive transaction at five seconds.
processJobwrote one statement per test inside a single transaction, so any suite past roughly a thousand tests overran the cap.The failure mode is the bad part. The API has already answered
202 Acceptedwith a receipt id by the time the worker touches the batch. The job then fails all five attempts, lands in the dead letter, and the only trace is a line on the worker's stderr. The run never appears in the dashboard and nothing tells anyone why.The contract accepts
executions.max(5000), so the ingest path advertised several times the size it could actually process.This is squarely in the way of integrating Flakemetry into a real project — a suite of a few thousand tests is the normal case, not the large one.
Reproduction
Against a live Postgres, 3000 executions through
processJob:Because the cliff is time-based rather than count-based, a slower or busier database moves it down — 1000 tests took 5.6s here and would fail on a loaded instance.
The fix
Identity resolution already ran in memory; only the writes were per-row. They are now batched:
updateManyfor identities matched exactly on their primary fingerprint, where last-seen is the only field that can have moved (the fingerprint covers file path, suite, title and params hash, so the old per-row rewrite of those was a no-op)createManyfor new identities, then a re-read by fingerprint so a concurrent run that inserted the same test first wins the id rather than us dangling a foreign keycreateManyfor stitches and for the executions themselvesExecution ids are generated up front, so retry links resolve inline instead of needing a second pass, and a replay reuses whatever id is already stored for each ordinal — RCA reports and artifacts point at those.
Per-row work remains only where it is genuinely per-row and genuinely rare: renames and moves.
The transaction bound raised alongside it is headroom for a loaded database, not the fix.
Verification
apps/worker/src/__tests__/scale.test.tsingests 3000 executions and asserts every row landed. Revertingprocessor.tsto its previous state fails that test with the timeout above, so the guard is not vacuous. A second case replays a batch and asserts execution ids are unchanged.Measured against local Postgres: 3000 executions now persist in ~15s, and 5000 — the contract maximum — in ~29s. Both previously failed outright.
Full suite green: 56/56 tasks.
Still open
Most of that wall-clock is the per-identity scoring loop after the transaction, which is still one round trip per test. It is slow rather than broken, and I would rather it were measured and fixed on its own than folded in here.