Skip to content

fix(dataplane): close idempotency dedup race with transactional advisory lock claim#2747

Merged
mekilis merged 1 commit into
mainfrom
smart/pde-929-events-idempotency-dedup-races-on-near-simultaneous-submits
Jul 20, 2026
Merged

fix(dataplane): close idempotency dedup race with transactional advisory lock claim#2747
mekilis merged 1 commit into
mainfrom
smart/pde-929-events-idempotency-dedup-races-on-near-simultaneous-submits

Conversation

@mekilis

@mekilis mekilis commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • The idempotency duplicate check ran in the async workers as check-then-insert, so two near-simultaneous creates with the same key could both pass the check and both be persisted and delivered as non-duplicates.
  • The authoritative duplicate check now runs inside the event repository's insert transaction under pg_advisory_xact_lock(hashtextextended('events-idempotency:<project>:<key>', 0)); the re-check under the lock sees rows committed by earlier holders and flips IsDuplicateEvent, so exactly one row per key persists as non-duplicate. Lock or lookup errors fail closed and the worker retries.
  • A unique index is not an option because the events table is partitioned and its unique indexes must include the partition key, which (project_id, idempotency_key) cannot satisfy.

Design notes

  • The fanout service's novelty predicate (FindFirstEventWithIdempotencyKey, non-duplicate rows only) diverges from the claim's (FindEventsByIdempotencyKey, any row). They only disagree if a key's original non-duplicate row was deleted while duplicate rows survive; the claim errs toward flagging the new event duplicate, which is the safe direction.
  • The advisory lock is held through the endpoint batch inserts. This only serializes writers of the same project + idempotency key; unrelated writers are unaffected.

Test plan

  • go test ./internal/events/ (concurrent same-key creates yield exactly one non-duplicate row; lock/lookup error path fails closed)
  • gofmt, go vet, golangci-lint clean on touched packages

Fixes PDE-929

the duplicate check for idempotency keys ran in the async workers as
check-then-insert, so two creates with the same key inside the race
window could both pass the check and both be persisted and delivered
as non-duplicates.

move the authoritative duplicate check into the event repository's
CreateEvent transaction, guarded by a transaction-scoped postgres
advisory lock (pg_advisory_xact_lock) keyed on project id +
idempotency key. the lock serializes same-key writers across all
instances and the re-check under the lock sees rows committed by
earlier holders, so exactly one row per key is persisted with
is_duplicate_event=false. an advisory lock is used instead of a
unique index because the events table can be partitioned by
(project_id, created_at), which a unique index on
(project_id, idempotency_key) cannot span.

the workers' existing pre-checks are kept as a cheap short-circuit
for sequential duplicates; events already flagged duplicate skip the
claim. failure policy: fail closed, a lock or lookup error aborts the
create and the worker retries.
@linear

linear Bot commented Jul 20, 2026

Copy link
Copy Markdown

PDE-929

@cursor

cursor Bot commented Jul 20, 2026

Copy link
Copy Markdown

Current version of PR was reviewed by /review-bugbot on Jul 20, 15:50 GMT+1. It flagged 0 findings.

Bugbot on commit 7fc0057 is skipped.

@mekilis
mekilis enabled auto-merge July 20, 2026 15:48
@mekilis
mekilis merged commit 74c063a into main Jul 20, 2026
32 checks passed
@mekilis
mekilis deleted the smart/pde-929-events-idempotency-dedup-races-on-near-simultaneous-submits branch July 20, 2026 15:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant