feat(jobs/idempotency): Idempotency-Key middleware with two-tier store#363
Merged
Conversation
…e — closes #264 Implements the IETF-draft Idempotency-Key pattern (HTTP API operations that MUST NOT be replayed on retry: payments, one-time emails, webhook enqueues). Two-tier store: Redis hot path via Lua-atomic claim, Postgres durable backing for audit + cold-start recovery. - `migrations/000014_idempotency` adds the `idempotency_keys` table (key PK, sha256 request_hash, status enum, result_code/result_body JSONB, expires_at) plus an index on expires_at for the scheduled prune. Both up and down migrations land per repo convention. - `packages/go/jobs/idempotency/`: - `key.go` — header validation (length, control bytes, whitespace), canonical SHA-256 over method+path+body, body-buffering helper that swaps r.Body so the downstream handler still sees the bytes. - `store.go` — `Store` interface, `RedisStore` with the atomic Lua claim script (parses "hash|status|code|body_hex" payload, returns NEW/PENDING/REPLAY/MISMATCH outcomes in one round-trip), `PostgresStore` with INSERT...ON CONFLICT DO NOTHING for atomic claim, terminal-status WHERE-guard on Finish, expires_at-based Prune, and a {_raw_base64, json} wrapper so JSONB normalisation can't corrupt the replay-byte fidelity. `TieredStore` composes both with warm-on-cold-miss semantics. - `middleware.go` — HTTP middleware: method allowlist (default POST/PUT/PATCH/DELETE), 400/413/422/409/503 error envelope, captureWriter that mirrors the response into the store while flushing inline to the client, replay sets `Idempotency-Replayed` header. - Tests: 30+ unit tests with in-memory fakes for Redis and Postgres (Lua semantics + SQL outcomes), plus testcontainers integration tests covering the real Lua script, TTL-driven eviction (64 concurrent goroutines, exactly one ClaimNew), Postgres prune of expired rows, and full middleware end-to-end (200 → replay → mismatch). All pass under -race. Signed-off-by: Mohamed Tayeb Mokni <tayeb.mokni@gmail.com>
tayebmokni
force-pushed
the
feat/264-idempotency
branch
from
f0cee2d to
f23df0f
Compare
4 tasks
tayebmokni
pushed a commit
that referenced
this pull request
May 18, 2026
Introduces packages/go/plugins/abi/jobs, the bridge between a plugin's manifest jobs[] declarations and the host TaskSpec + Asynq dispatch chassis. Mirrors the hooks bridge (abi/hooks, #350/#364) but for queued background jobs. The bridge: - Defines the gn_handle_job ABI (abi.go) with the same packed-i64 return shape as gn_handle_hook so plugin SDKs share most of their host-call plumbing. - Provides a Dispatcher (dispatcher.go) that drives the ABI on a plugin Module — serialised allocation, name+payload write, invocation, packed-result unpack, typed JobError on failure. - Provides a Bridge (bridge.go) that walks manifest.Jobs and registers one taskspec.TaskSpec per declared job, routed to the "plugin" Asynq queue (per #362), with the asynq Task ID threaded through the envelope as an idempotency key (per #363). - Gates registration on the jobs.enqueue capability via the plugin capabilities Checker (#356/#344). Manifests declaring jobs without the cap are rejected before any TaskSpec installs. Hot-reload contract: Unregister flips the closed flag so the installed TaskSpec.Handler short-circuits with a retryable error; the lifecycle Manager constructs a fresh Bridge against a fresh registry on plugin reload. Tests (22 cases, race-clean): - Pack/unpack round-trip and envelope marshal shapes. - Happy-path invocation, unknown job, trap, OOM, payload too large. - Bridge registers from manifest with correct queue/MaxRetry. - Capability denial rejects registration with no half-wired state. - Trap inside job returns retryable error (no asynq.SkipRetry). - Hot reload: gen1 unregister + gen2 register on a fresh registry. - Duplicate job name surfaces taskspec.ErrAlreadyRegistered. - 100 concurrent task pickups for the same plugin (race detector). Signed-off-by: Mohamed Tayeb Mokni <tayeb.mokni@gmail.com>
tayebmokni
added a commit
that referenced
this pull request
May 18, 2026
## Summary - Introduces `packages/go/plugins/abi/jobs` — the bridge between a plugin's manifest `jobs[]` declarations and the host TaskSpec + Asynq dispatch chassis (#355, #362). Mirrors the hooks bridge (#350, #364) for queued background jobs. - Defines the `gn_handle_job(name_ptr, name_len, payload_ptr, payload_len) -> i64` ABI with the same packed-i64 return shape as `gn_handle_hook`, so plugin SDKs share most of their host-call plumbing. - For each `manifest.Jobs` entry, registers a `taskspec.TaskSpec` routed to the `plugin` queue, with the asynq Task ID threaded through the envelope as an idempotency key (#363). Capability-gated on `jobs.enqueue` via the plugin capabilities Checker (#356, #344) — manifests declaring jobs without the cap are rejected before any TaskSpec installs. ## Test plan - [x] `go vet ./plugins/abi/jobs/...` clean. - [x] `go test -race -count=1 ./plugins/abi/jobs/...` — 22 cases pass: pack/unpack round-trip, envelope marshal shapes, happy-path/unknown/trap/OOM/oversize, manifest registration with correct queue + MaxRetry, capability denial leaves the registry untouched, trap returns a retryable error (no `asynq.SkipRetry`), hot-reload across two fresh registries, duplicate job surfaces `taskspec.ErrAlreadyRegistered`, and 100 concurrent task pickups for the same plugin. - [x] Full `./plugins/... ./jobs/...` test suite stays green. - [x] DCO sign-off present. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Mohamed Tayeb Mokni <tayeb.mokni@gmail.com> Co-authored-by: Mohamed Tayeb Mokni <tayeb.mokni@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #264. Implements the Idempotency-Key pattern (IETF-draft) for HTTP operations that MUST NOT replay on retry — payments, one-time emails, webhook enqueues.
TieredStorewith warm-on-cold-miss.Idempotency-Replayed: trueheader (same key + same hash), 422idempotency_key_reused(different hash), 409idempotency_key_pending(concurrent in-flight), 413 (body too large), 503 (backend out).000014_idempotencyaddsidempotency_keys(key TEXT PK,request_hash BYTEA,statusCHECK,result_code/result_body JSONB,expires_atindexed). Reversible down. Includes a CHECK constraint that physically prevents anin_progressrow from carrying a result and a terminal row from being missing one.PostgresStore.Pruneis the scheduled-cleanup hook; expired rows are also treated as missing on Claim so the middleware self-heals between prune cycles.{_raw_base64, json}JSONB wrapper so Postgres normalisation can't corrupt replay byte fidelity while operators retain JSON-path indexing.Test plan
go vet ./jobs/idempotency/...go test -race -count=1 ./jobs/idempotency/...— 30+ unit tests + testcontainers integration tests against real Postgres + Redis, including a 64-goroutine concurrency test that asserts exactly oneClaimNew.go build ./...andgo vet ./...across all ofpackages/go— clean.Signed-off-by: Mohamed Tayeb Mokni tayeb.mokni@gmail.com