docs: durable task store + delivery idempotency design (#2)#43
Merged
Conversation
Proposes embedded SQLite (rusqlite, WAL) behind a new crates/store crate: persist-then-ack webhook handling keyed by X-GitHub-Delivery, a tasks table as the queue (atomic claims, startup recovery, supersession tombstones), task_attempts audit records, and Cave list continuity across restarts. Phased into three separately-mergeable PRs. Signed-off-by: Val Alexander <bunsthedev@gmail.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a proposed design document for implementing issue #2’s durable task store and GitHub delivery idempotency in coven-github, intended as a design-for-review precursor to phased implementation PRs.
Changes:
- Introduces a SQLite (
rusqlite) durable store design, including schema, config surface, and migration strategy. - Specifies end-to-end webhook idempotency flow keyed by
X-GitHub-Deliveryand a persist-then-ack contract. - Describes worker-side durable claiming, restart recovery, and supersession behavior moving from memory into the store.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+77
to
+79
| # Directory for durable adapter state (SQLite database + WAL files). | ||
| # The doctor command checks it is creatable/writable. | ||
| path = "data/coven-github.db" |
| installation_id INTEGER, | ||
| repo TEXT, -- owner/name when parseable | ||
| payload_hash TEXT NOT NULL, -- sha256 of raw body | ||
| routing TEXT NOT NULL, -- 'task:<id>' | 'ignored:<reason>' |
Comment on lines
+212
to
+221
| | Component | Change | | ||
| |---|---| | ||
| | new `crates/store` | `Store`: open/migrate, delivery insert-or-dup, task enqueue+tombstone, claim, terminal updates, attempt records, Cave list query, startup recovery | | ||
| | `crates/webhook/routes.rs` | Delivery-id gate; persist-then-ack; drop `task_tx`; `list_tasks` reads the store | | ||
| | `crates/worker/lib.rs` | Claim loop replaces `mpsc` recv; terminal states write to the store; supersession check reads `state='superseded'` instead of the in-memory registry | | ||
| | `crates/github/tasks.rs` | In-memory `TaskStore` retired; `TaskListItem`/`TaskListStatus` stay as the API projection | | ||
| | `crates/config` | `[storage] path` + doctor checks | | ||
| | `crates/server/main.rs` | Open store, run recovery, wire Notify | | ||
| | `compose.yaml` | Named volume for `/data` | | ||
| | `README.md` | Durable queue row: planned → implemented (only once true) | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Design-for-review ahead of implementing #2, per maintainer direction (SQLite via rusqlite, design doc first).
Key decisions proposed in
docs/durable-task-store.md:rusqlite(WAL, single-writer) in a newcrates/storecrate — zero-infra self-hosting, compose stays single-container, Postgres graduation path behind the same type if hosted scale demands itX-GitHub-Delivery(400 without) → insert delivery withON CONFLICT DO NOTHING(duplicate → 200 + no-op) → insert taskqueued→ notify → 200. Store down = 500, so GitHub retries instead of losing worktaskstable IS the queue — the mpsc channel and its silenttry_senddrop path are removed; workers claim atomically viaUPDATE … RETURNINGrunningrows re-queue at boot (attempt-capped so a crash-looping task landsfailed); the marker-backed status comment (Add marker-backed comments and maintainer command protocol #13) keeps the user surface deduplicated across restartssupersede_key), replacing the in-memory registry; the Resolve Check Run head SHA and target ref correctly #8 mid-flight staleness gate is unchangedTest plan maps 1:1 to #2's acceptance criteria (duplicate delivery, queue-full inversion, restart recovery, ignored routing, Cave continuity).
Refs #2 (implementation PRs will close it).