feat: deterministic idempotent workflow engine - #7
Merged
Conversation
…, replay, invariants
backend/app/workflow.py is the M6 differentiator. Three layers, sharp boundaries:
1. Pure rule layer: route(RoutingInputs) -> RoutingDecision is a function with no
I/O. Identical inputs always produce identical outputs.
2. Idempotency key recipe: compute_idempotency_key(extraction_id, schema_name,
routing_version) is a SHA-256 hex digest. ROUTING_VERSION ('v1') lets future
milestones bump the rule set without colliding with old keys.
3. Persistence: apply_routing(session, *, extraction_id, decision) upserts a
workflow_items row keyed by the deterministic idempotency key. Re-running with
the same decision is a no-op. REJECTED -> AUTO_APPROVED is refused with
IllegalTransition; that promotion requires a human event (M7).
4. Replay: replay(session, *, extraction_id) reads the persisted extraction and
recomputes the decision via route() with no DB writes. route_extraction()
ties replay-style routing and idempotent persistence together for callers.
Decision rules (top-down precedence):
1. 'invalid_citation' guardrail flag -> REJECTED ('guardrail_rejected').
2. Any field below confidence_review_threshold -> NEEDS_REVIEW ('low_confidence').
3. Any non-rejecting guardrail flag -> NEEDS_REVIEW ('guardrail_review').
4. Otherwise -> AUTO_APPROVED ('ok').
Invariants checked at decision time (raise AssertionError):
- AUTO_APPROVED with any field below threshold is impossible.
- A rejecting guardrail flag MUST yield REJECTED.
The engine never touches the LLM, never reads chunks, and never writes anywhere
except workflow_items (in apply_routing). Tests in the next commit cover the four
DoD categories: determinism, idempotency, replay, and invariants.
22 tests across the four M6 DoD categories: Determinism (4): - route() is a pure function of inputs (a == b == c for equal inputs). - field_confidence dict insertion order does not change the decision. - compute_idempotency_key depends only on extraction_id, schema_name, version. - The key is independent of confidence and guardrail flags (so the same extraction always upserts to the same workflow_items row). Rule precedence (5): - invalid_citation flag -> REJECTED (guardrail_rejected) - low confidence -> NEEDS_REVIEW (low_confidence) - non-rejecting flag -> NEEDS_REVIEW (guardrail_review) - clean inputs -> AUTO_APPROVED (ok), routing_version captured - invalid_citation beats low_confidence (rejection is terminal) Idempotency (3): - apply_routing inserts a workflow_items row. - 3x apply_routing of the same decision produces ONE row in the DB (verified by SQL count, not just same id). - A re-run with a different decision updates the same row, never duplicates. Replay (3): - replay() returns the same decision route_extraction did against the same data. - replay handles low-confidence extractions correctly. - replay raises KeyError for unknown extraction id. Invariants (3): - _check_invariants raises on AUTO_APPROVED with low confidence. - _check_invariants raises on non-REJECTED with a rejecting flag. - route() never returns AUTO_APPROVED when low confidence (user-visible invariant pinning the rule precedence). Transitions (2): - REJECTED -> AUTO_APPROVED via apply_routing raises IllegalTransition. - REJECTED -> NEEDS_REVIEW via apply_routing is allowed (documents policy). route_extraction integration (2): - Persists idempotently: same extraction id yields a single workflow_items row. - Propagates guardrail_flags through to the decision. Total project tests: 150 (was 128 in M5).
…invariants Covers why determinism is the M6 contract (audit defensibility, safe re-runs, eval reproducibility), the four-row rule precedence table, the SHA-256 key recipe with what is and is not included (and why), apply_routing's transition policy (REJECTED -> AUTO_APPROVED is refused; demotions allowed), the replay protocol and its two uses (audit assertion, rule-version migration), the two invariants enforced at decision time, and the test categories.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c7b460bcb3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
9 tasks
div0rce
added a commit
that referenced
this pull request
May 29, 2026
* docs(progress): record M6 merged (PR #7) and mark M7 in progress * feat(audit): semantic emitters, HITL review router, state-from-replay primitive backend/app/audit.py adds the M7 layer on top of the M1 append-only audit_events table: - AuditAction string constants for the catalogue (extraction.created, workflow.routed, review.approved, review.rejected); strings (not enums) so future actions can land without an enum migration. - emit_extraction_created(session, extraction): records every model-produced Extraction with the full payload + per-field confidence + citations + model_name in the 'after' JSONB. - emit_workflow_routed(session, workflow_item, prior_status): records routing decisions that actually changed state; idempotent re-routes do NOT emit. - emit_review_decision(session, workflow_item, prior_status, decision, actor, note): records human approve/reject decisions with the human's actor and optional note. - replay_workflow_state(session, workflow_item_id): walks audit events oldest-first and reconstructs the final WorkflowStatus from 'after.status' values. M7 keystone. Wiring: - extract.extract_document calls emit_extraction_created after persistence. - workflow.route_extraction captures prior status, calls apply_routing, and emits workflow.routed only when the workflow_item was inserted or transitioned. backend/app/routers/review.py: - GET /review (limit, offset; returns needs_review queue). - POST /review/{id}/approve -> AUTO_APPROVED + review.approved audit event. - POST /review/{id}/reject -> REJECTED + review.rejected audit event. - 409 on transition from any status other than needs_review (the human path is scoped to this transition; M6's IllegalTransition guard remains for re-routes). - main.py includes the review router. * test(audit): emission counts, review router, state-from-replay (M7 keystone) Three groups mapping onto the M7 DoD: Emission counts (3): - route_extraction emits exactly one workflow.routed event (before=None on insert). - An idempotent re-route emits zero additional events (DoD: exactly one event per state-changing decision). - A re-route that flips status emits a second event with before set to the prior status. Review router (6): - GET /review returns needs_review items. - POST /review/{id}/approve transitions to AUTO_APPROVED, persists, and emits exactly one review.approved audit event whose actor and note match the body. - POST /review/{id}/reject transitions to REJECTED and emits review.rejected. - A second approve on an already-decided item is 409. - 404 for unknown ids; 422 for empty actor. State from replay (2 + 1): - A multi-event lifecycle (route insert -> route transition -> reject -> reopen -> approve) emits 5 audit events; replay_workflow_state walks them and reproduces the current workflow_items.status. **M7 keystone.** - replay_workflow_state returns None for a workflow item with no events. - AuditAction catalogue is pinned (extraction.created, workflow.routed, review.approved, review.rejected). Total project tests: 163 (was 150 in M6). * docs(audit): action catalog, emission rules, replay protocol, HITL transitions Documents the four-action catalogue (extraction.created, workflow.routed, review.approved, review.rejected), the before/after JSON shapes, the emission rules (idempotent re-routes do not double-emit, failures do not persist or emit), the HITL transition table (only needs_review can be approved/rejected via this router; supervisor reversals are deferred), the state-from-replay contract and limits, and the wiring diagram. * docs(progress): mark M7 complete on branch with DoD verification and decision log * fix: make audit transitions concurrency-safe
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Milestone
M6 — Deterministic, idempotent workflow engine
Summary
The differentiator. Adds
backend/app/workflow.pywith a pure rule layer(
route()is a function ofRoutingInputsonly — no I/O), a deterministicSHA-256 idempotency-key recipe gated by a
ROUTING_VERSIONconstant, anidempotent
apply_routingupsert that refusesREJECTED → AUTO_APPROVEDpromotion (M7's audit-driven approval is the only legitimate path), a
replay()primitive that recomputes decisions from stored extractions, runtime invariant
checks, and 22 tests covering all four M6 DoD categories. Plus
docs/workflow.mddocumenting the rules, key recipe, transition policy, replayprotocol, and invariants.
Definition of Done
make checkpasses (ruff + ruff-format + mypy strict + pytest)with PR feat: guardrails (PII redaction, confidence gating, refusal) #6 link; 4 new decision-log entries
logging) — rule layer routes
invalid_citation→rejected; low confidence →needs_review. Audit-event emission for routing arrives in M7.M6 DoD verification (from MILESTONES.md)
route()is a pure function of inputs. Tests pinthat identical inputs produce equal
RoutingDecisioninstances across runs;field_confidenceinsertion order is irrelevant; the idempotency-key recipedepends on
extraction_id,schema_name, androuting_versiononly.apply_routingupserts by deterministic key.test_apply_routing_is_idempotent_no_duplicate_rowsruns three consecutivecalls and verifies at the SQL level that exactly one row exists with that
key.
test_apply_routing_updates_status_when_decision_changesverifies astatus change updates the row in place rather than inserting.
replay()reads the persistedextractionsrow andruns
route()against it. Tests cover the auto-approved happy path(
route_extractionfollowed byreplayreturns the same decision andidempotency key), the low-confidence path (
needs_reviewrecovered fromstorage alone), and unknown-id
KeyError._check_invariantsraisesAssertionErroronAUTO_APPROVED + low confidenceand on
non-REJECTED + invalid_citation.route()never returnsauto_approvedwhen any field is low (a parametrized user-visible test pins this).
apply_routingrefusesREJECTED → AUTO_APPROVEDwithIllegalTransition;the policy of allowing
REJECTED → NEEDS_REVIEWdemotion is documented andtested.
Notes / decisions
See PROGRESS.md "Decision log" — four M6 entries:
route(RoutingInputs) → RoutingDecision).(they change between calls; including them would break re-run idempotency).
ROUTING_VERSION = "v1"; bumping is auditable and triggers re-routing.rejected; low confidence →needs_review;non-rejecting flag →
needs_review; otherwise →auto_approved. Rejection isterminal at routing time.
REJECTED → NEEDS_REVIEWdemotion is allowed;REJECTED → AUTO_APPROVEDisrefused with
IllegalTransition.Other deliberate choices:
except
workflow_items(inapply_routing).it composable inside an HTTP handler or a worker job.
route_extraction(session, extraction_id, guardrail_flags=())is a smallconvenience that ties replay-style routing and idempotent persistence for
callers (e.g., the M9 eval harness, a future review-queue worker).
Schema/migration concerns
None. The M1
workflow_itemstable already hasidempotency_key(unique),extraction_id(FK),status(enum),reason, and timestamps — exactly whatM6 needs. Zero migrations in this PR.
Test coverage
backend/tests/test_workflow.py(22 tests):route_extractionintegration (2)Total project tests: 150 (was 128 in M5).
Reminder
Please squash-merge this PR (Golden Rule #2: one milestone = one squash-merge
commit on
main).