Claimify-loss ledger: no kept span dies silently (#161)

.github/workflows/ci.yml

@@ -166,7 +166,7 @@ jobs:
           trap - EXIT
           test "$(docker compose --env-file .env.example exec -T postgres \
             psql -U rememberstack -d rememberstack -Atc \
-            'SELECT version_num FROM alembic_version')" = 'p7_05_0017'
+            'SELECT version_num FROM alembic_version')" = 'p1_03_0018'
           test "$(docker compose --env-file .env.example exec -T postgres \
             psql -U rememberstack -d rememberstack -Atc \
             'SELECT count(*) FROM deployments')" = '1'

decisions.md

@@ -697,6 +697,16 @@ eval metrics come for free.
 **Context.** The same durable-state discipline the resolution and supersession layers use, applied to
 the extraction transcript. (C8.)
 
+**Amendment (2026-07-27, issue #161).** The transcript also records the Claimify-stage losses that
+were previously silent: `claimify_omitted` (a kept span the model returned no claim for — including
+spans not verbatim-findable in the document, which formerly vanished traceless) and
+`grounding_rejected` (a returned claim a D32 gate rejected, with the gate named in `edit_detail`).
+Accounting rules: every returned claim independently ends accepted or rejected; every keep with no
+range-attributable returned claim gets exactly one omission row; orphan rejections suppress no
+omission. On D56 zero-claim reuse the prior transcript is copied forward — the `no_info` marker is
+fabricated only when the prior transcript is itself empty. Detail:
+`plan/designs/e2_e3_claims_relations_design.md` §3 amendment.
+
 ## D34. E2 Selection is the value filter — there is no pre-extraction value gate
 
 **Decision.** Junk-control lives at the **proposition grain**, in-call: Selection (D31) decides

plan/designs/e2_e3_claims_relations_design.md

@@ -148,17 +148,52 @@ in the bundle. The dropped opinion never reaches grounding.
 
 Two safeguards keep aggressive Selection safe:
 
-- **A decision ledger (D33).** Every Selection drop and every decontextualization edit is written to an
-  append-only, version-stamped `claim_extraction_decisions` table. A better prompt can later re-examine
-  *only the drops*; a rebuild reads stored claims + decisions and never re-calls the model (the LLM
-  rungs are replay-from-storage, like any non-deterministic stage — D7); the per-chunk worker is
-  idempotent on content-hash + extractor version (D12).
+- **A decision ledger (D33).** Every Selection drop, every decontextualization edit, and every
+  Claimify-stage loss is written to an append-only, version-stamped `claim_extraction_decisions`
+  table. A better prompt can later re-examine *only the drops*; a rebuild reads stored claims +
+  decisions and never re-calls the model (the LLM rungs are replay-from-storage, like any
+  non-deterministic stage — D7); the per-chunk worker is idempotent on content-hash + extractor
+  version (D12).
 - **A recall envelope (D35).** Selection biases toward KEEP when unsure; **never-drop classes**
   (quantities, dates, named-entity + predicate, change-of-state language) are protected even if phrased
   opinionatedly; a low-confidence `kept_flagged` outcome marks-for-review instead of hard-deleting; and
   planted rare-fact canaries fail CI if Selection drops them. Drop-rates are tuned against **per-fact**
   loss, never a corpus average — a uniquely-attested fact has no second copy to fall back on.
 
+**Amendment (2026-07-27, issue #161):** Selection keep/keep_flagged is not the end of the story.
+Between a keep and an accepted `claims` row sit Claimify (the fused decontextualize+decompose call)
+and the deterministic D32 grounding gates. Without ledger rows for those stages, a keep that never
+lands a claim is indistinguishable from a keep that produced a claim the gate rejected — and plain
+(unflagged) keeps that die are completely invisible. The D33 transcript therefore also records:
+
+| `decision_type` | When written | `source_span` | `edit_detail` |
+|---|---|---|---|
+| `claimify_omitted` | A kept Selection span for which Claimify returned **no claim at all** (the model simply skipped it). One row per dead keep. | The Selection span. | null |
+| `grounding_rejected` | A Claimify-returned claim rejected by a D32 gate. One row per rejected claim. | The claim's returned `source_span` (even if not findable in the chunk). | `{"gate": "span_not_found" \| "outside_kept_ranges" \| "added_context_unverified", "claim_span": <truncated>}`; for `added_context_unverified` also `{"kind": ..., "text": <truncated>}`. |
+
+**Invariant — every kept span is accounted for end-to-end.** Two independent rules (revised
+2026-07-27 after review — one keep can decompose into several returned claims with mixed fates,
+so "exactly one category per keep" was wrong):
+
+1. **Every Claimify-returned claim** independently ends either **accepted** (a `claims` row, plus
+   any `decontext_edit` / `selection_keep_flagged` pairing that already applied) or
+   **`grounding_rejected`** (one row naming which gate fired). A mixed outcome — same keep, one
+   claim accepted, another rejected — records both and is not an omission.
+2. **Every keep or keep_flagged span with no attributable returned claim** gets exactly one
+   `claimify_omitted` row. Attribution is **anchored-range overlap only** (the claim's resolved
+   char range overlaps the keep's) — text containment is deliberately excluded so one claim
+   cannot suppress omission rows for unrelated keeps that merely share text. Two conservative
+   consequences: a returned claim whose span anchors nowhere is an **orphan rejection** (its
+   `grounding_rejected` row stands; it suppresses no omission), and a Selection span that is not
+   verbatim-findable can never be marked "tried," so it always gets its omission row — the case
+   that previously vanished with no trace.
+
+Cross-model extraction comparisons can then show *why* a stronger model lands more claims (fewer
+omissions vs fewer gate rejections) instead of only *that* it does. On D56 chunk reuse with zero
+attached claims, the prior occurrence's transcript is copied forward verbatim — the synthetic
+`no_info` marker is written only when the prior transcript is itself empty, so reuse never
+rewrites a real loss reason.
+
 ## 4. Why there is no value gate (the non-goal)
 
 It is tempting to put a cheap "is this section even worth extracting?" gate *before* E2. We

plan/designs/postgres_schema_design.md

@@ -1356,8 +1356,8 @@ verbatim `source_span` + offsets + the `added_context` substrings (D32), so grou
 -- change, and claim validity is never superseded (D3) — adjudicated validity lives only on relations.
 -- A row in claims is an ACCEPTED claim: the deterministic grounding gate (anchor + window
 -- membership, D32 layers 1-2) MUST pass, enforced by the CHECK — a claim that fails the gate is
--- never produced (it becomes a ledger entry or is discarded), so the flags exist for audit and are
--- always true here. Large (~5×10⁷) ⇒ monthly partition by ingested_at; logical FKs (D23).
+-- never produced (it becomes a grounding_rejected ledger entry, #161; never silently discarded),
+-- so the flags exist for audit and are always true here. Large (~5×10⁷) ⇒ monthly partition by ingested_at; logical FKs (D23).
 -- ─────────────────────────────────────────────────────────────────────────
 CREATE TABLE claims (
   claim_id        uuid NOT NULL,
@@ -1421,10 +1421,14 @@ CREATE INDEX ix_claims_audit    ON claims (deployment_id) WHERE audit_status = '
 
 -- ─────────────────────────────────────────────────────────────────────────
 -- claim_extraction_decisions — the append-only, version-stamped extraction transcript (D33). It
--- records every Selection DROP (with reason), every low-confidence KEEP-FLAG, and every
--- decontextualization EDIT. Plain keeps are NOT recorded (they ARE the claims row) — keeping the
--- table sized for drops+flags+edits, not every keep. Rebuild reads stored claims + these decisions
--- and never re-calls the model (D7). Large ⇒ monthly partition by decided_at.
+-- records every Selection DROP (with reason), every low-confidence KEEP-FLAG, every
+-- decontextualization EDIT, and (issue #161, 2026-07-27) the Claimify-stage losses:
+-- CLAIMIFY_OMITTED (a kept span the model returned no claim for) and GROUNDING_REJECTED (a
+-- returned claim a D32 gate rejected — gate named in edit_detail; these are ledgered, not
+-- discarded). Plain keeps that land are NOT recorded (they ARE the claims row); a plain keep
+-- that dies now leaves a loss row, so the table sizes for drops+flags+edits+losses, still not
+-- every keep. Rebuild reads stored claims + these decisions and never re-calls the model (D7).
+-- Large ⇒ monthly partition by decided_at.
 -- INVARIANT: a kept_flagged claim is the pair (claims row with kept_flagged=true) + (a
 -- selection_keep_flagged decision here); the ledger is the replay source from which
 -- claims.kept_flagged is reconstituted on rebuild.
@@ -1435,7 +1439,7 @@ CREATE TABLE claim_extraction_decisions (
   doc_id          uuid NOT NULL,               -- LOGICAL FK → documents
   chunk_id        uuid NOT NULL,               -- LOGICAL FK → chunks
   claim_id        uuid,                        -- LOGICAL FK → claims; set for decontext edits + selection_keep_flagged; NULL for selection_drop (no claim produced)
-  decision_type   extraction_decision_type NOT NULL, -- selection_drop | selection_keep_flagged | decontext_edit
+  decision_type   extraction_decision_type NOT NULL, -- selection_drop | selection_keep_flagged | decontext_edit | claimify_omitted | grounding_rejected (#161)
   source_span     text,                        -- the proposition/sentence the decision was about
   reason          selection_drop_reason,       -- for drops: opinion|advice|hypothetical|generic|question|intro|conclusion|no_info|ambiguous|references_boilerplate (D31)
   edit_detail     jsonb,                        -- for decontext edits: what was resolved/added and from which bundle source

src/rememberstack/model/claims.py

@@ -207,6 +207,8 @@ class DecisionType(StrEnum):
     SELECTION_DROP = "selection_drop"
     SELECTION_KEEP_FLAGGED = "selection_keep_flagged"
     DECONTEXT_EDIT = "decontext_edit"
+    CLAIMIFY_OMITTED = "claimify_omitted"
+    GROUNDING_REJECTED = "grounding_rejected"
 
 
 class DecisionRecord(BaseModel):

src/rememberstack/spine/claim_catalog.py

@@ -105,6 +105,23 @@ def attach_reused_claims(
                 )
         return prior_links
 
+    def copy_reused_decisions(self, *, chunk_id: UUID, prior_chunk_id: UUID) -> int:
+        """Replicate a prior chunk's decision transcript under a reused chunk.
+
+        The D56 zero-claim reuse rung: a prior extraction with no accepted
+        claims may still carry claimify_omitted / grounding_rejected rows
+        (#161) — the reused occurrence adopts that transcript verbatim
+        instead of fabricating a no_info marker. Returns how many rows were
+        copied; zero means the prior transcript is empty and the caller owns
+        the terminal marker.
+        """
+        with self._engine.begin() as connection:
+            result = connection.execute(
+                _COPY_CHUNK_DECISIONS,
+                {"chunk_id": chunk_id, "prior_chunk_id": prior_chunk_id},
+            )
+        return result.rowcount or 0
+
     def claims_for_chunks(
         self, *, chunk_ids: tuple[UUID, ...]
     ) -> tuple[ClaimForNormalization, ...]:
@@ -271,6 +288,25 @@ def record_extraction(
     """
 ).bindparams(bindparam("edit_detail", type_=JSON))
 
+_COPY_CHUNK_DECISIONS = text(
+    """
+    -- D56 reuse of a zero-claim extraction: replicate the prior occurrence's
+    -- transcript verbatim under the new chunk so the loss accounting (#161)
+    -- survives reuse. Fresh ids and decided_at=now(): this row records when
+    -- THIS occurrence adopted the outcome, on the current partition.
+    INSERT INTO claim_extraction_decisions (
+        decision_id, deployment_id, doc_id, chunk_id, claim_id,
+        decision_type, source_span, reason, edit_detail,
+        protected_class, extractor_version
+    )
+    SELECT gen_random_uuid(), deployment_id, doc_id, :chunk_id, claim_id,
+           decision_type, source_span, reason, edit_detail,
+           protected_class, extractor_version
+    FROM claim_extraction_decisions
+    WHERE chunk_id = :prior_chunk_id
+    """
+)
+
 _SELECT_CLAIMS_FOR_CHUNKS = text(
     """
     SELECT claim_id, doc_id, chunk_id, claim_text, is_attributed

src/rememberstack/spine/migrations/versions/p1_03_0018_claimify_loss_ledger.py

@@ -0,0 +1,32 @@
+"""Add Claimify-stage loss ledger values to extraction_decision_type (#161).
+
+Every kept Selection span must end in one of {accepted claim(s),
+grounding_rejected row(s), claimify_omitted row}. The two new enum values
+make the previously silent Claimify omissions and D32 gate rejections
+auditable on the append-only D33 transcript. Additive only; no new indexes
+on the partitioned claim_extraction_decisions table.
+"""
+
+from collections.abc import Sequence
+
+from alembic import op
+
+revision: str = "p1_03_0018"
+down_revision: str | None = "p7_05_0017"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def upgrade() -> None:
+    """Append the two loss-ledger values (additive; existing rows untouched)."""
+    op.execute(
+        "ALTER TYPE extraction_decision_type ADD VALUE IF NOT EXISTS 'claimify_omitted'"
+    )
+    op.execute(
+        "ALTER TYPE extraction_decision_type"
+        " ADD VALUE IF NOT EXISTS 'grounding_rejected'"
+    )
+
+
+def downgrade() -> None:
+    """PostgreSQL cannot remove an enum value in place; additive no-op."""

src/rememberstack/workers/e1.py

@@ -54,10 +54,11 @@
 """The context-prefix call's prompt generation (D58; conventional mode, D63).
 07b pins temperature=0.0 — generation parameters are part of provenance."""
 
-E2_EXTRACTOR_VERSION: Final = "e2-extract-2026.07d:temp0-1"
+E2_EXTRACTOR_VERSION: Final = "e2-extract-2026.07e:loss-ledger-1"
 """The extractor generation baked into extraction_input_hash (D56); the E2
-stage (WP-1.3) binds its handler to this same constant. 07d pins
-temperature=0.0 on the Selection call (Claimify already carried it)."""
+stage (WP-1.3) binds its handler to this same constant. 07e ledgers Claimify
+omissions and grounding-gate rejections on the D33 transcript (#161); 07d
+pinned temperature=0.0 on the Selection call (Claimify already carried it)."""
 
 _PREFIX_PROMPT_TEMPLATE: Final = (
     "In one sentence, state where this passage sits in the document — "