Skip to content

Leveson concern: no end-to-end test for failed-extraction propagation through synthesis #3

@melek

Description

@melek

Problem

When EXTRACT_FIELD oracle contract validation fails, recovery sets value="extraction_failed" and confidence="low". This record propagates to Phase 5 synthesis. The SYNTHESIZE_CLAIM contract checks data_completeness and requires qualification for low-completeness papers — but there is no test verifying this end-to-end chain.

A plausible failure mode: a paper with multiple failed extractions gets cited in the review without qualification, making claims appear better-supported than the evidence warrants.

Proposed Fix

Add an integration test that:

  1. Creates a mock workspace with extraction records containing extraction_failed values
  2. Runs validate_synthesis_claims() against a review that cites those papers without qualification
  3. Verifies the validator catches the missing qualification
  4. Verifies the validator passes when qualification is present

Also: when Phase 4→3 feedback loop terminates at max_feedback_iterations with conceptual saturation still above θ_c, the progress line should include a warning — not just the neutral "returning to Phase 3" message. The current reporting understates the risk of an incomplete concept space.

Context

Leveson (STAMP/STPA Safety) panelist concern during panel review (2026-03-23). The individual pieces work (oracle contract recovery is tested, synthesis claim validation is tested) — the gap is the composition.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions