Skip to content

v1.4.0: did the run answer the question?

Choose a tag to compare

@HarperZ9 HarperZ9 released this 25 Jun 07:11
· 173 commits to main since this release

Forum validates each task against its own instruction, but a run can pass every task and still drift from the original request. This release witnesses that gap.

  • Witnessed intent check: a completed submit() appends an intent_check entry, chained to the answer, with a deterministic coverage of the request's content words by the answer, the terms it missed, and whether that falls below a configurable threshold (Orchestrator(intent_threshold=...)). The payload names its own method (lexical_coverage). It is a lexical floor, not a semantic verdict: low coverage flags a run for a closer look, it does not block the run or declare the answer wrong. A grounded model intent-judge is the next rung above this floor.
  • In the summary and A/B: forum ledger summary and forum bench now report intent checks and how many were flagged, so a prompt or model change that makes runs drift more or less is measured from the record.

Honest about its limits: a correct but paraphrased answer can still flag (the lexical floor's known blind spot). examples/run_intent.py demonstrates exactly that case.

Pure standard library, deterministic and reproducible. 186 tests, plus 2 gated real-model tests.

Install from the attached wheel: pip install forum_engine-1.4.0-py3-none-any.whl