Skip to content

v0.3.1 — Corrected-grader checkpoint

Choose a tag to compare

@ericjuta ericjuta released this 13 Oct 18:00
· 21 commits to main since this release
43e92a2

omp-jevify v0.3.1

Summary

v0.3.1 is a tagged checkpoint of the corrected evaluation grader (9109a59, cb9aacd). It adds no plugin runtime capability. The skill and helper files match v0.3.0 apart from version metadata. No plugin migration from v0.3.0 is needed.

The grader is repository tooling, not part of the installed plugin payload. Check out v0.3.1 to reproduce its corrected scoring; the Git plugin update carries the matching version and documentation.

Install (replace an existing installation with --force):

omp plugin install 'git+https://github.com/ericjuta/omp-jevify.git#v0.3.1' --force

Grader corrections

  • Exact-ID evidence binding. Extraction binds each verdict to an exact claim ID and a bounded answer excerpt before judging. A missing ID is recorded as omitted, not inferred.
  • Foreign-claim block exclusion. A block that names another claim stops continuation, even if that other ID appears only after ID-free sentences. The same claim's heading or label keeps its own body, including text that repeats that claim's ID.
  • Explicit groups. Explicit groups remain valid when they unambiguously include the target. Bold or code formatting around individual IDs does not invalidate a group. Bounded Markdown sections owned by a single claim are accepted.
  • Ambiguous attribution stays unresolved. Bare labels without assertions and ambiguous attribution grant no repair credit. Attribution rejections are counted as unattributed, separate from extraction_errors. Both remain unresolved and stay in the scoring denominators.
  • Scoring revision. Schema-2 scoring uses independent-facets-fixed-denominators-v3. Historical v0.3.0 reports are not rewritten and keep their original scoring revision.

The correction constrains judge input. It does not prove the judge's interpretation is correct. Assigning ID-free prose to a preceding block remains a heuristic; source review is still required.

Historical evidence (unchanged)

The v0.3.0 smoke evidence (evals/results/release-v0.3.0-smoke.json) is retained as recorded. Its 18-record synthetic extractor check showed two unwarranted repairs (r03 → refuted C07, r06 → refuted C06) and two unearned credits (r02 → C02, r04 → C01), where true defects were credited without an exact-ID mention. Those outcomes are historical and are not corrected in place.

Review status

The post-release review of 9109a59 was performed by an agent that checked r02, r03, r04, r06 against the synthetic fixture source and confirmed r05's explicit-group attribution. This is agent adjudication, not human validation.

Verification

  • 306 unit tests pass on the release candidate.
  • CI passed for commit 43e92a28c58f154c176fbaacb6798f5acbb4fae6 on both main and the v0.3.1 tag.
  • Offline admission smoke preserves the omissions for r02/r03/r04/r06, accepts r05's explicit group, excludes foreign-claim blocks, retains same-claim section bodies, accepts bold/code-wrapped group IDs, and leaves disjunctions unresolved.
  • Skill frontmatter/version checks, Python compilation, and the 30-hunk fixture check pass. The updated README renders through GitHub Markdown.
  • Installed directly from the v0.3.1 Git tag. A fresh OMP process discovered skill version 0.3.1; the installed metadata, README, skill, helper and reference matched commit 43e92a28c58f154c176fbaacb6798f5acbb4fae6. The installed helper accepted an in-budget plan and rejected an over-budget plan without model requests.
  • These are deterministic/offline checks, not a new model-quality benchmark. No model requests were made.

Limits

  • No new quality or savings benchmark exists for this release. No new model judgments were run.
  • v0.3.1 makes no quality or cost-savings claim.
  • Recovery is not exactly-once execution. Host provenance and exhaustive billing remain unknown.