Repository navigation
v0.3.0 — Recoverable, evidence-bound judgment workflows
v0.3.0 — Recoverable, evidence-bound judgment workflows
v0.3.0 makes long jevify sweeps easier to plan, resume and review. Requests are sized before submission. Run state is kept in durable bundles, so a sweep can survive a kernel reset. Model labels are tied back to exact source snapshots before anyone acts on them. The method-first guidance is unchanged: prefer deterministic tools or direct reading where they fit, and use Jev (judge_batch) for repeated, independently judgeable semantic decisions.
This release makes no quality or savings claim. See Verification and Limitations.
What changed since v0.2.0
| Area | v0.2.0 | v0.3.0 |
|---|---|---|
| Interrupted runs | Docs said attaching to a surviving batch does not resume jv.run; recovery was manual |
jv.resume(bundle) reconciles a durable journal with surviving host items; resubmission is opt-in |
| Grouping | slots= argument |
Full grouping manifest; every original unit is either in a slot or listed as an explicit exclusion; slots= removed |
| Rubric identity | Reordered choices could share a hash | Question and choice order count toward qhash |
| Eval execution artifacts | Schema 4 | Schema 5 |
| Recipes | A–F | A–I (adds documentation drift, issue intake, candidate relevance) |
Practical capabilities
- Plan before you spend.
jv.plansizes the state plus all questions (including slot expansion) against total and single-request budgets without any inference call. Unknown sizing is reported as not admitted rather than accepted.jv.packgroups units to fit both budgets without truncating required evidence. - Durable runs and recovery.
jv.run(..., bundle_path=...)writes a plan and journal. After a kernel loss,jv.resumekeeps journaled answers and provenance and reattaches to a batch that is still alive in the owning OMP session. Missing deliveries are resubmitted only withresubmit_missing=True. Stale batches stay incomplete. - Source-bound review.
jv.extractionfreezes file, unit and state identities and per-original outcomes.jv.validate_extractionreports source drift separately from coverage.jv.review_queueandjv.adjudicateconfirm or overturn items only with snapshot-bound citations.jv.delete_spanscan require a matching manifest and a completed review, and still defaults to dry run. - Richer score handling. Score rows keep per-level distributions (
.dist), which are never reconstructed from the mean.jv.policyfreezes primitive-aware escalation thresholds against the ordered rubric.jv.calibratehandles choice, bool and score agreement, including grouped slots. - Sensitivity pilots.
jv.sensitivityruns explicit, bounded order/context variants and reports disagreements without voting. It shows stability across the supplied variants. It does not show correctness. - New recipes: G documentation/contract drift, H issue/support intake, I source/skill candidate relevance.
- Eval tooling. Revision-bound outcome comparisons (
compare_graded) and predeclared--thresholds. Authorized scheduling envelopes (--preflight,--authorize-attempts,--authorize-concurrency,--authorize-questions). An isolated grading launcher (evals/grade_launch.py) with cold/warm cache controls. Synthetic extractor validation (validate_extractor_host).
Correctness and safety changes
- Recovery never silently resubmits stale work. Delivered failures are not rerun on resume.
- Torn trailing journal fragments can be recovered. Corruption inside committed records fails closed. Source changes show up in recovery evidence instead of being hidden by old answers.
- Reordering questions or choices changes rubric identity, so differently ordered rubrics no longer share a hash.
- The grader now calls justified repair suggestions
recommended_defects. They count recommendations, not observed code repairs.
Migration
- Re-freeze rubrics for new sweeps. Hashes from earlier versions are not interchangeable with v0.3.0 hashes.
- Run inputs need string unit IDs.
- Pass
grouping=grouping(the full manifest fromjv.group/jv.pack) instead ofslots=. - Eval execution artifacts use schema 5. Saved schema-2 helper receipts remain readable, but they do not gain historical evidence they never recorded.
Verification
- 281 tests passed locally (
python3 -m unittest discover -s tests). CI is green onmainand on the tag at74ead5e. - The installed 0.3.0 plugin passed skill discovery and helper API loading in a fresh OMP process.
- A bounded live smoke on OMP 18.9.1 is retained as sanitized evidence in
evals/results/release-v0.3.0-smoke.json(evidence_kind: bounded_live_smoke_not_quality_benchmark). It covered:- kernel-loss recovery on the same batch: one submission, one reattachment, no resubmission
- planning and grouping with an explicit empty-unit exclusion
- a flat sweep with complete source coverage, plus a source-confirmed review
- an order-permutation sensitivity pilot
- bool and score calibration samples
- recipes G–I
- both isolated grading launches (direct and flat), which graded with no recorded problems
Limitations
- Recovery is not exactly-once execution. An orphan from the submit-to-ID crash window may still bill. Reattachment survives kernel loss, not host process or session loss. It currently depends on POSIX locks and Linux boot identity.
- The direct-versus-flat outcome comparison is inconclusive: it rests on one independent graded case, and a required metric has no interval.
- The extractor check uses 18 source-adjudicated synthetic records, with 0 human adjudication. Agreement was factual 13/18, action 12/18 and introduced 13/18. It made 4 identity-misattribution extraction errors: two unwarranted repairs (r03 and r06, which concern the fixture's refuted claims C07 and C06) and two unearned credits (r02 and r04, which credit the true defects C02 and C01 without an exact-ID mention). Source review remains required, and labels grant no edit authority. The retained JSON keeps its historical field names (
high_impact_false_repairs). - Stock OMP 18.8.7 through 18.9.1 still injects an item-count-based mandatory
jevifykeyword notice that conflicts with the method-first guidance. Installing this plugin does not change that host notice. The companion host patch that would fix it is local and unreleased. - On stock OMP up to 18.9.1, per-item route/cache provenance is unavailable. Rows are flagged
provenance_unknown, and smoke provenance was incomplete. Exhaustive billing is also unknown: evalcompletioncalls are not journaled, and unknown costs stay unknown rather than zero. - The eval scheduling envelope limits which attempts start. It is not a dollar cap, and an in-flight attempt can run past a limit.
--authorize-questionsstops future attempts when the observed limit is exhausted or accounting coverage is unknown. On stock hosts, this guard stops after the first attempt; it cannot cap nested calls already in flight.
Install
omp plugin install 'git+https://github.com/ericjuta/omp-jevify.git#v0.3.0' --force--force replaces an existing install. Start a new OMP session afterwards.
Docs: README · Upgrading to 0.3.0 · Skill reference
Full changelog: v0.2.0...v0.3.0 (6e9be5c runtime, 884e5d8 evals, 74ead5e release)