Repository navigation
Releases
v0.5.1 — Source-backed Clairic evidence workflows
Source-backed evidence workflows
- Immutable source snapshots, exact Unicode spans, original-ID scope accounting, and dependency-aware invalidation.
- Default-on
jevify_extractfor selected non-secret public/private evidence, with explicit opt-out and deterministic fallback. Uses host OpenRouter credentials; no plugin-owned keys. Descriptor-confined transport requires Linux. - Ready-only review batches and a source-bound adjudication inbox. Missing evidence remains visible; stale evidence revokes approval. Review confirmation never grants edit or deletion authority.
- Documentation, migration, issue/log, and explicit external-history import workflows. Imports preserve provenance, not authority; no ambient OptChat traversal or claimed upstream exporter.
- Preregistered four-arm preparation evaluations, with an isolated C-versus-B comparison, independent-case gates, controller-bound receipts, and explicit uncertainty.
Verification and limits
575 Python tests, 29 adapter tests, and TypeScript checking passed. The packaged smoke captured four controller receipts and four lifecycle observations through the real SDK's in-memory session components, with five loopback requests and zero outbound provider requests. The final focused reviews cleared the reported blockers. Five signed commits separate implementation, documentation, release metadata, and the CI compatibility repair; the signed v0.5.1 tag points to de7ecf2.
Provider-free helper and packaged fresh-process SDK/Python checks exercise Unicode/CRLF handling, abstention, opt-out, safe zero-transmission recovery, path refusal, interruption, source drift, review readiness, and receipt capture. Synthetic CLI demonstrations test decision behavior; they are not measured model results.
No real Clairic or Jev calls were made for this release. Production provider-wire compatibility, settled billing, quality improvements, review-time savings, and cost savings are not established. A real-provider smoke requires separate approval of the exact manifest digest and spending cap; controlled comparisons additionally require independently adjudicated realistic gold.
Bun compatibility
The initial v0.5.0 tag exposed a CI incompatibility: Bun 1.3.14 cannot read Bun 1.4's newer lockfile format. That tag is preserved. v0.5.1 uses a compatible lockfile without changing any dependency version or integrity hash. Frozen installation, typechecking, all 29 adapter tests, and the packaged runtime smoke also passed on the declared Bun 1.3.14 minimum.
Install
omp plugin install 'git+https://github.com/ericjuta/omp-jevify.git#v0.5.1'Requires OMP 18.10.3–18.x and Bun >=1.3.14. Start a new OMP session after installation to load the updated extension.
v0.4.0
omp-jevify v0.4.0
Install:
omp plugin install 'git+https://github.com/ericjuta/omp-jevify.git#v0.4.0' --forceWhy
A review of recent real sessions on OMP 18.12.5 found that none of them used the jv.* helpers.
The host jevify-notice sends agents straight to the built-in judge_batch(), so the receipts,
flags and review did not apply. In 3 of 4 of those sessions a direct script did the real work.
A paired eval on v0.3.x gave the same result: direct scripts scored 70/70 and Jev scored 64/70 on
structural audit tasks. This release makes notice-driven runs auditable and says plainly when a
direct method is the better choice.
Added
Structured findings grading (evals/findings.py, jevify-findings/1). This replaces the
v0.3.1 prose extractor.
- Each finding has exact claim IDs, the reviewed
source_revision, factual/action/introduced
facets, and evidence quotes that must equal the cited source lines. - Prose, misnamed IDs, a stale revision or a quote that doesn't match earns no credit. The
scoring denominators are fixed.
Durable, source-bound review inbox (jv.review_queue, review_questions, adjudicate,
review_items, review_status).
- Verdicts are stored in files, bound to rows and record the reviewer's identity.
- Adjudication is all-or-nothing.
- Changes to the source, rubric or evidence mark items stale.
grants_authorityis always false.delete_spansrefuses to run while anything is drifted or unresolved.
Stricter pilot reports (evals/run.py --pilot, jevify-pilot/1, direct-reading
strategy, result schema 6, compare_graded).
- A method recommendation, including "use direct scripting", is withheld unless all of these
hold: the arms are comparable, every case has gold from an independent adjudicator with a
distinct key digest, and each arm meets its own thresholds. - Duplicate fixture/key aliases count once.
Host-notice adoption helpers.
jv.adopt(items, states, Q, batch=b)andjv.scalarturn built-injudge_batch()output into
receipt-bound rows.jv.pilot_checkreturnsuse_directwhen more than about 40% of items are flagged, in the
uncertainty band or caught by the catch-all.jv.remainingreuses unchanged verdicts,jv.gatechecks for regressions,jv.powersizes a
sample, andjv.Ledgerkeeps tuned units out of held-out evidence.jv.facts,jv.units_from_tableandjv.refuse_elidedhandle input hygiene, and
jv.flag(report=)was added.jv.choiceenforces the 255-label route limit, andjv.plan
now reportslabels,over_labelsandpack_suggestion.jv.splice,jv.comparable,jv.exportandjv.receipt_mdwere added.- Rows now carry
state_sha256,judged_atandassembly. Receipts now carryserved_models,
comparable,host_countsandbilling.
Verification
- 456 unit tests pass. Each of the four commits passes the suite on its own in a clean worktree.
- CI passed on main (run 39577134316) and on the tag (run 39577136651).
- Offline checks made no model requests: structured grading, inbox persistence across
processes, stale-evidence rejection, replay of retained real judgment rows, and pilot
withholding. - A live smoke on OMP 18.12.5 cost about $0.0002 in total. It ran adopt, run, remaining,
splice, gate, receipt_md and export.
Boundaries
- These are workflow checks only. This release makes no claim about quality or savings.
- Provenance stays unknown. Stock 18.12.5 items expose only
key,ok,error,answers
andmodel, with no fallback or cache fields. Cached answers report the alias
~typesafe/jev-latest, so receipts that mix cached and fresh rows arecomparable: false. - Billing is an unverified estimate.
- The repository has one schema-2 answer key, which cannot support a method recommendation.
Pilot comparisons on the development cases come out withheld or inconclusive. - Installing the plugin does not change the host notice.
v0.3.1 — Corrected-grader checkpoint
omp-jevify v0.3.1
Summary
v0.3.1 is a tagged checkpoint of the corrected evaluation grader (9109a59, cb9aacd). It adds no plugin runtime capability. The skill and helper files match v0.3.0 apart from version metadata. No plugin migration from v0.3.0 is needed.
The grader is repository tooling, not part of the installed plugin payload. Check out v0.3.1 to reproduce its corrected scoring; the Git plugin update carries the matching version and documentation.
Install (replace an existing installation with --force):
omp plugin install 'git+https://github.com/ericjuta/omp-jevify.git#v0.3.1' --forceGrader corrections
- Exact-ID evidence binding. Extraction binds each verdict to an exact claim ID and a bounded answer excerpt before judging. A missing ID is recorded as omitted, not inferred.
- Foreign-claim block exclusion. A block that names another claim stops continuation, even if that other ID appears only after ID-free sentences. The same claim's heading or label keeps its own body, including text that repeats that claim's ID.
- Explicit groups. Explicit groups remain valid when they unambiguously include the target. Bold or code formatting around individual IDs does not invalidate a group. Bounded Markdown sections owned by a single claim are accepted.
- Ambiguous attribution stays unresolved. Bare labels without assertions and ambiguous attribution grant no repair credit. Attribution rejections are counted as
unattributed, separate fromextraction_errors. Both remain unresolved and stay in the scoring denominators. - Scoring revision. Schema-2 scoring uses
independent-facets-fixed-denominators-v3. Historical v0.3.0 reports are not rewritten and keep their original scoring revision.
The correction constrains judge input. It does not prove the judge's interpretation is correct. Assigning ID-free prose to a preceding block remains a heuristic; source review is still required.
Historical evidence (unchanged)
The v0.3.0 smoke evidence (evals/results/release-v0.3.0-smoke.json) is retained as recorded. Its 18-record synthetic extractor check showed two unwarranted repairs (r03 → refuted C07, r06 → refuted C06) and two unearned credits (r02 → C02, r04 → C01), where true defects were credited without an exact-ID mention. Those outcomes are historical and are not corrected in place.
Review status
The post-release review of 9109a59 was performed by an agent that checked r02, r03, r04, r06 against the synthetic fixture source and confirmed r05's explicit-group attribution. This is agent adjudication, not human validation.
Verification
- 306 unit tests pass on the release candidate.
- CI passed for commit
43e92a28c58f154c176fbaacb6798f5acbb4fae6on both main and the v0.3.1 tag. - Offline admission smoke preserves the omissions for r02/r03/r04/r06, accepts r05's explicit group, excludes foreign-claim blocks, retains same-claim section bodies, accepts bold/code-wrapped group IDs, and leaves disjunctions unresolved.
- Skill frontmatter/version checks, Python compilation, and the 30-hunk fixture check pass. The updated README renders through GitHub Markdown.
- Installed directly from the
v0.3.1Git tag. A fresh OMP process discovered skill version0.3.1; the installed metadata, README, skill, helper and reference matched commit43e92a28c58f154c176fbaacb6798f5acbb4fae6. The installed helper accepted an in-budget plan and rejected an over-budget plan without model requests. - These are deterministic/offline checks, not a new model-quality benchmark. No model requests were made.
Limits
- No new quality or savings benchmark exists for this release. No new model judgments were run.
- v0.3.1 makes no quality or cost-savings claim.
- Recovery is not exactly-once execution. Host provenance and exhaustive billing remain unknown.
v0.3.0 — Recoverable, evidence-bound judgment workflows
v0.3.0 — Recoverable, evidence-bound judgment workflows
v0.3.0 makes long jevify sweeps easier to plan, resume and review. Requests are sized before submission. Run state is kept in durable bundles, so a sweep can survive a kernel reset. Model labels are tied back to exact source snapshots before anyone acts on them. The method-first guidance is unchanged: prefer deterministic tools or direct reading where they fit, and use Jev (judge_batch) for repeated, independently judgeable semantic decisions.
This release makes no quality or savings claim. See Verification and Limitations.
What changed since v0.2.0
| Area | v0.2.0 | v0.3.0 |
|---|---|---|
| Interrupted runs | Docs said attaching to a surviving batch does not resume jv.run; recovery was manual |
jv.resume(bundle) reconciles a durable journal with surviving host items; resubmission is opt-in |
| Grouping | slots= argument |
Full grouping manifest; every original unit is either in a slot or listed as an explicit exclusion; slots= removed |
| Rubric identity | Reordered choices could share a hash | Question and choice order count toward qhash |
| Eval execution artifacts | Schema 4 | Schema 5 |
| Recipes | A–F | A–I (adds documentation drift, issue intake, candidate relevance) |
Practical capabilities
- Plan before you spend.
jv.plansizes the state plus all questions (including slot expansion) against total and single-request budgets without any inference call. Unknown sizing is reported as not admitted rather than accepted.jv.packgroups units to fit both budgets without truncating required evidence. - Durable runs and recovery.
jv.run(..., bundle_path=...)writes a plan and journal. After a kernel loss,jv.resumekeeps journaled answers and provenance and reattaches to a batch that is still alive in the owning OMP session. Missing deliveries are resubmitted only withresubmit_missing=True. Stale batches stay incomplete. - Source-bound review.
jv.extractionfreezes file, unit and state identities and per-original outcomes.jv.validate_extractionreports source drift separately from coverage.jv.review_queueandjv.adjudicateconfirm or overturn items only with snapshot-bound citations.jv.delete_spanscan require a matching manifest and a completed review, and still defaults to dry run. - Richer score handling. Score rows keep per-level distributions (
.dist), which are never reconstructed from the mean.jv.policyfreezes primitive-aware escalation thresholds against the ordered rubric.jv.calibratehandles choice, bool and score agreement, including grouped slots. - Sensitivity pilots.
jv.sensitivityruns explicit, bounded order/context variants and reports disagreements without voting. It shows stability across the supplied variants. It does not show correctness. - New recipes: G documentation/contract drift, H issue/support intake, I source/skill candidate relevance.
- Eval tooling. Revision-bound outcome comparisons (
compare_graded) and predeclared--thresholds. Authorized scheduling envelopes (--preflight,--authorize-attempts,--authorize-concurrency,--authorize-questions). An isolated grading launcher (evals/grade_launch.py) with cold/warm cache controls. Synthetic extractor validation (validate_extractor_host).
Correctness and safety changes
- Recovery never silently resubmits stale work. Delivered failures are not rerun on resume.
- Torn trailing journal fragments can be recovered. Corruption inside committed records fails closed. Source changes show up in recovery evidence instead of being hidden by old answers.
- Reordering questions or choices changes rubric identity, so differently ordered rubrics no longer share a hash.
- The grader now calls justified repair suggestions
recommended_defects. They count recommendations, not observed code repairs.
Migration
- Re-freeze rubrics for new sweeps. Hashes from earlier versions are not interchangeable with v0.3.0 hashes.
- Run inputs need string unit IDs.
- Pass
grouping=grouping(the full manifest fromjv.group/jv.pack) instead ofslots=. - Eval execution artifacts use schema 5. Saved schema-2 helper receipts remain readable, but they do not gain historical evidence they never recorded.
Verification
- 281 tests passed locally (
python3 -m unittest discover -s tests). CI is green onmainand on the tag at74ead5e. - The installed 0.3.0 plugin passed skill discovery and helper API loading in a fresh OMP process.
- A bounded live smoke on OMP 18.9.1 is retained as sanitized evidence in
evals/results/release-v0.3.0-smoke.json(evidence_kind: bounded_live_smoke_not_quality_benchmark). It covered:- kernel-loss recovery on the same batch: one submission, one reattachment, no resubmission
- planning and grouping with an explicit empty-unit exclusion
- a flat sweep with complete source coverage, plus a source-confirmed review
- an order-permutation sensitivity pilot
- bool and score calibration samples
- recipes G–I
- both isolated grading launches (direct and flat), which graded with no recorded problems
Limitations
- Recovery is not exactly-once execution. An orphan from the submit-to-ID crash window may still bill. Reattachment survives kernel loss, not host process or session loss. It currently depends on POSIX locks and Linux boot identity.
- The direct-versus-flat outcome comparison is inconclusive: it rests on one independent graded case, and a required metric has no interval.
- The extractor check uses 18 source-adjudicated synthetic records, with 0 human adjudication. Agreement was factual 13/18, action 12/18 and introduced 13/18. It made 4 identity-misattribution extraction errors: two unwarranted repairs (r03 and r06, which concern the fixture's refuted claims C07 and C06) and two unearned credits (r02 and r04, which credit the true defects C02 and C01 without an exact-ID mention). Source review remains required, and labels grant no edit authority. The retained JSON keeps its historical field names (
high_impact_false_repairs). - Stock OMP 18.8.7 through 18.9.1 still injects an item-count-based mandatory
jevifykeyword notice that conflicts with the method-first guidance. Installing this plugin does not change that host notice. The companion host patch that would fix it is local and unreleased. - On stock OMP up to 18.9.1, per-item route/cache provenance is unavailable. Rows are flagged
provenance_unknown, and smoke provenance was incomplete. Exhaustive billing is also unknown: evalcompletioncalls are not journaled, and unknown costs stay unknown rather than zero. - The eval scheduling envelope limits which attempts start. It is not a dollar cap, and an in-flight attempt can run past a limit.
--authorize-questionsstops future attempts when the observed limit is exhausted or accounting coverage is unknown. On stock hosts, this guard stops after the first attempt; it cannot cap nested calls already in flight.
Install
omp plugin install 'git+https://github.com/ericjuta/omp-jevify.git#v0.3.0' --force--force replaces an existing install. Start a new OMP session afterwards.
Docs: README · Upgrading to 0.3.0 · Skill reference
Full changelog: v0.2.0...v0.3.0 (6e9be5c runtime, 884e5d8 evals, 74ead5e release)