Skip to content

Releases

v0.5.1 — Source-backed Clairic evidence workflows

Choose a tag to compare

@ericjuta ericjuta released this 07 Nov 21:23
v0.5.1
de7ecf2

Source-backed evidence workflows

  • Immutable source snapshots, exact Unicode spans, original-ID scope accounting, and dependency-aware invalidation.
  • Default-on jevify_extract for selected non-secret public/private evidence, with explicit opt-out and deterministic fallback. Uses host OpenRouter credentials; no plugin-owned keys. Descriptor-confined transport requires Linux.
  • Ready-only review batches and a source-bound adjudication inbox. Missing evidence remains visible; stale evidence revokes approval. Review confirmation never grants edit or deletion authority.
  • Documentation, migration, issue/log, and explicit external-history import workflows. Imports preserve provenance, not authority; no ambient OptChat traversal or claimed upstream exporter.
  • Preregistered four-arm preparation evaluations, with an isolated C-versus-B comparison, independent-case gates, controller-bound receipts, and explicit uncertainty.

Verification and limits

575 Python tests, 29 adapter tests, and TypeScript checking passed. The packaged smoke captured four controller receipts and four lifecycle observations through the real SDK's in-memory session components, with five loopback requests and zero outbound provider requests. The final focused reviews cleared the reported blockers. Five signed commits separate implementation, documentation, release metadata, and the CI compatibility repair; the signed v0.5.1 tag points to de7ecf2.

Provider-free helper and packaged fresh-process SDK/Python checks exercise Unicode/CRLF handling, abstention, opt-out, safe zero-transmission recovery, path refusal, interruption, source drift, review readiness, and receipt capture. Synthetic CLI demonstrations test decision behavior; they are not measured model results.

No real Clairic or Jev calls were made for this release. Production provider-wire compatibility, settled billing, quality improvements, review-time savings, and cost savings are not established. A real-provider smoke requires separate approval of the exact manifest digest and spending cap; controlled comparisons additionally require independently adjudicated realistic gold.

Bun compatibility

The initial v0.5.0 tag exposed a CI incompatibility: Bun 1.3.14 cannot read Bun 1.4's newer lockfile format. That tag is preserved. v0.5.1 uses a compatible lockfile without changing any dependency version or integrity hash. Frozen installation, typechecking, all 29 adapter tests, and the packaged runtime smoke also passed on the declared Bun 1.3.14 minimum.

Install

omp plugin install 'git+https://github.com/ericjuta/omp-jevify.git#v0.5.1'

Requires OMP 18.10.3–18.x and Bun >=1.3.14. Start a new OMP session after installation to load the updated extension.

v0.4.0

Choose a tag to compare

@ericjuta ericjuta released this 21 Oct 09:28
9fe7062

omp-jevify v0.4.0

Install:

omp plugin install 'git+https://github.com/ericjuta/omp-jevify.git#v0.4.0' --force

Why

A review of recent real sessions on OMP 18.12.5 found that none of them used the jv.* helpers.
The host jevify-notice sends agents straight to the built-in judge_batch(), so the receipts,
flags and review did not apply. In 3 of 4 of those sessions a direct script did the real work.
A paired eval on v0.3.x gave the same result: direct scripts scored 70/70 and Jev scored 64/70 on
structural audit tasks. This release makes notice-driven runs auditable and says plainly when a
direct method is the better choice.

Added

Structured findings grading (evals/findings.py, jevify-findings/1). This replaces the
v0.3.1 prose extractor.

  • Each finding has exact claim IDs, the reviewed source_revision, factual/action/introduced
    facets, and evidence quotes that must equal the cited source lines.
  • Prose, misnamed IDs, a stale revision or a quote that doesn't match earns no credit. The
    scoring denominators are fixed.

Durable, source-bound review inbox (jv.review_queue, review_questions, adjudicate,
review_items, review_status).

  • Verdicts are stored in files, bound to rows and record the reviewer's identity.
  • Adjudication is all-or-nothing.
  • Changes to the source, rubric or evidence mark items stale.
  • grants_authority is always false.
  • delete_spans refuses to run while anything is drifted or unresolved.

Stricter pilot reports (evals/run.py --pilot, jevify-pilot/1, direct-reading
strategy, result schema 6, compare_graded).

  • A method recommendation, including "use direct scripting", is withheld unless all of these
    hold: the arms are comparable, every case has gold from an independent adjudicator with a
    distinct key digest, and each arm meets its own thresholds.
  • Duplicate fixture/key aliases count once.

Host-notice adoption helpers.

  • jv.adopt(items, states, Q, batch=b) and jv.scalar turn built-in judge_batch() output into
    receipt-bound rows.
  • jv.pilot_check returns use_direct when more than about 40% of items are flagged, in the
    uncertainty band or caught by the catch-all.
  • jv.remaining reuses unchanged verdicts, jv.gate checks for regressions, jv.power sizes a
    sample, and jv.Ledger keeps tuned units out of held-out evidence.
  • jv.facts, jv.units_from_table and jv.refuse_elided handle input hygiene, and
    jv.flag(report=) was added. jv.choice enforces the 255-label route limit, and jv.plan
    now reports labels, over_labels and pack_suggestion.
  • jv.splice, jv.comparable, jv.export and jv.receipt_md were added.
  • Rows now carry state_sha256, judged_at and assembly. Receipts now carry served_models,
    comparable, host_counts and billing.

Verification

  • 456 unit tests pass. Each of the four commits passes the suite on its own in a clean worktree.
  • CI passed on main (run 39577134316) and on the tag (run 39577136651).
  • Offline checks made no model requests: structured grading, inbox persistence across
    processes, stale-evidence rejection, replay of retained real judgment rows, and pilot
    withholding.
  • A live smoke on OMP 18.12.5 cost about $0.0002 in total. It ran adopt, run, remaining,
    splice, gate, receipt_md and export.

Boundaries

  • These are workflow checks only. This release makes no claim about quality or savings.
  • Provenance stays unknown. Stock 18.12.5 items expose only key, ok, error, answers
    and model, with no fallback or cache fields. Cached answers report the alias
    ~typesafe/jev-latest, so receipts that mix cached and fresh rows are comparable: false.
  • Billing is an unverified estimate.
  • The repository has one schema-2 answer key, which cannot support a method recommendation.
    Pilot comparisons on the development cases come out withheld or inconclusive.
  • Installing the plugin does not change the host notice.

v0.3.1 — Corrected-grader checkpoint

Choose a tag to compare

@ericjuta ericjuta released this 13 Oct 18:00
43e92a2

omp-jevify v0.3.1

Summary

v0.3.1 is a tagged checkpoint of the corrected evaluation grader (9109a59, cb9aacd). It adds no plugin runtime capability. The skill and helper files match v0.3.0 apart from version metadata. No plugin migration from v0.3.0 is needed.

The grader is repository tooling, not part of the installed plugin payload. Check out v0.3.1 to reproduce its corrected scoring; the Git plugin update carries the matching version and documentation.

Install (replace an existing installation with --force):

omp plugin install 'git+https://github.com/ericjuta/omp-jevify.git#v0.3.1' --force

Grader corrections

  • Exact-ID evidence binding. Extraction binds each verdict to an exact claim ID and a bounded answer excerpt before judging. A missing ID is recorded as omitted, not inferred.
  • Foreign-claim block exclusion. A block that names another claim stops continuation, even if that other ID appears only after ID-free sentences. The same claim's heading or label keeps its own body, including text that repeats that claim's ID.
  • Explicit groups. Explicit groups remain valid when they unambiguously include the target. Bold or code formatting around individual IDs does not invalidate a group. Bounded Markdown sections owned by a single claim are accepted.
  • Ambiguous attribution stays unresolved. Bare labels without assertions and ambiguous attribution grant no repair credit. Attribution rejections are counted as unattributed, separate from extraction_errors. Both remain unresolved and stay in the scoring denominators.
  • Scoring revision. Schema-2 scoring uses independent-facets-fixed-denominators-v3. Historical v0.3.0 reports are not rewritten and keep their original scoring revision.

The correction constrains judge input. It does not prove the judge's interpretation is correct. Assigning ID-free prose to a preceding block remains a heuristic; source review is still required.

Historical evidence (unchanged)

The v0.3.0 smoke evidence (evals/results/release-v0.3.0-smoke.json) is retained as recorded. Its 18-record synthetic extractor check showed two unwarranted repairs (r03 → refuted C07, r06 → refuted C06) and two unearned credits (r02 → C02, r04 → C01), where true defects were credited without an exact-ID mention. Those outcomes are historical and are not corrected in place.

Review status

The post-release review of 9109a59 was performed by an agent that checked r02, r03, r04, r06 against the synthetic fixture source and confirmed r05's explicit-group attribution. This is agent adjudication, not human validation.

Verification

  • 306 unit tests pass on the release candidate.
  • CI passed for commit 43e92a28c58f154c176fbaacb6798f5acbb4fae6 on both main and the v0.3.1 tag.
  • Offline admission smoke preserves the omissions for r02/r03/r04/r06, accepts r05's explicit group, excludes foreign-claim blocks, retains same-claim section bodies, accepts bold/code-wrapped group IDs, and leaves disjunctions unresolved.
  • Skill frontmatter/version checks, Python compilation, and the 30-hunk fixture check pass. The updated README renders through GitHub Markdown.
  • Installed directly from the v0.3.1 Git tag. A fresh OMP process discovered skill version 0.3.1; the installed metadata, README, skill, helper and reference matched commit 43e92a28c58f154c176fbaacb6798f5acbb4fae6. The installed helper accepted an in-budget plan and rejected an over-budget plan without model requests.
  • These are deterministic/offline checks, not a new model-quality benchmark. No model requests were made.

Limits

  • No new quality or savings benchmark exists for this release. No new model judgments were run.
  • v0.3.1 makes no quality or cost-savings claim.
  • Recovery is not exactly-once execution. Host provenance and exhaustive billing remain unknown.

v0.3.0 — Recoverable, evidence-bound judgment workflows

Choose a tag to compare

@ericjuta ericjuta released this 11 Oct 18:43
74ead5e

v0.3.0 — Recoverable, evidence-bound judgment workflows

v0.3.0 makes long jevify sweeps easier to plan, resume and review. Requests are sized before submission. Run state is kept in durable bundles, so a sweep can survive a kernel reset. Model labels are tied back to exact source snapshots before anyone acts on them. The method-first guidance is unchanged: prefer deterministic tools or direct reading where they fit, and use Jev (judge_batch) for repeated, independently judgeable semantic decisions.

This release makes no quality or savings claim. See Verification and Limitations.

What changed since v0.2.0

Area v0.2.0 v0.3.0
Interrupted runs Docs said attaching to a surviving batch does not resume jv.run; recovery was manual jv.resume(bundle) reconciles a durable journal with surviving host items; resubmission is opt-in
Grouping slots= argument Full grouping manifest; every original unit is either in a slot or listed as an explicit exclusion; slots= removed
Rubric identity Reordered choices could share a hash Question and choice order count toward qhash
Eval execution artifacts Schema 4 Schema 5
Recipes A–F A–I (adds documentation drift, issue intake, candidate relevance)

Practical capabilities

  • Plan before you spend. jv.plan sizes the state plus all questions (including slot expansion) against total and single-request budgets without any inference call. Unknown sizing is reported as not admitted rather than accepted. jv.pack groups units to fit both budgets without truncating required evidence.
  • Durable runs and recovery. jv.run(..., bundle_path=...) writes a plan and journal. After a kernel loss, jv.resume keeps journaled answers and provenance and reattaches to a batch that is still alive in the owning OMP session. Missing deliveries are resubmitted only with resubmit_missing=True. Stale batches stay incomplete.
  • Source-bound review. jv.extraction freezes file, unit and state identities and per-original outcomes. jv.validate_extraction reports source drift separately from coverage. jv.review_queue and jv.adjudicate confirm or overturn items only with snapshot-bound citations. jv.delete_spans can require a matching manifest and a completed review, and still defaults to dry run.
  • Richer score handling. Score rows keep per-level distributions (.dist), which are never reconstructed from the mean. jv.policy freezes primitive-aware escalation thresholds against the ordered rubric. jv.calibrate handles choice, bool and score agreement, including grouped slots.
  • Sensitivity pilots. jv.sensitivity runs explicit, bounded order/context variants and reports disagreements without voting. It shows stability across the supplied variants. It does not show correctness.
  • New recipes: G documentation/contract drift, H issue/support intake, I source/skill candidate relevance.
  • Eval tooling. Revision-bound outcome comparisons (compare_graded) and predeclared --thresholds. Authorized scheduling envelopes (--preflight, --authorize-attempts, --authorize-concurrency, --authorize-questions). An isolated grading launcher (evals/grade_launch.py) with cold/warm cache controls. Synthetic extractor validation (validate_extractor_host).

Correctness and safety changes

  • Recovery never silently resubmits stale work. Delivered failures are not rerun on resume.
  • Torn trailing journal fragments can be recovered. Corruption inside committed records fails closed. Source changes show up in recovery evidence instead of being hidden by old answers.
  • Reordering questions or choices changes rubric identity, so differently ordered rubrics no longer share a hash.
  • The grader now calls justified repair suggestions recommended_defects. They count recommendations, not observed code repairs.

Migration

  • Re-freeze rubrics for new sweeps. Hashes from earlier versions are not interchangeable with v0.3.0 hashes.
  • Run inputs need string unit IDs.
  • Pass grouping=grouping (the full manifest from jv.group/jv.pack) instead of slots=.
  • Eval execution artifacts use schema 5. Saved schema-2 helper receipts remain readable, but they do not gain historical evidence they never recorded.

Verification

  • 281 tests passed locally (python3 -m unittest discover -s tests). CI is green on main and on the tag at 74ead5e.
  • The installed 0.3.0 plugin passed skill discovery and helper API loading in a fresh OMP process.
  • A bounded live smoke on OMP 18.9.1 is retained as sanitized evidence in evals/results/release-v0.3.0-smoke.json (evidence_kind: bounded_live_smoke_not_quality_benchmark). It covered:
    • kernel-loss recovery on the same batch: one submission, one reattachment, no resubmission
    • planning and grouping with an explicit empty-unit exclusion
    • a flat sweep with complete source coverage, plus a source-confirmed review
    • an order-permutation sensitivity pilot
    • bool and score calibration samples
    • recipes G–I
    • both isolated grading launches (direct and flat), which graded with no recorded problems

Limitations

  • Recovery is not exactly-once execution. An orphan from the submit-to-ID crash window may still bill. Reattachment survives kernel loss, not host process or session loss. It currently depends on POSIX locks and Linux boot identity.
  • The direct-versus-flat outcome comparison is inconclusive: it rests on one independent graded case, and a required metric has no interval.
  • The extractor check uses 18 source-adjudicated synthetic records, with 0 human adjudication. Agreement was factual 13/18, action 12/18 and introduced 13/18. It made 4 identity-misattribution extraction errors: two unwarranted repairs (r03 and r06, which concern the fixture's refuted claims C07 and C06) and two unearned credits (r02 and r04, which credit the true defects C02 and C01 without an exact-ID mention). Source review remains required, and labels grant no edit authority. The retained JSON keeps its historical field names (high_impact_false_repairs).
  • Stock OMP 18.8.7 through 18.9.1 still injects an item-count-based mandatory jevify keyword notice that conflicts with the method-first guidance. Installing this plugin does not change that host notice. The companion host patch that would fix it is local and unreleased.
  • On stock OMP up to 18.9.1, per-item route/cache provenance is unavailable. Rows are flagged provenance_unknown, and smoke provenance was incomplete. Exhaustive billing is also unknown: eval completion calls are not journaled, and unknown costs stay unknown rather than zero.
  • The eval scheduling envelope limits which attempts start. It is not a dollar cap, and an in-flight attempt can run past a limit.
  • --authorize-questions stops future attempts when the observed limit is exhausted or accounting coverage is unknown. On stock hosts, this guard stops after the first attempt; it cannot cap nested calls already in flight.

Install

omp plugin install 'git+https://github.com/ericjuta/omp-jevify.git#v0.3.0' --force

--force replaces an existing install. Start a new OMP session afterwards.

Docs: README · Upgrading to 0.3.0 · Skill reference

Full changelog: v0.2.0...v0.3.0 (6e9be5c runtime, 884e5d8 evals, 74ead5e release)