Skip to content

v0.4.0

Choose a tag to compare

@ericjuta ericjuta released this 21 Oct 09:28
· 17 commits to main since this release
9fe7062

omp-jevify v0.4.0

Install:

omp plugin install 'git+https://github.com/ericjuta/omp-jevify.git#v0.4.0' --force

Why

A review of recent real sessions on OMP 18.12.5 found that none of them used the jv.* helpers.
The host jevify-notice sends agents straight to the built-in judge_batch(), so the receipts,
flags and review did not apply. In 3 of 4 of those sessions a direct script did the real work.
A paired eval on v0.3.x gave the same result: direct scripts scored 70/70 and Jev scored 64/70 on
structural audit tasks. This release makes notice-driven runs auditable and says plainly when a
direct method is the better choice.

Added

Structured findings grading (evals/findings.py, jevify-findings/1). This replaces the
v0.3.1 prose extractor.

  • Each finding has exact claim IDs, the reviewed source_revision, factual/action/introduced
    facets, and evidence quotes that must equal the cited source lines.
  • Prose, misnamed IDs, a stale revision or a quote that doesn't match earns no credit. The
    scoring denominators are fixed.

Durable, source-bound review inbox (jv.review_queue, review_questions, adjudicate,
review_items, review_status).

  • Verdicts are stored in files, bound to rows and record the reviewer's identity.
  • Adjudication is all-or-nothing.
  • Changes to the source, rubric or evidence mark items stale.
  • grants_authority is always false.
  • delete_spans refuses to run while anything is drifted or unresolved.

Stricter pilot reports (evals/run.py --pilot, jevify-pilot/1, direct-reading
strategy, result schema 6, compare_graded).

  • A method recommendation, including "use direct scripting", is withheld unless all of these
    hold: the arms are comparable, every case has gold from an independent adjudicator with a
    distinct key digest, and each arm meets its own thresholds.
  • Duplicate fixture/key aliases count once.

Host-notice adoption helpers.

  • jv.adopt(items, states, Q, batch=b) and jv.scalar turn built-in judge_batch() output into
    receipt-bound rows.
  • jv.pilot_check returns use_direct when more than about 40% of items are flagged, in the
    uncertainty band or caught by the catch-all.
  • jv.remaining reuses unchanged verdicts, jv.gate checks for regressions, jv.power sizes a
    sample, and jv.Ledger keeps tuned units out of held-out evidence.
  • jv.facts, jv.units_from_table and jv.refuse_elided handle input hygiene, and
    jv.flag(report=) was added. jv.choice enforces the 255-label route limit, and jv.plan
    now reports labels, over_labels and pack_suggestion.
  • jv.splice, jv.comparable, jv.export and jv.receipt_md were added.
  • Rows now carry state_sha256, judged_at and assembly. Receipts now carry served_models,
    comparable, host_counts and billing.

Verification

  • 456 unit tests pass. Each of the four commits passes the suite on its own in a clean worktree.
  • CI passed on main (run 39577134316) and on the tag (run 39577136651).
  • Offline checks made no model requests: structured grading, inbox persistence across
    processes, stale-evidence rejection, replay of retained real judgment rows, and pilot
    withholding.
  • A live smoke on OMP 18.12.5 cost about $0.0002 in total. It ran adopt, run, remaining,
    splice, gate, receipt_md and export.

Boundaries

  • These are workflow checks only. This release makes no claim about quality or savings.
  • Provenance stays unknown. Stock 18.12.5 items expose only key, ok, error, answers
    and model, with no fallback or cache fields. Cached answers report the alias
    ~typesafe/jev-latest, so receipts that mix cached and fresh rows are comparable: false.
  • Billing is an unverified estimate.
  • The repository has one schema-2 answer key, which cannot support a method recommendation.
    Pilot comparisons on the development cases come out withheld or inconclusive.
  • Installing the plugin does not change the host notice.