Repository navigation
Using JEV models for OpenSRE #6332
Replies: 1 comment
|
Thanks for starting this. I missed this discussion and opened #6475, so I am closing mine to keep the Jev evaluation in one place. I agree that checklist validation is the strongest first spike. The existing validator already makes independent per-item I suggest evaluating it in opt-in shadow mode first:
@IRONICBo also made a useful point on #6475: evaluate whether a decision improves the actual outcome, not only whether it matches a label. Here that means measuring premature completion and unnecessary extra loops as well as verdict agreement. If maintainers agree with this shape, I can turn it into a small benchmark/spike with explicit acceptance criteria and report a ship / revise / do-not-ship result back here. |
Uh oh!
There was an error while loading. Please reload this page.
A week ago a new model (JEV model) was announced: https://typesafe.ai/blog/introducing-system-one-models-and-jev
JEV is a typed-decision model family (state + questions → probabilities, no generation)
Jev (TypeSafe’s System One model, announced 15 Sep 2026) takes one state plus typed questions and returns probabilities. It never generates text. Question types are yes/no (noul), unordered choice (up to 255 options), and ordered score (2–10).
Where we can use it - this needs some investigation and experiments
Checklist tick validator (session_goal/validate.py) — best first spike. Same transcript, many independent VALID / INVALID questions. That is Jev’s native shape, and it looks like BoolQ/NLI, where even the open clones score decently.
ReAct goal review (goal_review.py → invoke_closed_goal_verdict) — already a closed GOAL_REACHED / NOT_REACHED Literal, opt-in, fail-open on parse errors. Jev would remove the parse-error class and add a probability the host can threshold.
Confidence gate — accept GOAL_REACHED only above a probability; otherwise continue the SessionGoal loop. That is a better uncertainty policy than “LLM JSON failed, so treat as not reached / fail open.”
All reactions