Should OpenViking integrate Jev? Use cases, expected benefits, and tradeoffs

[yufeng201]

Background

This is an architectural discussion based on OpenViking source and TypeSafe documentation. The scenarios and benefits below are proposals; no OpenViking/Jev integration results are available yet.

Why consider Jev?

Is it worth integrating Jev into OpenViking? Which scenarios would benefit, and would those benefits justify the integration and maintenance cost?

OpenViking manages resources, memories, and skills for agents. Across their lifecycle, it needs semantic decisions: what context helps with this request, which skill applies, whether a candidate memory is worth retaining, and how new information relates to existing memory.

Jev is a possible component for these decisions. It returns a selection (Choice), a probability that a condition holds (Noul), or a graded judgment (Score). The architectural opportunity is to let a specialized decision model handle suitable bounded judgments while OpenViking retains retrieval, workflow, and storage control and generative models continue producing queries, summaries, and memory text.

The potential payoff is better context and memory quality, less unnecessary downstream work, or cheaper/faster semantic decisions. None of these is automatic: OpenViking already has relevant mechanisms, and another model call may add cost without improving outcomes. This discussion asks where the combination makes sense, rather than assuming Jev should become a core dependency or treating reranking as the only possible use case.

Possible applications and benefits

Scenarios worth discussing

Scenario	Possible Jev responsibility	Expected benefit if effective	What determines whether it is worthwhile
Context selection and ranking	Judge each retrieved candidate against the request: relevance, useful constraints, or direct evidence. Use independent judgments when several sources apply.	More useful evidence within the context budget; potentially fewer distractions and less downstream input.	Must improve on existing reranking/assembly/rewrite. False negatives can remove essential evidence. Fewer candidates do not necessarily reduce tokens if the budget is reused for deeper content.
Skill matching	Given a retrieved skill shortlist and task context, judge which skills actually apply; support no match and multiple applicable skills.	Fewer irrelevant skill descriptions loaded and fewer unsuitable recommendations to the agent.	Descriptions must contain enough evidence. OpenViking can recommend; the consuming agent retains execution and authorization decisions.
Memory admission	Check already-extracted candidate memories against explicit retention criteria: for example, an enduring preference versus a temporary instruction.	Less low-value persistent memory, indexing work, and future recall noise.	Must preserve useful one-off events as well as durable preferences; criteria depend on memory type. Post-extraction filtering does not save the extraction call and may add a new cost.
Memory relationship classification	Compare a new candidate with retrieved existing memories: duplicate, complementary, contradictory, unrelated, or insufficient evidence.	Better signals for avoiding duplicate writes and choosing when an update needs further processing.	Must account for identity, scope, time, and provenance. Classification does not generate merged text, and a contradiction does not authorize overwriting/deleting a memory. Savings require actually avoiding downstream work.
Retrieval-scope routing	Judge whether the request needs resources, memories, skills, or several of these scopes.	Potentially avoid unnecessary searches and candidate processing.	The decision must cost less than the work it avoids. Incorrect routing loses recall; ambiguous cases need broad retrieval. It cannot replace query generation or decomposition.

A Jev rerank provider is one possible implementation of the first scenario, not the whole proposal. Likewise, a single large Choice should not substitute for independent relevance judgments when multiple memories or skills may be useful.

Where the value would come from

There are two different integration cases:

• Replacing an existing model judgment: worthwhile if Jev preserves quality while reducing latency/cost, or improves quality within the same budget. The incumbent may be a dedicated reranker or a generative-model step that only needs to return a decision. If judgment and text generation are currently combined, splitting them does not necessarily save a call.
• Adding a new judgment: worthwhile only if it prevents enough bad context, unnecessary processing, or low-value memory writes to outweigh its own latency, cost, and error risk. This needs an end-to-end benefit, not just a plausible classifier output.

Structured answers can make outputs easier to consume and individual decisions easier to inspect. That is useful engineering behavior, but it does not by itself establish accuracy or distinguish Jev from every existing structured-output alternative.

Fit with the existing architecture

Source references are pinned to a55afb001bca201f09830b90dc2f64eed20cb799:

• Hierarchical retrieval already has a rerank boundary and fallback. Pure scoring should build on that boundary where appropriate.
• Context assembly already gathers candidates, allocates a token budget, and optionally rewrites context. A separate selector needs a distinct purpose; filtering after retrieval cannot save work already performed or recover excluded candidates.
• Intent analysis produces query text as well as categorical fields. Jev could assist bounded routing, but is not a replacement for the whole planner.
• Memory extraction/update orchestration generates content and operations. Admission and relationship judgments would require an explicit stage boundary; they are not drop-in replacements for this loop.

If we proceed, integration should be optional, reuse existing boundaries where possible, and preserve baseline behavior on provider failure. Provider availability, data-handling requirements, deployment compatibility, and ongoing model/prompt calibration are part of the cost. A general decision abstraction should follow demonstrated needs rather than precede them.

Initial assessment

There is a reasonable case for exploring Jev as an optional component, but insufficient evidence to recommend a default or broad core integration.

Context selection and skill matching are approachable starting points because they operate on observable candidates and their outputs can be compared with existing behavior. Memory admission and relationship classification could offer longer-term memory-quality benefits, but errors persist and their interaction with generation makes the payoff harder to establish. Routing is attractive only where there is meaningful unnecessary retrieval to avoid.

These are prioritization suggestions for discussion, not a decision to limit the work to reranking. Maintainers' experience with actual bottlenecks should determine the first scenario.

Alternatives Considered

Existing rerankers, tuned retrieval/assembly policies, deterministic rules, and general-purpose models remain valid alternatives. If they deliver the same benefit with less complexity, there is no reason to add Jev. Jev is also not a replacement for embeddings, open-ended reasoning, query rewriting, summary generation, or memory-text synthesis.

Use Case

For deployment assistance, the desired result may be a relevant guide, a prior failure containing an environment constraint, and an applicable deployment skill. The value of Jev would be helping OpenViking supply this useful combination—or maintain cleaner memories for later requests—with a better quality/cost tradeoff. Simply returning valid selections is not the outcome we care about.

Additional Context

Questions for core developers:

1. Which of these semantic decisions is a real quality or cost problem in current OpenViking workloads?
2. Where would Jev replace existing work, and where would it add a genuinely useful capability?
3. Which scenario offers the clearest expected benefit: context quality, skill matching, memory quality, or avoided processing?
4. Should the relevant responsibility live in an existing model adapter, a memory-processing stage, or a consuming agent/plugin?
5. What evidence would make the integration worth maintaining, and what would make us reject it?

After choosing a scenario, a small representative replay/shadow comparison against the existing approach can validate the hypothesis. Measure task-specific quality and consequential errors together with total latency/cost; for context-related changes, include final answer quality and actual injected tokens. Keep tuning and held-out samples separate. Detailed evaluation design can follow agreement on the use case.

Related issues: #2880 (rerank budgets) and #4948 (closed historical latency report), relevant to the retrieval scenario but not evidence that Jev resolves it.

References: TypeSafe primitives, reranking example, skill suggestion example, confidence semantics. These document feasibility patterns, not measured OpenViking benefits.

Originally proposed in #5210; continuing here because the integration scope and value are still open for discussion.

[ZaynJarvis]

throw my existing thought first

1. Jev probably brings a standardized decision API interface. The underlining service, can be Jev or any Jev-like fine tuned model, or even just llm which prompted to produce or structured in that response format
2. Jev is good at parallel decision
3. Typesafe.ai requires waitlisting. Vercel ai can try the model now
4. Agent plugins deciding recalls compaction, ai colleague deciding message to include in context. May include Jev.
5. some guys tried Jev for a new compaction. Effect is controversial.

[MaojiaSheng]

Will be officially integrated, but maybe, with an open-source interface standard.

[MaojiaSheng]

#5247

[ByteDanceLiuYang]

Another promising use case is letting Jev choose the retrieval strategy for each query : semantic, keywords, or hybrid, and whether reranking is needed. We can map its decision to a few pre-calibrated weight profiles. This could sit between query planning and retrieval as an optional policy layer.

[t0saki]

Three places where a bounded decision model has the clearest leverage, and all of them sit at the recall boundary rather than inside ranking:

1. At user-input time: decide whether to recall at all.

As far as I can tell from the code paths cited above, retrieval runs per turn rather than being gated on whether the turn needs memory at all. A large share of turns in a long session — "run the tests", "continue", a follow-up on a tool result — reference nothing in storage. A single Noul over the incoming message ("does answering this require anything the agent does not already hold in context?") runs before embedding and search. This makes it the cheapest decision in the pipeline and the only one that can eliminate the entire retrieval cost instead of trimming its tail. The errors are asymmetric: a false negative silently drops evidence the agent needed, so the gate should fall back to retrieving whenever confidence is low.

2. After recall: decide per memory whether it is actually needed.

Ranking produces an ordering; it cannot say "none of these belong in the prompt". A per-candidate judgment ("is this memory required to answer this request?") fits the independent, parallel shape Jev is suited to, avoiding the failure mode of a single large Choice forced to pick something. The output doubles as a budget policy: keep what is judged necessary and drop the rest to free space, rather than filling the window simply because the capacity exists.

3. Query rewriting: decide whether the input needs rewriting at all.

Rewriting is enabled by default in the plugin today, so every turn pays the cost whether or not the input is under-specified. Many inputs are already self-contained; those that are not typically depend on a referent from earlier in the thread. A Noul check on "is this input under-specified for retrieval without prior context?" turns rewriting from a fixed cost into a conditional one. Unlike the gate in (1), a wrong answer here is cheap: the fallback is retrieving with the raw query, which matches the behavior before rewriting existed.

Context width is the accuracy knob, and it is priced.

All three decisions improve significantly with more agent context: the last few turns, the active task, and what the agent already has loaded. Because these are bounded decisions rather than generation, that context can be scaled per call — a one-line summary when downstream work is cheap, or the recent transcript when downstream work involves an expensive generation or a memory write. It is worth treating context width as an explicit cost/accuracy parameter of the integration rather than a fixed prompt.

None of this is measured yet. It is simply where I would look first, because all three decisions are observable: you can replay a session, compare what the gate refused against what retrieval would have returned, and count the turns where it was right.

[Jevals]

On point 1 (Jev vs an LLM prompted into the same response format): we measured that trade-off against human labels. I maintain Jevals, an independent benchmark.

• Choice, Banking77 (77 intents, close to skill matching): Jev 79.7% accuracy, $0.043 per 1,000 decisions, p95 0.69 s. Best prompted LLM (Gemini 3.8 Flash): 84.6%, $1.37, p95 3.9 s.
• Noul, PubMedQA (a yes/no gate, like the recall check above): Jev 91.3% vs 92.5%, at 1/28 of the cost.
• With a confidence gate, Jev decided 59% of Choice items alone at 94.3% accuracy (gate 0.96) and 49% of Noul items at 98.6% (gate 0.91). The rest would fall back, which fits the asymmetric-error design above.

Boards: https://jevals.com/choice/