Jev as a judge for agent evals

Agent evaluators usually fall into two categories: deterministic code and LLM-as-a-judge. Code is fast and reliable but limited to behavior that can be expressed as explicit logic. LLM judges can evaluate open-ended agent behavior, but they add cost, latency, and variance.

This project tests a third option: Jev, TypeSafe AI's System One decision model. We compare Jev with GPT-5.6 Luna, GPT-5.6 Terra, and Claude Sonnet 4.6 on the same fixed agent runs, measuring binary accuracy, score reliability, cost, and latency.

Jev is not an autoregressive LLM and does not generate text. It evaluates typed questions against structured state and returns typed answers with probabilities.

Jev supports three question types:

Multiple atomic questions can be evaluated in parallel against the same state.

Agent evaluation is a decision task: given an agent's state and behavior, assign a score that provides feedback. Jev is designed for this pattern. It evaluates typed questions against structured state and returns typed answers with probabilities. Autoregressive models, on the other hand, reach a judgment through token-by-token generation. In our experiment, that decision-first design coincided with lower latency, lower cost, and lower variance.

That does not make any judge correct by default. A repeatable evaluator can still be consistently wrong, so we compare each judge with human labels and keep accuracy separate from reliability.

We built a weather agent with Deep Agents and gave it access to Tavily web search. We defined five cases in a LangSmith dataset:

We ran the weather agent once for each case and stored its complete output, including the final answer, evidence, tool calls, and expected behavior. Freezing the agent runs meant that only the judges could introduce variation between repetitions.

Each judge evaluated the five captured runs 100 times with two signals:

A human reviewer labeled each fixed response against the same rubric. We use those labels as the oracle for binary accuracy. We use the continuous quality score only to measure reliability through variance, not accuracy.

Accuracy measures whether a judge's binary does_pass decision agrees with the human oracle. Across 500 repeated decisions per judge, Jev matched every human pass-or-fail label.

This is a small corpus with five agent runs and one human reviewer. The result describes this experiment; it is not a general ranking of judge accuracy.

Reliability asks whether a judge produces the same score when the agent behavior is unchanged. We measured it using the observed variance of each judge's continuous quality scores. Lower variance is better.

Jev had the lowest observed mean per-case variance: 0.0000149. Luna was 433× higher, Terra was 913× higher, and Claude was 92× higher.

This experiment cannot establish why Jev varied less. One hypothesis is that a model designed to return bounded decisions is a better fit for this task than a model designed for autoregressive generation. The result is observational, not evidence that the model architecture caused the lower variance.

At $0.00035 per call, Jev makes repeated judgments and frequent regression checks inexpensive. These costs depend on the prompts, inputs, and provider pricing at the time of the run.

The important result is not simply a lower evaluation bill. When high-quality judgment becomes cheap enough to use broadly, builders can evaluate more agent runs, test more dimensions, and measure more changes without narrowing the feedback loop around cost.

That can speed up the entire agent development lifecycle. Agent engineers can turn more traces into feedback, catch regressions sooner, and move faster as they build, test, monitor, and deploy agents. Low cost can also amplify mistakes, so human review, representative datasets, and judge alignment still matter.

Requires Python 3.13+, a Tavily API key, a TypeSafe API key, and a workspace-scoped LangSmith API key with Gateway access.

cp .env.example .env
# Add your API keys to .env
uv sync

Run the local evaluation without uploading an experiment:

uv run python main.py

Upload the dataset and record an evaluation experiment in LangSmith:

uv run python src/evals/offline_evals.py

Reproduce the repeated-judge benchmark:

uv run python src/evals/judge_reliability.py

Use --local with the benchmark command to run without uploading an experiment.

The published LangSmith experiment is benchmark-jev-luna-terra-sonnet (6d08df72-c878-458c-b7c5-a7824ee6e721), started at 2026-09-18T17:53:25Z.

• Frozen cases and variance analysis
• Human oracle labels
• Generated result assets

The LLM judges ran through LangSmith Gateway with openai/gpt-5.6-luna, openai/gpt-5.6-terra, and anthropic/claude-sonnet-4-6. Jev ran through langchain-typesafe==0.0.1a2. The experiment used deepagents==0.7.15, langchain-openai==1.6.2, langsmith==0.12.6, and tavily-python==0.8.3.

We did not set temperature, top-p, seed, or max tokens for the LLM judges, so provider and gateway defaults applied. The experiment metadata did not expose the hosted Jev service version.

• src/weather_agent/ — Deep Agents weather agent and Tavily tool
• src/evals/dataset.py — LangSmith dataset definition
• src/evals/judges/ — Jev and LLM evaluators
• src/evals/offline_evals.py — Dataset evaluation runner
• src/evals/judge_reliability.py — Repeated-judge benchmark
• analysis/ — Accuracy, variance, cost, and latency analysis
• assets/ — Archived benchmark data and generated charts
• tests/ — Focused analysis tests

• TypeSafe introduction
• TypeSafe primitives
• LangSmith evaluation quickstart
• LangSmith LLM-as-a-judge
• Deep Agents quickstart