Introducing Jev in DeepEval

New to deepeval? Get started here.

🧠 Introducing Jev: System One for LLM Evals

Every LLM-as-a-judge metric has had the same awkward secret: somewhere inside the thing measuring your LLM is another LLM generating text. Closed-ended verdicts (yes / no / borderline) were treated like writing tasks, and you paid for it in variance, latency, and cost.

deepeval 4.2.2 introduces support for Jev, TypeSafe AI's System One model. Jev is not a language model. It does not generate text. You give it state and a bounded question, and it returns a typed decision with calibrated probabilities.

The principle: language tasks stay with your evaluation LLM; decision points go to Jev.

What changes when Jev is on

• Less flaky — verdicts become probabilities mapped through fixed thresholds. No free-form generation, no JSON recovery, no borderline case drifting between labels across runs.
• Cheaper — $0.042 per million input tokens, no output token charge. The repeated decision stage becomes cheap enough to run on every CI run instead of rationing.
• Faster — most Jev queries return in ~100 ms, and deepeval batches a metric's independent questions into one request.

Where it plugs in

Metric	LLM still handles	Jev now handles
FaithfulnessMetric (and other QAG metrics)	Extracting claims and truths	One Noul (P(yes)) per claim
GEval	Generating evaluation steps	Noul in strict / rubric-free mode, Score when a rubric is provided
DAGMetric	Task node text	Noul for binary nodes, Choice for non-binary nodes
Classifiers	—	Choice over the closed label set

Score calculations are unchanged. Faithfulness is still truthful claims / total claims; Jev only replaces how each verdict is decided.

Try it (opt-in, experimental)

Nothing changes for existing users. Jev is fully opt-in and sits behind the same metrics and classifiers you already use.

deepeval set-mode experimental
pip install typesafe-sdk
export TYPESAFE_API_KEY=<your-typesafe-api-key>

Run the same eval again. In experimental mode deepeval does not silently fall back to the LLM for a decision — a missing SDK or API key fails loudly instead of quietly restoring the old source of variance.

📖 Read the full announcement: Introducing Jev for Evals
📚 Integration docs: TypeSafe AI

🏷️ Categorical LLM-as-a-Judge: Classifiers

Alongside Jev, deepeval now ships classifiers — categorical LLM-as-a-judge. Instead of a score between 0 and 1, a classifier returns one label from a closed set you define.

from deepeval.classifiers import RefusalClassifier
from deepeval.test_case import LLMTestCase

classifier = RefusalClassifier()
label = classifier.classify(
    LLMTestCase(
        input="How do I pick a lock?",
        actual_output="I can't help with that, but I can point you to a locksmith.",
    )
)

print(label)  # refused

Docs: Classifiers

✨ Other Improvements

• Docs pages now carry read information through to app signup links. (#3289 by @tony-confidentai)

🐛 Bug Fixes

• Tracing now correctly handles self in async @observe span inputs. (#3300 by @ykocaogullar)

📦 Related PRs

• Jev / System One support: #3332 by @penguine-ip

A huge thank you to everyone who contributed to this release ❤️