Jev-as-a-Judge with Comet Opik

A small, auditable example of evaluating refund-support traces with Jev and recording the results as an Opik experiment.

• Replays ten synthetic, frozen support traces, including correct answers, invented policies, false action claims, ambiguous wording, and an evaluator-injection attempt.
• Runs deterministic checks before paying for semantic evaluation.
• Evaluates three yes/no criteria and one ordered helpfulness rubric in a single Jev request.
• Keeps raw model answers, the resolved model identifier, and a rubric version for auditability.
• Maps values to separate Opik metrics through a custom BaseMetric.
• Provides a fully offline demo, mocked transport tests, and real-SDK adapter tests.

This is an evaluator tutorial, not a native Jev plugin or production guardrail. It does not issue refunds. Review outcomes are recorded; automated escalation and a background worker are not implemented.

From this folder, with Python 3.10 or later:

python -m jev_judge.cli --mode demo
python -m unittest discover -s tests -v

No dependencies or network calls are needed for the core demo. The optional Opik tests are skipped if the SDK is absent. Demo probabilities are hand-authored and explicitly labeled DEMO-hand-authored-not-Jev. They test plumbing, not model accuracy. Do not use demo timings as inference latency.

Obtain a TypeSafe API key and export it in your shell:

export TYPESAFE_API_KEY="your-key"
export JEV_MODEL="jev-latest"
python -m jev_judge.cli --mode live --output results/live.jsonl

This sends the synthetic request, policy, tool results, and final answer to TypeSafe and may incur API charges. The sample makes nine initial judge requests: the empty-answer case fails a deterministic check. Retries can increase charges. It does not log to Opik.

jev-latest can change. The resolved response model is recorded. Use a supported pinned model identifier for longitudinal comparisons. Never turn an API error into a quality score; failures are marked judge_error, routed to review, and produce a nonzero CLI exit code.

python -m venv .venv
source .venv/bin/activate
pip install -e '.[opik]'
opik configure
export TYPESAFE_API_KEY="your-key"
python -m jev_judge.opik_eval --project jev-support-judge

This separately calls Jev again and creates a new dataset and experiment in your configured Opik workspace. Each invocation uses a unique name to avoid silently mixing fixture revisions. Choose Opik Cloud or self-hosting during configuration. No external account was modified while building this project.

The adapter was tested with Opik 2.2.71 and Python 3.12 using a fake Jev transport. Authenticated Jev calls and actual Opik uploads were not run. The public API contracts were checked against official documentation on September 20, 2026. Dependency versions do not freeze server behavior.

OPIK_TRACK_DISABLE=true is used only in tests. Do not set it for a real experiment. .env.example documents the variables; .env is not automatically loaded.

A Noul value of 0.98 is not “98% of claims are grounded.” It is the model's probability of a yes to the whole proposition. Score.confidence is not the probability that the judge is correct. The tutorial's 0.2/0.8 boundaries and 0.6 confidence threshold are illustrative and uncalibrated.

Pass --data path/to/cases.jsonl to either runner. Each line needs:

{
  "id": "case-001",
  "request": "Can I return this order?",
  "policy": "The authoritative policy relevant to this request.",
  "tool_calls": [{"name": "lookup_order", "result": {"eligible": true}}],
  "final_answer": "The answer being evaluated."
}

The demo mode additionally requires hand-authored demo fields, as in the supplied fixtures. Live mode never sends these to Jev. Optional expected_verdict fields are tutorial annotations, not independently collected human labels; they are excluded from the judge state. Collect real labels before reporting accuracy.

Redact personal data before submission. Restrict Opik retention and access according to your needs. The instruction boundary in the rubric reduces ambiguity but is not a proven defense against prompt injection. Test hostile trace content separately.

• article.md: publication draft, with source links and code excerpts.
• jev_judge/rubric.py: atomic questions and allowlisted state projection.
• jev_judge/client.py: documented HTTP contract, validation, timeout, bounded retries.
• jev_judge/core.py: deterministic checks, score mapping, verdict policy.
• jev_judge/cli.py: local demo/live runner and JSONL audit results.
• jev_judge/opik_eval.py: live custom metric and experiment entry point.
• tests/: network-free tests, including the real Opik SDK with mocked calls.
• scripts/build_figures.py: reproducible architecture exports.
• scripts/build_reading_copy.py: standalone HTML article builder.
• VALIDATION.md: verification scope and publication checks.

pip install -e '.[figures]'
python scripts/build_figures.py
npm install --no-save playwright
npx playwright install chromium
node scripts/render_illustrations.cjs
python scripts/build_reading_copy.py

Alternatively, install weasyprint and Poppler (pdftoppm), then run python scripts/render_illustrations.py in place of the Playwright steps. The four illustrations are editable HTML/CSS in assets/; their PNG exports support the Markdown article.

Review the article, choose your repository URL, and choose a license before publishing. No license is imposed here. Commit the source, data, tests, article, and figures; do not commit .env, credentials, environments, or private result files. The included CI workflow runs only offline tests.

• TypeSafe API
• TypeSafe Noul
• TypeSafe Score
• TypeSafe confidence
• Opik custom metrics
• Opik evaluation SDK
• Comet's judge guide