feat(harvey-lab): switch judge to Claude Sonnet + fix docs + enable CLI progress by devin-ai-integration[bot] · Pull Request #264 · benchflow-ai/benchflow

devin-ai-integration · 2026-05-15T22:23:02Z

Summary

Three changes bundled in this PR:

Switch Harvey LAB verifier judge from Gemini to Claude Sonnet — the original Harvey LAB benchmark uses claude-sonnet-4-6 as the judge model. Our converter was hardcoded to gemini-3.1-flash-lite-preview. This PR updates the converter (benchflow.py) to use claude-sonnet-4-6 via the Anthropic API, matching the original benchmark's judge. Docker dependencies updated from google-genai to anthropic.
Fix docs to use benchflow-ai/benchmarks — docs previously showed harveyai/harvey-labs as the --source-repo. Updated with correct examples using benchflow-ai/benchmarks and datasets/harvey-lab/tasks/..., including a note that task names are flattened with hyphens.
Enable CLI progress output — added logging.basicConfig(level=logging.INFO) to cli/main.py so logger.info() messages from benchflow internals are visible in the terminal during execution.
Fix parity_test.py for eval-parity mode — updated check_eval_parity() to pass ANTHROPIC_API_KEY instead of GEMINI_API_KEY (since the generated evaluate.py now uses Anthropic). Added --anthropic-api-key CLI flag. Side-by-side mode still uses --gemini-api-key as before.

Review & Testing Checklist for Human

Run a single Harvey LAB task end-to-end with ANTHROPIC_API_KEY set and verify the Claude judge scores correctly
Verify parity_test.py --mode eval-parity --anthropic-api-key ... works
Confirm parity_test.py --mode side-by-side --gemini-api-key ... still works (unchanged)

Notes

Pre-converted tasks in benchflow-ai/benchmarks will need regeneration to pick up the new Claude judge in their evaluate.py files.
The GEMINI_API_KEY is still needed for the agent (if using gemini agent) and for side-by-side parity mode. Only the verifier/judge now uses ANTHROPIC_API_KEY.

Link to Devin session: https://app.devin.ai/sessions/6b1fbe50a16b4fd497ed4a26cc1d4b6c
Requested by: @xdotli

… CLI progress output - Update running-benchmarks.md with correct --source-repo benchflow-ai/benchmarks and --source-path datasets/harvey-lab/tasks/... examples - Add note about flattened task names in the benchmarks dataset repo - Configure logging.basicConfig(level=INFO) in CLI so logger.info progress messages from SDK/Job are visible during execution

devin-ai-integration · 2026-05-15T22:23:05Z

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

Disable automatic comment and CI monitoring

devin-ai-integration

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 2 additional findings.

…(matching original) Harvey LAB's original benchmark uses claude-sonnet-4-6 as the default judge model. Our converter was hardcoding gemini-3.1-flash-lite-preview instead. This switches to the original model for accurate parity: - Replace call_gemini() with call_judge() using Anthropic API - Change verifier.env from GEMINI_API_KEY to ANTHROPIC_API_KEY - Update Dockerfile: google-genai -> anthropic pip package - Update README to document Claude-based judge Pre-converted tasks in benchflow-ai/benchmarks will need to be regenerated to pick up this change.

…+ resolve merge conflicts

devin-ai-integration · 2026-05-15T22:55:05Z

Test Results — Claude Judge Switch

Tested by extracting the generated evaluate.py template and running it against the Anthropic API with a synthetic legal rubric. Devin session

Claude Judge E2E (primary test)

Test	Result
Generated `evaluate.py` contains `import anthropic` (not `genai`)	✅ passed
Model is `claude-sonnet-4-6` (not `gemini-3.1-flash-lite-preview`)	✅ passed
Function named `call_judge` (not `call_gemini`)	✅ passed
Zero references to `genai` or `GEMINI` in generated code	✅ passed
evaluate.py runs successfully with Anthropic API (exit code 0)	✅ passed
reward.txt = `1.0` (valid float 0.0–1.0)	✅ passed
Per-criterion verdicts printed: `criterion-1: PASS`, `criterion-2: PASS`	✅ passed

Found 1 deliverable(s): ['analysis.md']
Evaluating 2 criteria...
  [1/2] criterion-1: Identifies the key legal issue...
    -> PASS: The agent's output clearly identifies multiple legal issues related to contract
  [2/2] criterion-2: Provides a recommendation...
    -> PASS: The output includes a clear 'Recommendation' section with three specific actiona

Score: 2/2 (100.0%)

Template verification

Test	Result
`task.toml` has `ANTHROPIC_API_KEY = "${ANTHROPIC_API_KEY}"` in `[verifier.env]`	✅ passed
Dockerfile `pip install` includes `anthropic` (not `google-genai`)	✅ passed
`parity_test.py --help` shows `--anthropic-api-key` (eval-parity)	✅ passed
`parity_test.py --help` still shows `--gemini-api-key` (side-by-side)	✅ passed

CLI logging fix

Test	Result
`bench eval create` shows progress messages (Cloning, Starting environment, etc.)	✅ passed

Cloning benchflow-ai/benchmarks from https://github.com/benchflow-ai/benchmarks.git ...
Starting environment: corporate-ma-analyze-cim-deal-teaser-scenario-01
Installing gemini in sandbox (timeout=900s)...

Note: Pre-converted tasks in benchflow-ai/benchmarks still have the old Gemini evaluate.py and will need regeneration.

devin-ai-integration · 2026-05-15T23:04:26Z

Dogfood: Full E2E Run with Claude Judge

Patched a pre-converted task (corporate-ma-analyze-cim-deal-teaser-scenario-01) with the new Claude evaluate.py + ANTHROPIC_API_KEY in task.toml + anthropic in Dockerfile, then ran the full pipeline:

bench eval create --source-repo benchflow-ai/benchmarks \
  --source-path datasets/harvey-lab/tasks/corporate-ma-analyze-cim-deal-teaser-scenario-01 \
  -a gemini -m gemini-3.1-flash-lite-preview -e docker

Result:

Rewards: {'reward': 0.10256410256410256}

Task: corporate-ma-analyze-cim-deal-teaser-scenario-01
Agent: gemini (gemini-3.1-flash-lite-preview)
Reward: 0.10256410256410256
Tool calls: 10

The full pipeline works: agent → 10 tool calls → Claude judge evaluated 39 criteria → reward 0.103. No crashes, no GEMINI_API_KEY errors.

devin-ai-integration

Devin Review found 1 new potential issue.

View 7 additional findings in Devin Review.

devin-ai-integration · 2026-05-15T23:17:30Z

+Reward = (criteria passed) / (total criteria). Same per-criterion individual grading and
+default judge model as Harvey LAB's original `evaluation/judge.py`.


🟡 README falsely claims same judge model as Harvey LAB's original when it was changed from Gemini to Claude

The new README text states "Same per-criterion individual grading and default judge model as Harvey LAB's original evaluation/judge.py." However, this PR changes the judge from gemini-3.1-flash-lite-preview to claude-sonnet-4-6. The old README correctly said only "Same per-criterion individual grading as Harvey LAB's original" (without the model claim). All parity data in the repo — end-to-end parity (benchmarks/harvey-lab/README.md:94-99), prompt-level parity (benchmarks/harvey-lab/README.md:118-122), benchmark.yaml:35, and parity_experiment.json — consistently show the original judge model was Gemini, not Claude. The added clause "and default judge model" is factually incorrect and misleading.

Suggested change

Reward = (criteria passed) / (total criteria). Same per-criterion individual grading and

default judge model as Harvey LAB's original `evaluation/judge.py`.

Reward = (criteria passed) / (total criteria). Same per-criterion individual grading

as Harvey LAB's original `evaluation/judge.py`.

Was this helpful? React with 👍 or 👎 to provide feedback.

The claim is actually correct — Harvey LAB's original benchmark uses claude-sonnet-4-6 as the default judge model (per their eval-strategies.md). This PR switches from our previous gemini-3.1-flash-lite-preview to match the original.

The parity data in the repo was recorded with the old Gemini judge (before this PR), which is why it shows Gemini — that's an artifact of when parity was run, not the original benchmark's choice. The "default judge model" clause refers to matching Harvey LAB upstream, not our previous adapter.

That said, good catch that the parity data is now stale relative to the judge model. The parity experiment validated prompt equivalence (same prompts produce same verdicts), not judge-model equivalence. I'll add a note clarifying this in the README.

devin-ai-integration Bot assigned xdotli May 15, 2026

devin-ai-integration Bot commented May 15, 2026

View reviewed changes

devin-ai-integration Bot changed the title ~~docs: fix Harvey LAB examples to use benchflow-ai/benchmarks + enable CLI progress output~~ feat(harvey-lab): switch judge to Claude Sonnet + fix docs + enable CLI progress May 15, 2026

This comment was marked as resolved.

Sign in to view

fix: update parity_test.py eval-parity mode to use ANTHROPIC_API_KEY …

ccffb83

…+ resolve merge conflicts

devin-ai-integration Bot mentioned this pull request May 15, 2026

feat: Support LLM-as-judge verification as a first-class task verification method #270

Closed

docs: use backslash line continuations for long CLI commands

16076dd

devin-ai-integration Bot commented May 15, 2026

View reviewed changes

docs: clarify parity data was recorded with previous Gemini judge

161621e

devin-ai-integration Bot mentioned this pull request May 15, 2026

feat(harvey-lab): switch judge from Gemini to Claude Sonnet benchflow-ai/benchmarks#3

Merged

xdotli merged commit 37073f2 into main May 16, 2026
3 checks passed

xdotli deleted the devin/1778883688-docs-harvey-lab-fix branch May 17, 2026 05:51

bingran-you mentioned this pull request Jul 22, 2026

feat(sandbox): Enforce network_mode allowlist on docker #785

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(harvey-lab): switch judge to Claude Sonnet + fix docs + enable CLI progress#264

feat(harvey-lab): switch judge to Claude Sonnet + fix docs + enable CLI progress#264
xdotli merged 5 commits into
mainfrom
devin/1778883688-docs-harvey-lab-fix

devin-ai-integration Bot commented May 15, 2026 •

edited

Loading

Uh oh!

devin-ai-integration Bot commented May 15, 2026

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration Bot commented May 15, 2026

Uh oh!

devin-ai-integration Bot commented May 15, 2026

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

devin-ai-integration Bot May 15, 2026

Uh oh!

devin-ai-integration Bot May 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		Reward = (criteria passed) / (total criteria). Same per-criterion individual grading and
		default judge model as Harvey LAB's original `evaluation/judge.py`.

Uh oh!

Conversation

devin-ai-integration Bot commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Review & Testing Checklist for Human

Notes

Uh oh!

devin-ai-integration Bot commented May 15, 2026

🤖 Devin AI Engineer

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

✅ Devin Review: No Issues Found

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration Bot commented May 15, 2026

Test Results — Claude Judge Switch

Uh oh!

devin-ai-integration Bot commented May 15, 2026

Dogfood: Full E2E Run with Claude Judge

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot May 15, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot May 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

devin-ai-integration Bot commented May 15, 2026 •

edited

Loading