Judge may be incorrectly penalizing tool use in meeting tasks

[kxfan2002]

Hi, thanks for maintaining PinchBench. I noticed a possible issue with the judge prompt for meeting tasks.

In my run, many meeting tasks received judge comments such as:

• used prohibited tools
• violated instructions by using tools
• unnecessary tool usage

However, the task_meeting_*.md prompts seem to mainly ask the agent to read the transcript and write the requested output. I did not see instructions that prohibit the tested agent from using tools.

My guess is that the judge may be confusing this judge-side instruction:

Do NOT use any tools

as a restriction on the evaluated agent, rather than only on the judge itself.

Could you clarify the judge prompt so that tool-use restrictions apply only to the judge, unless the task prompt explicitly forbids tool use?
This seems important because normal file read/write tool use is expected for these meeting tasks, so penalizing it may affect the validity of the scores.

Thanks!

[JiMI-Ha]

I ran into the same issue locally.

In my run, several llm_judge tasks got hard 0.0 because the judge notes said the agent
“violated CRITICAL RULE: Do NOT use any tools / respond with ONLY JSON”, simply because
the agent transcript contained tool calls like apply_patch/todowrite. But those tasks’
prompts did not forbid tool use, so this looks like the judge is mistakenly applying the
grader-only constraints to the agent being graded.

Root cause seems to be the judge prompt wording in scripts/
lib_grading.py::_build_judge_prompt():

• It currently says “CRITICAL RULES: Do NOT use any tools … Respond with ONLY a JSON
  object…”
• Some judge models interpret that as constraints on the agent under evaluation and zero
  out when they see tool calls in the transcript summary.

Minimal fix I applied: clarify the section header as “FOR YOU, THE GRADER — NOT THE AGENT
BEING GRADED” and add an explicit sentence like:

The agent being graded may have used tools during task execution. Do NOT treat tool
usage as a rule violation unless the task prompt explicitly forbids tools.

After that, the “tool-use => auto-fail” behavior stopped .

[ScuttleBot]

👋 Hi @kxfan2002! I'm @olearycrew's OpenClaw bot, helping triage this repo.

This is a confirmed bug — and @JiMI-Ha's independent reproduction + root cause analysis nails it. The judge prompt in scripts/lib_grading.py::_build_judge_prompt() says "CRITICAL RULES: Do NOT use any tools" as an instruction to the grader LLM, but some judge models interpret it as a constraint on the agent being evaluated and auto-fail when they see tool calls in the transcript.

The fix is straightforward: disambiguate the judge prompt so the "no tools" instruction is clearly scoped to the grader, not the agent. I'm opening a PR now.

Feel free to ignore if this isn't helpful!