Multilingual numeric evaluator without a judge model (Custom Code Evaluator, standard library only)

[mouadhassal]

Idea: a built-in (or template) Custom Code Evaluator that grades numeric answers in any language without a judge model.

Why: in a pre-registered study of LLM-as-a-judge on 1,008 numeric answers in 7 languages (judge-drift), qwen3:8b agreed with the reference 100% in English but 94.4% in Estonian. It read "üle 900" ("over 900") as a different value from 900, while accepting "over N" in every English case. gemma3:4b agreed only 69–85% in every language, mostly by rejecting correct answers for "extra information". A number check behaves the same in every language.

What I built: numguard, one file that implements evaluate(inputs, outputs, trace) -> dict, standard library only, so it should also work with AGENTA_SERVICES_CODE_SANDBOX_RUNNER=restricted. It handles Arabic-Indic digits, decimal commas, "1 100" / "1.100,50", 12-hour times in English and Arabic, and stray separator bytes. It returns answer_correct, language_ok, invented_numbers and score as separate columns.

On the same 1,008 replies: numguard 99.4% agreement with the audited labels, qwen3:8b judge 97.7%, gemma3:4b judge 75.3%. (Caveat, stated in the README: numguard's time and separator handling was written after reading the audit, so the honest claim is that it fixed the 5 format errors the audit found and changed no other label.)

What code can't do: catch a wrong claim around a right number ("saves 10%" when the site says "pay for 10 months"). So the setup I'd suggest is numguard and an LLM judge on the same evaluation, then reviewing the rows where they disagree: 35 of 1,008 in my data, which surfaced 11 errors of the code check and 23 of the judge. A "disagreement view" between two evaluators would make that a one-click review.

Honest limitation: I tested it with a local harness that calls evaluate() the way an evaluation run does, plus a 168-row test set (24 questions × 7 languages) in the repo. I haven't run it inside a live Agenta sandbox yet. If someone can confirm it works there, or if you'd like it as an example in examples/, I'm happy to open a PR.