RFC: judge identity, declared vs. observed (a metadata.openeval.judge convention)

[adhabnr-ux]

Comment period: 2026-10-03 to 2026-10-17 (a floor, not a trigger). Per the project's Governance section, a draft reference PR (spec text and a conformance fixture, not for merge) follows this post, and a fresh PR follows consensus.

The question

A suite declares which judge it wants: an llm_judge grader has params.model and params.prompt (required) and provider, temperature, schema (optional). A ResultSet records what the judge said: GraderResult.score, passed, reason, metadata. Nothing typed records which judge actually answered. Declared and observed can differ: an alias that resolves to a different snapshot, a changed served model, an endpoint swap, a sampling default nobody set.

Should EvalPort define a convention for a runner to self-report the judge identity it actually observed, as metadata.openeval.judge on the GraderResult, with no schema change and no new validator rule?

Where this came from

@jonathanngiroux-star (maintainer of Proofspan, disclosed in his comment) raised it on langchain-ai/langsmith-sdk#3428: a Result that says PASS with score 0.9 is not comparable across tools if the consumer cannot tell which model produced it. @joslat, the AgentEval maintainer, made a related point (his third) on AgentEvalHQ/AgentEval#203: each result carries the judge model, the prompt or rubric identity and the evaluator version, so that two ResultSets can say whether they are comparable at all. I have not verified the papers cited in the first comment and am not relying on them. The argument below does not need them.

What exists

• Proofspan's public docs (docs/judges.md) describe a working version: each judge is pinned as <model>@sha256:<hash>, where the hash is computed from the provider's model card (the exact GET /v1/models/<id> response body). Before any HTTP judge call the engine compares both the card's id and the body hash with the pin, and refuses to grade on a mismatch. The verdict is persisted with the exact fingerprint used. Builtin deterministic judges use a builtin:<id>@<version> fingerprint. The page itself is clear that this is a pin on the served model card, which catches silent re-serving under the same name. It is not a claim that verdicts are reproducible.
• OpenTelemetry's GenAI conventions already separate the two: gen_ai.request.model ("the name of the GenAI model a request is being made to") and gen_ai.response.model ("the name of the model that generated the response"). They have since moved to a separate semantic-conventions repository, so I am citing the split, not a stability level.
• EvalPort's own precedent: metadata.openeval.judge_hardening (Discussion #11) is a runner self-report on GraderResult.metadata that needed no schema change.

Proposal (a convention only)

An optional object metadata.openeval.judge on a GraderResult whose grader is an llm_judge. Every key optional; values are descriptive, not validated.

Key	Type	Meaning
model	string	The model identifier the serving side reported, as opposed to the suite's params.model (what was requested).
provider	string	Provider or endpoint host the call went to.
fingerprint	string	An opaque producer-defined identity digest.
fingerprint_method	string	Free text saying how fingerprint was computed, e.g. "sha256 of the provider model-card body". Without it a fingerprint is not interpretable by anyone else.
temperature	number	The sampling temperature actually sent.
temperature_source	string	"explicit" or "provider_default": whether the producer set it or inherited it.
prompt_sha256	string	SHA-256 of the judge prompt as rendered, or of the template, stated by prompt_basis.
prompt_basis	string	"template" or "rendered".
observed_at	string	RFC 3339 time of the call.

{
  "grader_id": "gr_faithfulness",
  "type": "llm_judge",
  "score": 0.9,
  "passed": true,
  "metadata": {
    "openeval": {
      "judge": {
        "model": "z-ai/glm-5.3",
        "provider": "integrate.api.nvidia.com",
        "fingerprint": "z-ai/glm-5.3@sha256:<hash>",
        "fingerprint_method": "sha256 of GET /v1/models/<id> response body",
        "temperature": 0,
        "temperature_source": "explicit",
        "observed_at": "2026-10-03T12:00:00Z"
      }
    }
  }
}

Landing it would add one Appendix B row, one Extension Mechanism section modeled on the judge_hardening one, and one conformance fixture. No schema, SDK or validator change.

What this deliberately does not do

• No refusal rule. "Refuse to compare results with no judge identity" is consumer policy, as AgentVerity's refusal of unpaired input is. A data format can make the identity available; it cannot make a consumer strict.
• No determinism claim. A fingerprint says which model card was served. It does not make a judge's verdict repeatable, and the convention says so.
• No standard fingerprint algorithm. Tools hash different things, so fingerprints are comparable only when fingerprint_method matches. The portable keys are model, provider, temperature and prompt_sha256.
• No claim about non-LLM graders. exact_match and friends are deterministic; this is for llm_judge.

Alternatives considered

• A typed GraderResult.judge field. Rejected for now. The fingerprint method is the unsettled part, and a typed field would freeze a shape from one implementation. A convention can be promoted to a field later as a MINOR change if several producers converge. The reverse is a breaking change.
• Put it in the suite only. That records the declared judge, which is the part that already exists.
• ResultSet-level only. A run can use several judges (one per grader), so identity belongs on the grader's result. A ResultSet-level summary can still be derived.

One thing to settle first

The spec's prose and examples spell reserved keys in the dotted form ("openeval.judge_hardening": ..., "openeval.aggregation": ...). The SDK converters and the conformance fixtures write them as a nested object ("openeval": { ... }). This proposal uses the nested form. If the dotted form is the intended one, the example above changes; if both are tolerated, the spec should say so. This affects every existing openeval.* key, not only this one.

Questions for commenters

1. Does your system record an observed judge identity today? What does it store, and would you fill these keys or want different ones? A real record (redacted as needed) is worth more than an opinion.
2. Is fingerprint plus fingerprint_method the right pair, or should there be no fingerprint key at all until two tools agree on one?
3. Should temperature_source exist? It records "the harness didn't set it and the provider default applied", which is the case the original comment cared about, but it asks producers to know it.
4. Nested or dotted keys (see above)?
5. Is a convention enough, or does anyone have a case that needs a typed field and a validator rule?

Evidence status: the Proofspan description is from its published docs, not from a run of it. Nothing here has been implemented against a second tool. A fixture from a real record from any tool that stores judge identity would be the strongest input.

[adhabnr-ux]

A real record from one tool that stores judge identity, answering question 1, with the caveat that I produced it (the AgentEval maintainer invited the package in AgentEval#203 but has not reviewed it), so it is not an independent implementer's fixture.

Source. AgentEval 0.42.0-beta (.NET), read at source commit 18538af and run through adapters/agenteval-dotnet-openeval-adapter/ (#123). Every EvalResult carries an EvalProvenance(Type, JudgeModel, PromptId, PromptHash, TokensUsed, EstimatedCost, CacheHit) record. This is what one leaf's provenance looks like in the checked-in sample (sample_output/judge_required/results.json, carried under metadata.agenteval.provenance):

{
  "type": "atomic-llm",
  "judge_model": "canned-judge",
  "prompt_id": "agenteval.judge.default-system.v1",
  "prompt_hash": "ec5f9a4be9894eb7",
  "estimated_cost": 0,
  "cache_hit": false
}

How its fields line up with the proposed keys.

proposed key	AgentEval has	fit
model (served)	JudgeModel, the identifier the eval was constructed with	does not fit. It is the declared model, i.e. the suite's params.model, not what answered. The served id is on the wire (Microsoft.Extensions.AI.ChatResponse.ModelId, "the model ID used in the creation of the chat response"), but ChatClientEvaluator does not record it. So the exporter writes judge.model only when the user opted in, and the README says it is the configured model.
provider	nothing	—
fingerprint + fingerprint_method	PromptHash: first 16 hex chars of SHA-256 over "atomic-llm.judge-input.v3" + evaluator prompt material + the criteria list, CRLF-normalised	fits this pair, and only this pair. The method is documented in source as "the instrument fingerprint". Without fingerprint_method nobody could tell it from a prompt digest.
prompt_sha256 + prompt_basis	PromptHash	does not fit: truncated, and the basis is neither "template" nor "rendered" (it covers framing version and criteria, not the rendered user turn). I did not write it as prompt_sha256.
temperature + temperature_source	nothing. ChatClientEvaluator sends ChatOptions { ResponseFormat = Json } with no temperature, so the provider default applies	the producer could truthfully write temperature_source: "provider_default" and no temperature, which is the exact case question 3 describes. It cannot write the value, because it does not know it.
observed_at	EvaluatedAt on the result	fits.
(none proposed)	PromptId (e.g. agenteval.judge.default-system.v1), TokensUsed, EstimatedCost, CacheHit	PromptId is a stable rubric identity that is more useful for "are these comparable" than any hash, and the proposal has no key for it. The other three are cost accounting, not identity.

What I take from it.

1. On question 2: this producer's one identity digest is a fingerprint with a method, not a prompt_sha256. The pair is the right shape for it; a bare prompt_sha256 would have forced either a lie or an omission.
2. A prompt_id (or rubric_id) key is missing from the table. Two AgentEval runs are comparable when PromptId and PromptHash match; the id is the human-readable half.
3. On question 3: temperature_source: "provider_default" is writable by a producer that knows it did not set a temperature, even when the applied value is unknown. That is an argument for keeping the key and making temperature independent of it.
4. The declared-vs-observed gap the RFC is about is present in this framework: it records the judge it asked for, while the served id is available on the response object and not persisted. I have not asked the maintainer whether that is deliberate; it is a one-line change on their side if they want it, and not something to request from here.

Not a run of a real judge: the chat client in the sample is canned, so judge_model: "canned-judge". The shape of the record, and what is and is not in it, comes from the framework's own types; the values come from my scenario.