RFC: rate-based GraderResult denominators (trials/successes) — promoted from #58

[adhabnr-ux]

Summary

This promotes issue #58 to a full RFC and opens the two-week comment period that this project's Governance section requires before any schema change. The question is narrow: should a rate-based GraderResult.score carry its denominator, and if it should, where does it live and is it required?

A reference implementation of one option (Option A below) is on a draft, do-not-merge PR so there is something concrete and tested to review: #66. It is not a proposal to merge ahead of this discussion.

Where this came from

@sattyamjjain raised this while we were discussing whether a garak-openeval-adapter made sense (NVIDIA/garak#2110). He maintains a red-team harness for robot vision-language-action policies and is not affiliated with garak or EvalPort. The framing is his:

An attempt outcome without its benign floor is not readable, because 0/5 and 0/500 both serialise as 0.0.

The discussion started in #58, which asked whether the gap was real enough to warrant a schema change at all. This Discussion is the next step in the same process that #22 and #45 went through.

The gap

GraderResult.score in spec/schemas/resultset.json is a bare number in [0, 1], or null. Some graders are really successes / trials: pass rate over N trials, attack success rate over N attempts. For those graders the schema has no place for N. A score of 0.0 from 0 successes in 5 trials and a score of 0.0 from 0 in 500 serialize identically, even though their statistical support differs by two orders of magnitude.

Today a producer that wants to record N has to put it in GraderResult.metadata under a key of its own choosing. No generic consumer knows to look for it, and a producer that forgets produces a document that looks complete and is not.

Why Result.attempt does not already cover this

When each trial is its own Result row, N is already derivable: count the rows that share (test_case_id, run_id) using Result.attempt (Discussion #22). That is the right model for harnesses that emit one row per trial.

It does not cover a single GraderResult that aggregated N trials internally. For example, a grader might run 500 episodes of one attack and report one attack-success rate. There is one Result and one GraderResult, so there are no rows to count. The denominator exists only inside the grader, and the schema currently has no place for it. This RFC is about that case.

Options

Option A: optional trials / successes on GraderResult (implemented in #66)

• trials: integer >= 1, the denominator. Optional.
• successes: integer >= 0, the numerator. Optional. If present, it requires trials, expressed in the JSON Schema as dependentRequired: {"successes": ["trials"]}, so the raw schema and the hand-rolled SDK validators agree (REQUIRED).
• successes <= trials (SUCCESSES_EXCEED_TRIALS) and, when score is non-null, |score - successes/trials| <= 1e-6 (RATE_SCORE_MISMATCH). These are cross-field rules that plain JSON Schema cannot express without $data. Both reference SDKs enforce them in their hand-rolled validators only, the same class of rule as SELF_PARENT and DUPLICATE_ATTEMPT. Each SDK's schema-consistency tests document the gap explicitly.

Pros: purely additive (MINOR). Every existing document stays valid. The field names and semantics are the same across both SDKs. Consumers get a standard place to look for N.

Cons: optional fields do not fix the failure mode #58 is most concerned with. In @sattyamjjain's words: "If a ResultSet entry can serialise a score without its denominator, every adapter author has to remember to carry it, and the ones who forget produce something that looks complete and is not." Option A makes carrying N possible and standard, but not mandatory.

Option B: Option A, plus trials required when the grader opts in as rate-based (not implemented)

A Grader definition declares itself rate-based, for example with a grader-level flag or a reserved key such as openeval.rate_based: true. A GraderResult for that grader without trials is then rejected.

Pros: provides the enforcement #58 asks for, scoped to the graders where a denominator means something. Single-shot graders (exact_match, json_schema, a single llm_judge call) are unaffected.

Cons: the check becomes a cross-document rule. validate_result_set() validates a ResultSet alone and does not receive the suite's Grader definitions, so either its signature changes or the "rate-based" declaration has to be copied onto each GraderResult, where it could drift. It also means deciding where the declaration lives: a new top-level field on Grader (whose schema is additionalProperties: false), params, or a reserved metadata key. It still depends on the producer setting the flag. A producer that forgets the flag is in the same position as one that forgets trials under Option A, though the omission is at least made once per grader rather than once per result.

Option C: no schema change, a metadata.openeval.trials convention

Document a reserved GraderResult.metadata.openeval.trials (and successes) key convention in SPEC.md, with no change to the schema.

Pros: no schema change is needed at all. GraderResult.metadata is already additionalProperties: true, whereas GraderResult itself is additionalProperties: false, which is why Option A has to change the schema. It is the smallest possible change and costs nothing to adopt or ignore.

Cons: it is optional by convention, which is exactly the thing that #58 identifies as failing silently. A validator cannot type-check a convention inside open metadata without special-casing it, so it would not catch trials: "500", successes > trials, or a score that contradicts its own counts. A misspelled key (metadata.openeval.trails) validates cleanly.

Explicit non-goals

• Confidence-interval computation. Wilson, cluster-corrected Wilson, or any other interval is out of scope. @sattyamjjain is contributing a separate successes, trials -> interval utility with no framework dependency. That utility and this schema question are independent and do not share a review timeline.
• Any claim that trials alone supports a valid interval. Per @sattyamjjain's caveat, for correlated or clustered trials (e.g. episodes within one task) a naive interval over pooled trials is too narrow. trials lets a reader tell a well-supported rate from a noisy one at a glance. Nothing in the spec text should imply more than that.
• Changes to how summary is computed. summary remains producer-computed and optional.
• Any garak-specific decision. garak has not said whether it wants an adapter. This is EvalPort's schema question regardless.

Comment period

Opens 2026-09-27, closes 2026-10-11. The reference-implementation PR stays a draft throughout. If the discussion reaches rough consensus, the implementation will be re-opened fresh against whatever main looks like at that point, rather than merged as-is.

Questions for commenters

1. Is Option A's "optional, but standard and validated when present" enough, or does the silent-omission failure mode make Option B necessary? If B, is a cross-document check (passing grader definitions into ResultSet validation) acceptable, or should the rate-based declaration be carried per GraderResult?
2. If Option B, where should the "I am rate-based" declaration live: a new field on Grader, a params convention, or a reserved metadata key?
3. Is trials-only (no successes) a useful state to allow, given that score * trials already implies the numerator? Or should the two be required together so that the numerator is never reconstructed from a rounded float?
4. Is 1e-6 the right tolerance for RATE_SCORE_MISMATCH? Should the rule be dropped entirely in favor of treating score as authoritative?
5. For producers that already emit one Result per trial with attempt, should the spec say anything about the relationship between a per-row attempt count and a trials value on an aggregating grader, or leave the two independent?