[Spec Change] Unknown fields: strict validation by default, lenient consumption by flag (#107)

[adhabnr-ux]

Comment period: 2 weeks, until 2026-10-13. No normative text changes before then.

Problem

Issue #107 has the details and a reproduction. The short version: the three normative sources give three different answers about a key the spec doesn't define.

• SPEC.md §Forward Compatibility: "Runners MUST ignore unknown fields rather than failing."
• spec/schemas/*.json: additionalProperties: false at every object level, so the document is rejected.
• Python/TS reference validators: accept the document silently.

@ChelseaKR found this while shipping gauntlet report --format evalport (ChelseaKR/gauntlet#76). A document her SDK-based tests accepted would be rejected by anyone validating against the published schema. I hit the same thing myself: a config key in my llama-cookbook#1072 sketch got past the SDK and was caught only by the raw schema.

Proposal (option 3 in #107)

1. Validation is strict by default. A document that claims version X MUST NOT contain properties outside those defined for X, except under metadata objects, which stay open. The schemas already say this. Both SDKs would start rejecting unknown keys with a new code, UNKNOWN_FIELD, at the offending path.
2. Consumption is lenient. Reword §Forward Compatibility: a consumer reading a document whose version is a newer minor version than it implements SHOULD ignore properties it does not recognise rather than fail. Both SDKs gain an opt-in (validate_*(doc, allow_unknown=True) / { allowUnknown: true } / evalport-validate --allow-unknown) for that case.
3. Conformance. spec/conformance gains fixtures for both modes. The strict mode is the one used for conformance claims.

Why not the alternatives

• Strict everywhere, with no flag. Every minor version that adds a field would break older consumers reading newer files. The spec's forward-compatibility promise is worth keeping for readers.
• Open everywhere, dropping additionalProperties: false. Typos like config vs params, or a suite-level threshold, would pass silently. That silent acceptance is exactly the failure both real integrators hit. metadata already exists for producer-specific data.

Impact

• Producers that put custom keys outside metadata would start failing strict validation. Before merging, I'll run every adapter in adapters/ plus the interop examples in strict mode and fix anything that emits a stray key. fix(sdk): type-check optional fields, RFC 3339 timestamps, and Rule 6 null-score passed in both validators #106 already did the same kind of sweep for type errors, and 17 adapters needed fixes.
• Existing documents that follow the schema are unaffected.

Questions for reviewers

• Should metadata be the only open object, or should provider.extra stay open too? It already is type: object with no property constraints.
• Should the error code be UNKNOWN_FIELD, or reuse something existing?
• Is allow_unknown the right knob, or should leniency be automatic when the document's version is newer than the validator's?

A DO-NOT-MERGE reference PR will follow, per the usual process.

[adhabnr-ux]

The draft reference implementation is up: #110 (DO NOT MERGE; it stays a draft until this comment period ends on 2026-10-13).

What it does:

• Both SDK validators report UNKNOWN_FIELD at the key's own path on all 16 objects the schemas close with additionalProperties: false.
• metadata, provider.extra, params and summary.by_grader entries stay open.
• allow_unknown=True / { allowUnknown: true } / --allow-unknown (both CLIs and the Action) relaxes only that check.
• The allowed-key sets are pinned to the schemas by tests in both SDKs.
• 9 conformance fixtures cover both modes, with a new optional mode field on fixtures.

Sweep, run in strict mode on that tree:

• Adapters, before fixes: 5 of 68 failed, 10 failing tests (agenteval 2, ares 1, cannonade 2, mmmu 2, pyserini 3).
• Adapters, after moving their stray keys into metadata: 68 adapters, 0 failed, 824 passed, 414 skipped (CI's minimal install mode, so framework-dependent paths behind those skips were not run).
• Benchmarks: 22 suites / 8012 cases valid.
• Examples: every EvalPort document under examples/ and spec/examples/ valid.
• Interop: 6 passed.
• Conformance: 38 of 38.
• Python SDK: 670 passed, 2 skipped on 3.10, 3.11 and 3.13.
• TS SDK: 622 passed.

Two questions from the list above have a default in the PR but are still open:

• provider.extra is kept open (the schema already leaves it open).
• Leniency is an explicit flag, not automatic on a newer version.

The PR also flags that §Compatibility Policy says "MUST ... ignoring unknown optional fields" while the reworded §Forward Compatibility says SHOULD. Opinions welcome on which way to align them.

[adhabnr-ux]

A data point from outside the repo, recorded as promised. @jonathanngiroux-star (maintainer of Proofspan, disclosed in his comment) described what his converters do on langsmith-sdk#3278. I have not run Proofspan; everything below about it is from his comment, not from a test. His field-mapping tables are in docs/migrate.md (he says MIT).

1. Unknown keys are reported, not dropped. His report command prints a census of every key the converter does not map, so a user can check that nothing disappeared silently. This is the same failure [Spec Change] Unknown fields: strict validation by default, lenient consumption by flag (#107) #108 is about from the other direction: [Spec Change] Unknown fields: strict validation by default, lenient consumption by flag (#107) #108 makes a validator say "this key is not in the schema"; his census says "this key was in the source and the converter left it behind". EvalPort's strict mode covers the first only. It reports unknown keys with paths; it does not count anything, and the adapters have no import-time census.
2. Opaque subtrees must not be reported as unknown. His lesson: carry whole subtrees (token_usage, run metadata) across as units, and do not list their inner keys as unmapped, or the report looks mostly red on day one and people stop trusting it. That is an argument for the current DO NOT MERGE — reference implementation for Discussion #108 (unknown fields: strict validation by default, allow_unknown to opt out) #110 default, where metadata stays open and is not descended into. I checked that behavior on the DO NOT MERGE — reference implementation for Discussion #108 (unknown fields: strict validation by default, allow_unknown to opt out) #110 branch: a nested token_usage and langchain.run_id subtree under metadata produced no UNKNOWN_FIELD, while three stray keys at other depths were all reported in one pass.
3. Dry-run before write. He describes --dry-run emitting a JSON plan (counts, the field-mapping table, dropped-field notes, one converted sample) before anything is ingested. EvalPort converters do not do this and I am not proposing it here; it belongs to adapter design, not to this RFC. Noting it so it is on record.

None of this changes the proposal on the table. It is outside support for two of its defaults (keep metadata opaque; surface unknown keys instead of ignoring them) from someone who ships a converter, and it names a related gap (an import-time census) that strict validation does not close.

[adhabnr-ux]

One more concrete instance of the gap, from a producer written in a language neither SDK covers, so the only check it has is the JSON Schema itself.

The .NET AgentEval adapter (#123, adapters/agenteval-dotnet-openeval-adapter/) validates its output with JsonSchema.Net against schema/*.json directly. Its opt-in --proposed-verdict output adds the Result.verdict key from #49, which is not in the schema. Three validators, three answers on the same file (sample_output/proposed-verdict/judge_required/results.json, seven rows):

validator	result
JSON Schema (JsonSchema.Net 7.3.4, raw schema/result-set.schema.json)	rejected, 7 errors, one per /results/N/verdict (additionalProperties: false on Result)
Python SDK on main (44ad496)	valid, no errors
TypeScript SDK on main	valid, no errors
Python SDK on the #110 branch (187c7ac), default strict	rejected, 7 × UNKNOWN_FIELD at $.results[N].verdict
same, allow_unknown=True	valid

So a .NET producer that checks against the schema and a Python consumer that checks with the SDK currently disagree about whether the same document conforms, which is the situation this RFC describes. #110 brings the Python SDK into line with the schema and gives a flag for the other behavior; the adapter's own test pins the schema count (Assert.Equal(7, errors.Count)) so the two stay comparable.

One small observation for the error-path format: the schema library reports /results/0/verdict (JSON Pointer), #110 reports $.results[0].verdict (JSONPath-ish). Not a problem for #110, which should keep the SDK's existing path style, but a consumer correlating the two will need to normalise. Noting it, not proposing anything.

Everything above is reproducible: dotnet test in the adapter directory for the schema row; the main Python check is the one the adapter's CI workflow runs on every push, and the TypeScript and #110 rows were run by hand against sdk/typescript/dist and the fetched pull/110/head.