feat: runtime-agnostic conformance harness with thin adapter protocol#82
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Replace the Python-bound conformance runner with an adapter architecture so any OA runtime can be certified against the same suite: - PROTOCOL.md: versioned adapter contract (JSON over stdin/stdout), capability registry, and honesty rule (no silent feature degradation) - harness/: single driver for discovery, assertions, capability skipping, and markdown/JSON conformance matrix reporting - adapters/: thin wrappers for the Python reference runtime and the npm runtime; old conformance_runner.py becomes a deprecation shim - npm runner parity: full result envelope (input/prompt/engine/ raw_output/chain), chain cycle detection, required-input validation, legacy prompt maps, prompt overrides, response_format: text, injectable invokeFn, deep-copied inputs at task boundaries - npm guard: refuse specs declaring tools:/sandbox:/behavioural_contract with UNSUPPORTED_FEATURE instead of silently ignoring them - capability tags (requires:) on non-core cases; CI job runs both adapters and uploads the conformance matrix artifact Matrix: python-reference 29/29 pass; npm 27 pass + 2 honest UNSUPPORTED. Co-authored-by: Cursor <cursoragent@cursor.com>
aswhitehouse
force-pushed
the
feat/conformance-adapters
branch
from
June 11, 2026 03:00
a4a38c4 to
54b26fa
Compare
Co-authored-by: Cursor <cursoragent@cursor.com>
sgriffiths
requested changes
Jun 21, 2026
sgriffiths
left a comment
sgriffiths
left a comment
Contributor
There was a problem hiding this comment.
Can you look at these comments
…reading
Three fixes from PR review:
- npm extractJson now mirrors the reference runtime exactly: strip a
leading markdown fence, then JSON-parse the whole text — arrays and
scalars are returned as parsed, and the divergent "first {...} block"
heuristic is gone
- cycle detection in BOTH runtimes now uses a recursion stack (added on
descent, removed on backtrack) instead of a globally-visited set,
which falsely rejected legal diamond DAGs (D→[B,C], B→A, C→A)
- npm dependency runs now thread the delegation visited-set instead of
passing a fresh Set, so DELEGATION_CYCLE_ERROR fires at the same
point as the Python runtime
Locked in with three new conformance cases (json-array, json-scalar,
diamond-graph) that both runtimes pass, plus Python regression tests.
Co-authored-by: Cursor <cursoragent@cursor.com>
sgriffiths
self-requested a review
July 3, 2026 22:19
Merged
sgriffiths
pushed a commit
that referenced
this pull request
Jul 11, 2026
…p proposal Reverts the two doc downgrades from 2999cde that were based on a pre-#82 branch view. The conformance harness (spec.conformance.harness.harness) and the node adapter have been on main since #82, and CI runs both runtimes, so the original wording was correct against main: - AGENTS.md: restore the both-runtime conformance command (harness.harness --adapter python --adapter node). - README.md: restore "certified identical across runtimes". The skill-wrapper engine-swap docs from 2999cde are kept. Also flips the markdown-interop proposal to Status: Accepted, since merging the README positioning accepts it. Addresses review feedback from @aswhitehouse on #86. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XdPprn1xTRiR1rGZf4jUfP
sgriffiths
pushed a commit
that referenced
this pull request
Jul 11, 2026
…p proposal Reverts the two doc downgrades from 2999cde that were based on a pre-#82 branch view. The conformance harness (spec.conformance.harness.harness) and the node adapter have been on main since #82, and CI runs both runtimes, so the original wording was correct against main: - AGENTS.md: restore the both-runtime conformance command (harness.harness --adapter python --adapter node). - README.md: restore "certified identical across runtimes". The skill-wrapper engine-swap docs from 2999cde are kept. Also flips the markdown-interop proposal to Status: Accepted, since merging the README positioning accepts it. Addresses review feedback from @aswhitehouse on #86. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XdPprn1xTRiR1rGZf4jUfP
sgriffiths
pushed a commit
that referenced
this pull request
Jul 12, 2026
…p proposal Reverts the two doc downgrades from the previous commit, which were based on a pre-#82 branch view. The conformance harness (spec.conformance.harness.harness) and the node adapter have been on main since #82, and CI runs both runtimes, so the original wording was correct against main: - AGENTS.md: restore the both-runtime conformance command (harness.harness --adapter python --adapter node). - README.md: restore "certified identical across runtimes". The skill-wrapper engine-swap docs are kept. Also flips the markdown-interop proposal to Status: Accepted, since merging the README positioning accepts it. Addresses review feedback from @aswhitehouse on #86.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Resolves the conformance suite's biggest gap: it could only certify the Python reference runtime. The suite is now a true multi-runtime certification tool built on a thin adapter architecture.
spec/conformance/PROTOCOL.md— versioned adapter contract. Any runtime plugs in as an executable speaking JSON over stdin/stdout:--capabilitiesreturns a manifest; otherwise one case in → one result envelope out. Includes a capability registry and the honesty rule: a runtime must not declare a capability it doesn't enforce, and must refuse specs using features it can't honour.spec/conformance/harness/harness.py— single runtime-agnostic driver: case discovery, assertion engine, subprocess adapter invocation, capability-based skipping (PASS / FAIL / UNSUPPORTED), and markdown + JSON matrix reporting.spec/conformance/adapters/— thin wrappers (~100 lines each) for the Python reference runtime and the npm runtime. The oldconformance_runner.pyis now a deprecation shim.npm runtime parity
Running the suite against the npm runtime immediately exposed real gaps, all fixed here:
input,prompt,engine,raw_output,chain) — previously returned a different, undocumented shapedepends_onchain semantics matching the reference runtime (accumulating merge,chainkey, cycle detection — a dependency cycle previously caused infinite recursion)CHAIN_INPUT_MISSING), legacy per-task prompt maps, caller prompt overrides,response_format: textinvokeFnfor embedding/testing; deep-copied inputs at every task boundary (IIS immutability, now cross-runtime)tools:/sandbox:/behavioural_contract:are refused withUNSUPPORTED_FEATURErather than silently degradedConformance matrix
output-schema-validation,contracts)New CI job builds both runtimes, runs both adapters, and uploads the matrix as an artifact.
Test plan
python -m spec.conformance.harness.harness --adapter python --adapter node— both greenpython -m spec.conformance.runner.conformance_runner) still worksexamples/sandboxed-agent/spec.yamlwithUNSUPPORTED_FEATUREMade with Cursor