feat: experimental evaluator behind an explicit experimental surface (ADR-0007) - #19
Merged
Conversation
Add docs/mcp-clients.md with copy-paste setup for Claude Code, OpenAI Codex CLI, and GitHub Copilot (VS Code), the tool table, the PATH and restart gotchas, and the remote-only-clients boundary (external bridge or a separately governed hosted distribution; ADR-0004). Gitignore /.mcp.json: client configuration is personal wiring, not project source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…(ADR-0007) Implement the JPS Core §§7-8 experiment, exactly as pinned by the specification's RFC 0006 (Draft), as the first of the two independent implementations its acceptance requires. It claims NO evaluator conformance -- JPS 0.1.0-draft §3.4 forbids such claims -- and may change or be removed without compatibility promise. - internal/evaluation: three-valued conditions (§7 preamble + 7.1-7.5; RFC 6901 pointers; type-preserving equality with mathematical number comparison that degrades to unknown when equality cannot be determined; §2.2 grammar-string-only ordered comparison), the §8 resolution model with RFC 0006's chosen step-2 semantics, and §8.1 handoff incl. the no-destination direct-escalation edge. - Errors are not dispositions: a non-conformant pack, unsupported required extension, malformed/oversized input, or undeclared evidence key is refused; resource limits keep their JPS-RESOURCE-* codes and the 10 MiB boundary lives in the engine so both surfaces share it. - CLI `experimental evaluate <pack> --facts f [--evidence f] [--supported-extension x]`: EXPERIMENTAL banner, disposition + trace, exit 0 for any produced disposition; stdin/remote-path/empty-evidence guards. - MCP `experimental_evaluate` (strict argument decoding honoring additionalProperties: false), labeled experimental in name, description, and payload (experimental: true, conformanceClaim: none). - Tests: the nine RFC 0006 appendix instances (incl. 7a/7b, the resolved divergence, and the conflict probe) run as acceptance tests, plus unit tables for decimals, pointers, equality, three-valued operators, evidence tri-state, §8.1 trigger matching, escalate precedence, and the error boundary. - Docs: ADR-0007 (+ amended-by note on ADR-0003), README, architecture, authoring-lifecycle, and agent-testing all name the experimental exception instead of claiming the runtime evaluates nothing. Verified by an adversarial workflow (semantics fidelity vs §§7-8 and RFC 0006, guardrails, error boundary, test adequacy; 28 findings fixed) and an Opus re-check (31 verdicts FIXED incl. 21 targeted mutants all killed; the nine appendix instances reproduce identically on CLI and MCP, deterministic across runs). All conformance cases pass; validation behavior, claims, and exit classes unchanged. See ADR-0007. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
kikashy
added a commit
that referenced
this pull request
Jul 27, 2026
kikashy
added a commit
to Judgment-Pack/judgment-pack-spec
that referenced
this pull request
Jul 27, 2026
The evaluator now exists: [judgment-pack-runtime#19](Judgment-Pack/judgment-pack-runtime#19) implements this RFC's pinned semantics behind an explicitly experimental, non-conformance-claiming surface (runtime ADR-0007), with the nine appendix instances as its acceptance tests — reproduced identically over CLI and MCP. This amendment records the implementation experience RFC 0000 asks for: - the restated step 2 was implementable exactly as written (7a/7b deterministic); - **number representability** is a real gap — extreme exponents make equality undeterminable; mapped to §7.4's incomparable→`unknown`, added as an unresolved question; - §8.1's no-destination direct escalation shape needs confirming in the later draft; - honoring `additionalProperties: false` requires strict wire-argument decoding. Also pins the independence bar for implementation #2: derived from the RFC text, never ported from the first implementation. 45 tests pass; site builds. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jul 28, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The first of the two independent implementations spec RFC 0006 (Draft) needs for acceptance: the JPS Core §§7–8 experiment, exactly as pinned by the RFC's Specification sketch, behind an explicitly EXPERIMENTAL surface that claims no evaluator conformance (JPS
0.1.0-draft§3.4 forbids such claims).judgment-pack experimental evaluate <pack> --facts f [--evidence f] [--supported-extension x]— disposition + trace, exit 0 for any produced disposition.experimental_evaluate— strict argument decoding, labeled in name, description, and payload (experimental: true,conformanceClaim: "none").unknownwhen undeterminable; §2.2 grammar-string-only ordering), the §8 resolution model with RFC 0006's chosen step-2, §8.1 handoff incl. the no-destination direct-escalation edge. Conflict is never tie-broken.Also:
docs/mcp-clients.md(per-client setup for Claude Code / Codex / GitHub Copilot) and.mcp.jsongitignored as personal wiring.Verification
unknownper §7.4).go test ./...(13 packages) / conformance corpus: all green.ADRs
ADR-0007 (proposed → accepted on merge); amends ADR-0003 narrowly (the validation tools still evaluate nothing; the surface gains exactly one tool that evaluates experimentally and says so).
🤖 Generated with Claude Code