Truth reconciliation, watchable proof, and three shipped features (adapter-verify, Break the Ratchet, CheckerProof)#130
Merged
Conversation
Truth-reconciliation pass ahead of wider launch work: - Rewrite the demo walkthrough claims (README, examples/demo-app) to match the committed evidence: one dry-run plus one recorded, unapplied maker/checker cycle, not a week of merges and blocks. Note which planted debt items the dry-run proposes versus what stays planted for the corpus. - Reframe AgentProof as a self-graded regression benchmark everywhere it is described (agentproof/README, root README, site meta/JSON-LD, noscript), standardize on the 25/25 normative headline, and state plainly that the Sigstore attestation proves provenance, not independent validation. - Honest tagline: the package description and hero copy now claim what the line-based gate integrity check actually enforces (structural weakening), and a new README section documents what it cannot catch and which layers cover that gap. - Sweep present-tense claims about unshipped capabilities (enterprise estates, mirror mode, cross-repo network, compliance AG06) to explicit roadmap tense; label the network prompt module as not shipped. - Drop tests/ from the npm files allowlist (npmignore already excluded it), remove the OpenSSF Best Practices placeholder badge, de-stuff site keywords, and correct the site README's CNAME description (www vs apex). - Regenerate the prompt bundle, snapshot, and policy attestation.
- Capture a real gaming diff (skip injection + assertion removal, committed as a 'fix') being rejected by the gate integrity check with exit 1 and SARIF rule codes, and the honest fix for the same debt passing the identical gate: verbatim output and both patches committed under examples/demo-app/runs/2026-07-08T05-30-00Z/, reproducible bit-for-bit with no model involved. - Capture the Gauntlet before (0/3 UNHARDENED, no gate wired) and after (3/3 HARDENED) wiring the shipped workflow template, with the honest N/A denominator visible. - Rewrite the demo walkthrough around the committed evidence and add the gate-rejection and Gauntlet steps. - Add an animated terminal capture (SMIL SVG, no external assets) of the rejection above the README fold, linking to the committed evidence. - Promote the Gauntlet in the 60-second path and teach it to print a copy-paste share line and a shields.io badge snippet encoding exactly what was measured (blocked/applicable on the day it ran).
… scope - Rebuild README to a single-screen story: real terminal capture, the 60-second read-only path (dry-run + gauntlet), a five-minute CI gate section with one canonical workflow, the trust-boundary diagram, a 'how this compares' answer (branch protection, LLM-judge review, eval benchmarks), what it catches and what it cannot, the loop with evidence links, an honest development-practice section naming the solo, AI-assisted, self-governed reality, and a links table for everything else. Cost model and embedding guidance move to QUICKSTART. - Replace the site hero terminal's invented output with verbatim demo-app output including the real gate rejection lines. - Add ADR-045 (scope focus for launch): no AST tier (zero-dependency guarantee wins), false-positive-only ratchet improvements with a no-new-false-negative invariant, repo-level assertion delta as advisory only, and a recorded rationale for every deferred surface (challenge harness, CheckerProof, adapter verifier, second host example). - Move the cross-repo knowledge network design under docs/research/ and reframe AgentProof standards-body language from submission-in-progress to aspiration contingent on adoption. - Regenerate prompt bundle, snapshot, and policy attestation.
One scannable paragraph mapping the published 0.1.0-alpha (npm, with the next dist-tag tracking ongoing work) to the ~40 detailed entries below it, so an evaluator can see what shipped without reading commit-level prose. No version change; release timing stays an owner decision.
… gitignore silently dropped ADR-045 implementation (docs/adr/ADR-045-scope-focus.md): - Strip same-line string literals and block comments before the TS/JS type-escape check (scripts/guard-ratchet.mjs: stripLineLocalNoise), so a ': any' or 'as any' inside prose like a string or a same-line block comment is no longer a false positive. The stripper bails to the raw line on anything it cannot prove closed on that line (an open quote, an open block comment, a possible regex literal ahead of a quote), so a real cast hidden behind a string decoy on the same line is still caught. This is a false-positive-only relaxation: every existing gaming fixture still blocks. - Recognize Node's built-in assert module member-call style (assert.equal, assert.match, assert.rejects, and so on) as assertions, strong assertions, and weak-existence checks. This closes a real false negative: the ratchet previously only recognized bare assert(...) and Jest-style expect(...), missing the dominant style in this project's own test suite (2000+ occurrences). Discovered while building the fixtures for this change; folded in as the same class of fix. - Add an advisory-only, informational cross-file assertion tally, printed after the per-file findings and never affecting the exit code. It is explicitly NOT a new detection mechanism: a repo-level sum of per-file deltas cannot go negative without an already-blocking per-file delta, so it exists purely as review context, not as an anti-gaming boundary (documented in the code comment). - New fixtures and tests covering all of the above (6 new ratchet.test.mjs cases). - Reframe AgentProof's standards-body language and relocate the knowledge-network design doc into docs/research/, per ADR-045. Also: the previous commit's demo-app evidence run (examples/demo-app/runs/2026-07-08T05-30-00Z/) was silently dropped by the bare 'runs/' rule in .gitignore and was never actually committed, contradicting the commit message that claimed it was. Force-added it now, scoped an un-ignore exception for that specific path so this cannot silently recur, and refreshed the captures to reflect the corrected assertion count from the assert.match fix above (+0/-2, not +0/-1, since assert.match(...) is now recognized).
…cker, scope CLI permissions ADR-045 Phase 5 (docs-first, per the sustainability panel's guidance: no new benchmarks, verifiers, or examples until real demand exists): - docs/agents.md: the canonical reference for the crew roster, the config-driven role/runner/model/provider system, cost classes with fail-closed unknown-provider semantics, and worked recipes (a local maker with a paid checker, a fully local zero-cost pair, adding a new crew role with no code change). This surfaces a real, differentiated capability that previously lived only in config comments and two overlapping docs. docs/agent-org.md and docs/ops/runner-model-config.md now redirect here; incoming links (README, ARCHITECTURE, QUICKSTART, docs index) updated. - docs/adapters.md: the contract for bringing an external agentic CLI into the tool-loop execution path (ADR-032): what the adapter must do (stdin prompt, env credentials, containment, timeout/turn-cap independence, exit semantics), and what a reviewer checks by hand today, since no automated adapter-verify exists yet. - prompts/roles/checker.txt: add an adversarial rubric targeting exactly what the deterministic ratchet cannot see (expected-value drift, cross-file assertion migration, vacuous-in-spirit strong assertions, scope creep beyond allowed_edit_set). Purely additive; existing procedural content and the anchors test-prompt-behavior.mjs checks are untouched. - scripts/agent/run-cycle.mjs: replace --dangerously-skip-permissions on the CLI-transport maker/checker invocation with --permission-mode manual plus a scoped --allowedTools list (Read/Edit/Write, git/gh, and the gate runners the multi-language ratchet supports). This is not a guess: I spawned the real claude CLI against scratch repos and found --permission-mode acceptEdits does NOT confine execution to --allowedTools as the flag names suggest (an unlisted rm went through); --permission-mode manual does, denying cleanly with no hang in non-interactive mode, verified with both an Edit and a git/npm round-trip. Extracted the pure buildRoleCliArgs() and threaded a spawnImpl seam so this is now covered by a real regression test (tests/run-cycle-cli-permissions.test.mjs, 4 cases) instead of being the untested path it was. - ARCHITECTURE.md: new 'Host vs. self: the two governance targets' section distinguishing the product (governing a host repo, demonstrated by the demo-app evidence) from self-governance (modonome governing itself, continuously exercised but not a second host), anchored on the control panel's two-tier write-auth model, and honest that no public evidence yet shows modonome governing a third-party repo.
Phase 6, the final phase of the brilliance plan (docs/ADR-045): - SUPPORT.md: no-SLA support policy, real triage priorities, and what merging a community AgentProof scenario or adapter actually commits the maintainers to (permanent CI cost, ongoing upstream-compatibility burden). Added to the root-doc allowlist (scripts/check-md-governance.mjs). - Found and fixed while writing it: CONTRIBUTING.md's existing 'triage within 5 business days' promise directly contradicted the new honest framing. Pointed it at SUPPORT.md instead of carrying a specific promise this team's real commit history cannot back up. - docs/launch/: a Show HN draft (title candidates, a first comment that leads with the mechanism and issues the break-it challenge, anticipated pushback with honest answers already drafted), article seeds (a Goodhart cross-post update anchored on the new gate-rejection evidence, a post-launch CheckerProof piece, a standalone trust-boundary explainer), and an X thread skeleton. Every claim in all three links to a committed evidence file, no exceptions. - OWNER-ACTIONS.md: consolidates every owner-only item raised across this branch's six phases into one list, with two items marked as hard launch gates (a real armed week with published evidence; the GitHub Marketplace listing actually being live) rather than optional polish. Corrects an assumption from the original review: CODEOWNERS already lists two people, so the ask is confirming engagement, not recruiting from zero. Also lists the deferred, demand-gated work from ADR-045 (adapter-verify, CheckerProof, the Break-the-Ratchet harness, a second host-adoption example) with its trigger conditions, so it is discoverable without being built ahead of real demand. - docs/audits/claims-audit-2026-07-08.md: a closing audit that re-verifies this session's own claims by re-running the actual commands (guard-ratchet, agentproof/runner.mjs, test-prompt-behavior.mjs, git ls-files) rather than trusting the commit messages that made them, in the same deliberately uncharitable format as the two prior audits. Carries forward what it did not re-check (security/compliance-as-code, Level 3 conformance) rather than implying a full sweep.
docs/adapters.md previously promised this as a well-scoped follow-up once a second adapter showed up; building it now instead of waiting. - scripts/adapter-verify.mjs: two-tier verifier. Tier 1 (static, always runs) validates an adapters.json entry against schemas/adapters.schema.json and runs it through check-licenses.mjs's license/boundary gate. Tier 2 (live) spawns the real adapter binary through the actual runToolLoopAdapter code path, not a reimplementation of it, against a scratch target directory and a local mock OpenAI-compatible endpoint (no real network, no cost), and confirms the adapter both consumed its stdin-delivered prompt and confined its output to the pinned target. When the named binary is not on PATH, tier 2 reports SKIPPED, never a silent pass or a hard failure. - fixtures/adapters/reference-adapter.mjs: a minimal, real, working adapter that satisfies the docs/adapters.md contract end to end. Doubles as the target for --self-test, so the verifier's own logic is provable without needing any external binary installed. - Wired into bin/modonome.mjs as "npx modonome adapter-verify <name>" and "npx modonome adapter-verify --self-test". - Fixed a real, pre-existing bug this surfaced immediately: the registered opencode entry in adapters.json carries role/exec_mode fields the schema didn't declare, so the entry failed schema validation the moment a real validator was run against it. Widened the schema to match the already- shipped manifest. - Corrected two stale claims in docs/compliance/openssf-badge-evidence.md found while re-verifying it for the staleness gate: the DCO criterion cited a sign-off note in CONTRIBUTING.md that was never actually there (now correctly Gap, not Partial), and the bus-factor gap undercounted CODEOWNERS (which lists two people) while still correctly noting only one has authored commits. - tests/adapter-verify.test.mjs: 10 tests, including two real end-to-end runs of the actual reference-adapter subprocess talking to a real local HTTP server, not just fake-spawn mocks.
ADR-045 deferred a public challenge harness, judging the effort not worth it until real external interest existed. Building it now instead, with the exact safety constraint the original technical review specified: submissions are never executed. - challenge/judge.mjs: reads a submitted .patch file as plain text and hands it to guard-ratchet.mjs's existing --diff mode, itself a pure text analyzer that never applies a diff or executes anything it contains. The judge mechanically confirms one fact (did the ratchet flag it) and reports the submitter's declared claim for a maintainer to review; it never decides on its own whether a claim is true, since that requires understanding the diff's semantics, the exact class of judgment the ratchet cannot make. - schemas/break-the-ratchet-submission.schema.json: the declaration shape (title, category, claim, why it is a real weakening, expected verdict). - challenge/examples/expected-value-drift/: a real, working seeded example, not a stub. A discount-calculation assertion's expected value quietly moved from the correct result (95) to what the broken code actually returns (100). Verified: the ratchet does not flag it, confirming the documented semantic blind spot is real. Seeds the hall of fame honestly instead of launching with an empty table that implies no one has looked yet. - BREAK-THE-RATCHET.md: the public rules, submission process, and hall of fame. Wired into the README's existing break-it invitation. - .github/ISSUE_TEMPLATE/break-the-ratchet.yml: mirrors the existing agentproof-scenario.yml conventions, requires the local judge output pasted into the issue. - Wired into bin/modonome.mjs as `npx modonome break-the-ratchet <dir>`. - tests/break-the-ratchet-judge.test.mjs: 11 tests, including an explicit safety-boundary test (a patch referencing a nonexistent path is judged without ever creating that path) and both real end-to-end runs (the seeded example scores candidate-break; a known, already-caught gaming fixture correctly scores not-a-break).
ADR-045 deferred this as needing recurring, owner-run model calls. Built and run for real instead, not simulated. - checkerproof/runner.mjs: resolves the checker role from .modonome/config.yaml exactly as a real armed cycle would, reuses scripts/agent/review-diff.mjs's existing buildReviewPrompt/parseVerdict, and adds live anthropic-cli execution alongside the existing openai-http path. Always exits 0 (advisory, never a gate). When no model is reachable, every scenario reports SKIPPED, never a fabricated 0/N. - checkerproof/scenarios/: five real scenarios matching the categories in the hardened checker rubric (prompts/roles/checker.txt point 1a): expected-value drift, cross-file assertion migration, a vacuous-in-spirit assertion, scope creep past allowed_edit_set, plus a negative control so a checker that rejects everything cannot score well by accident. - checkerproof/evidence/: a real run against this repo's actual configured checker (claude-opus-4-8, anthropic-cli). 5/5 scored scenarios caught correctly on 2026-07-08. Reproducible: node checkerproof/runner.mjs --write-evidence. - Wired into bin/modonome.mjs and package.json, deliberately not added to the verify script chain since verify's guarantee is no network or secrets. - Fixed a repo-hygiene finding: adapter-verify.mjs's probe marker used Math.random(); switched to crypto.randomUUID(). - tests/checkerproof.test.mjs: 9 tests against injected reviewImpl, no network or cost.
ADR-046 supersedes the specific deferral in ADR-045 point 5, on record: direct owner instruction reversed the demand-gating call for adapter-verify, Break the Ratchet, and CheckerProof, while every safety and scope constraint ADR-045 attached to each stayed exactly as specified (submissions never executed, CheckerProof never fabricates a score and stays advisory, the adapter verifier degrades gracefully). ADR-045's other points (the false-positive-only ratchet constraint, the AST-tier cut, the advisory-only cross-file signal) are untouched and still stand. - ADR-045: strikethrough note on point 5 pointing to ADR-046, so a reader who only sees ADR-045 does not conclude these are still unbuilt. - OWNER-ACTIONS.md: moved the three from "deferred, demand-gated" to "shipped," with the real follow-up work that shipping does not solve by itself (recurring CheckerProof runs, Break the Ratchet submission triage). Good-first-issue 3 now points at using the live adapter-verify tool instead of proposing it; issue 6 points at extending the live CheckerProof corpus; added issue 7 (Break the Ratchet triage). - docs/launch/article-seeds.md and show-hn.md: updated to reference the real, shipped, evidenced tools instead of describing them as future work.
…the armed-week gate) Not a full live cycle. Real, honest evidence of exactly what it is: the full two-key arming mechanism (config autonomy_enabled + dry_run, plus the authoritative env-only MODONOME_ARMED) engaged for real, --execute attempted for real, and the budget gate correctly refusing to spend before any network call at the default remote_model_budget_usd_per_day: 0. - examples/demo-app/runs/2026-07-08T14-19-00Z/: dry-run-plan.txt (unarmed baseline), armed-execute-refused.txt (the live refusal), and network-policy-check.txt (evidence that no free-tier or local-model host was reachable from this session: both models.github.ai and huggingface.co returned a hard 403 policy denial from the network proxy, not an application error). - The repo's config was reverted to its safe defaults immediately after each test; it does not stay armed as a side effect of this evidence run. - OWNER-ACTIONS.md: narrowed the armed-week gate to what's actually still open. "Does arming work under a real attempt" is now closed. "Does a live maker/checker exchange happen" is not, and needs either a free/local model host reachable from wherever this next runs, or an explicit, deliberately raised paid budget.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
scripts/adapter-verify.mjs's main() destructured manifest from loadAndValidateManifest without using it, and imported containedCwd from tool-loop-adapter.mjs without using it. Dropped both; the function's own internal use of manifest is untouched. Regenerated .modonome/snapshot to match.
nateshpp
pushed a commit
that referenced
this pull request
Jul 8, 2026
Resolves the .gitignore conflict from PR #130 by keeping both additions (the anchored .claude/skills/ un-ignore and the examples/demo-app/runs/ un-ignore). Regenerates the snapshot and policy attestation against the merged tree.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
A comprehensive hardening and feature pass in three parts.
Part 1: truth reconciliation. Rewrote every claim across README, the demo walkthrough, and AgentProof's framing to match its committed evidence exactly (the walkthrough previously described a week of merges and ratchet blocks; the real evidence was one dry-run and one recorded, unapplied cycle). Restructured the README around proof-first ordering with a real terminal capture and a "how this compares" section. Hardened the gate integrity check itself: false-positive-only string/comment stripping around the type-escape check, plus a real coverage gap fix (Node's
assert.equal/assert.matchmember-call style, the dominant style in this project's own test suite, was previously invisible to the removal check). Added a scoped--permission-mode manual+--allowedToolsprofile for the CLI-transport maker/checker invocation, replacing a blanket--dangerously-skip-permissions, verified empirically against a live session (an initial attempt with--permission-mode acceptEditswas found to not actually confine execution to the allowlist and was discarded in favor ofmanual, which does).Part 2: three new, real, tested features (
docs/adr/ADR-046-ship-the-deferred-features.md):adapter-verify(npx modonome adapter-verify): a two-tier conformance checker for the tool-loop adapter contract. Static tier validates schema/license/boundary; live tier spawns the real binary through the production code path against a scratch target and a local mock endpoint, confirming stdin consumption and containment, and skips cleanly (never fails hard) when the binary is absent. Ships with a working reference adapter for--self-test.BREAK-THE-RATCHET.md,npx modonome break-the-ratchet): a public adversarial challenge. Submissions are never executed — the judge reads a.patchas plain text and hands it to the ratchet's own--diffmode, itself a pure text analyzer. Seeded with one real, honest hall-of-fame entry demonstrating the ratchet's documented expected-value-drift blind spot.checkerproof/,npx modonome checkerproof): a seeded-defect benchmark measuring the independent checker's actual catch rate on semantic weaknesses (expected-value drift, cross-file assertion migration, vacuous-in-spirit assertions, scope creep) the deterministic ratchet cannot see. Advisory only, always exits 0, skips rather than fabricates a score when no model is reachable. First evidence file is a real, live run against the repo's configured checker model: 5/5.Part 3: a live-tested arming gate. Ran the full two-key arming mechanism (
autonomy_enabled+dry_runin config, plus the authoritative env-onlyMODONOME_ARMED) for real, with--execute, at the default$0budget. Confirmed the budget gate correctly refuses to spend before any network call. Captured as evidence, with honest documentation of what remains open (a live maker/checker model exchange needs a reachable free or local model endpoint, or a deliberately raised budget — seeOWNER-ACTIONS.md).A closing claims audit (
docs/audits/claims-audit-2026-07-08.md) re-verifies this work by re-running the actual commands rather than trusting the commit messages that made them.Governance checklist
npm run verifypasses locally (drift guard + style check + tests + AgentProof)fixtures/ratchet-diffs/, tests added intests/ratchet.test.mjsprompts/roles/checker.txt):npm run build:promptregenerated the bundleTest evidence