Skip to content

Priorities 3, 4 & 7: MCP/skill governance, agent permissions, readiness benchmark#58

Merged
JRS1986 merged 1 commit into
mainfrom
feature/governance-and-eval
May 18, 2026
Merged

Priorities 3, 4 & 7: MCP/skill governance, agent permissions, readiness benchmark#58
JRS1986 merged 1 commit into
mainfrom
feature/governance-and-eval

Conversation

@JRS1986

@JRS1986 JRS1986 commented May 18, 2026

Copy link
Copy Markdown
Owner

Summary

Implements priorities 3, 4, and 7 from the maintainer's brief. Priorities 5 (worktree) and 6 (memory) remain explicitly out of scope.

Priority 4 — Machine-readable permissions

`coding-scaffold permissions write` generates `.coding-scaffold/agent-permissions.json` with the canonical shape from the brief (filesystem read/write/deny, shell allowed + requires_approval, network=disabled_by_default, mcp policy defaults). Project-aware: Python repos get `pytest`/`ruff` in the allowed list, Node gets `npm test`, etc. Idempotent — `--force` to regenerate.

Priority 3 — MCP governance

Command Effect
`mcp policy init` Write team policy at `.coding-scaffold/mcp-policy.json` (approved/denied lists, package-pinning default, review-required capabilities).
`mcp scan` Inspect `opencode.json` and `.claude/settings.json`, parse server definitions, flag remote / unpinned / risky-launcher / broad-fs-arg / unapproved / denied / review-required-capability.
`mcp snapshot` Write `.coding-scaffold/mcp-snapshot.json` recording the current server set with stable per-server fingerprints.
`mcp diff` Compare current scan with snapshot; report added / removed / changed; exit non-zero on any drift (CI-gate-friendly).

No commands are executed. No network calls. Pure JSON config parsing.

Priority 3 — Reviewable skill packs

Command Effect
`skills new ` Scaffold `.coding-scaffold/skills//` with `SKILL.md`, `manifest.json`, `README.md`, `scripts/`, `tests/`.
`skills lint` Flag broad-usage language (`always use this`), hidden-instruction phrases (`do not tell the user` — error severity), undeclared capabilities, missing required sections / manifest fields, invalid `risk_level`, placeholder owner, drift since CHECKSUM.
`skills approve ` Compute `sha256(SKILL.md
`skills export ` Bundle the skill directory into a `tar.gz` for inter-team sharing.

Priority 7 — Readiness benchmark

Command Effect
`eval init` Write `.coding-scaffold/eval-config.json` with per-check toggles. Optional — `eval run` works without it.
`eval run` Run all enabled checks; write `.coding-scaffold/eval-report.json`; exit non-zero when any check fails.
`eval report --cached` Re-print the most recent report without re-running.

Checks (deterministic, no shell, no LLM): build/test/lint signals, agent instructions exist, policy pack present, non-empty deny list, PR template present, MCP policy present when MCP is detected, session-trace location, context lint clean, context budget under limit.

The benchmark scores readiness — it does not measure model intelligence. The point is to confirm the repo gives the agent enough to work with.

Architecture & constraints honored

  • Four new modules, each with a clear responsibility (`permissions.py`, `mcp.py`, `skills.py`, `eval_harness.py`). `cli.py` stays a dispatcher.
  • Deterministic output everywhere (sorted findings, stable fingerprints, sorted JSON).
  • Zero new dependencies, zero LLM calls, zero network calls, zero telemetry.
  • Backwards compatible: every existing command and generated file is unchanged.
  • All 12 new commands ship with both plain-text and `--json` output.
  • `mcp diff`, `mcp scan` (on error), `skills lint` (on error), `eval run`, `eval report` exit non-zero so they can gate CI.

Tests

  • `tests/test_permissions.py` — 8 new tests.
  • `tests/test_mcp.py` — 17 new tests (policy / scan flags / snapshot / diff / CLI).
  • `tests/test_skills.py` — 17 new tests (new / lint flags / approve / export / CLI).
  • `tests/test_eval_harness.py` — 13 new tests (well-configured project passes basics; empty project fails most; MCP-detected-but-no-policy fails; cached/JSON output).

Total: 266 passing (was 205, +61). Ruff clean.

Docs

  • New "Machine-Readable Permissions Artifact", "MCP Governance", and "Skill Pack Governance" sections in Security.md.
  • New "Readiness Benchmark" section in Team-Rollout.md.
  • CHANGELOG `[Unreleased]` covers all 12 commands.

Known limitations

  • MCP scanner only reads `opencode.json` and `.claude/settings.json` in v1. Adding `.codex/config.toml` requires a TOML parser; deferred.
  • Skill checksum covers `SKILL.md` + `manifest.json` only; `scripts/` and `tests/` aren't fingerprinted in v1 (rationale: those are reviewed by code-review, not by approval gate).
  • Readiness benchmark is a static check — it does not run a real agent session.

Out of scope (per the brief)

  • Priority 5 — Git worktree / checkpoint mode.
  • Priority 6 — Memory governance (MemPalace remains a backend choice for the knowledge base; structured memory classes / capture / promote / expire / audit are not in this PR).

…orities 3, 4, 7)

Priority 4 - Machine-readable permissions

src/coding_scaffold/permissions.py:
  permissions write    writes .coding-scaffold/agent-permissions.json
                       with filesystem read/write/deny patterns, shell
                       allowed + requires_approval lists, network=
                       disabled_by_default, mcp defaults. Lightly
                       project-aware (Python -> pytest/ruff,
                       Node -> npm test, etc.). Idempotent; --force
                       to regenerate.

Priority 3 - MCP governance

src/coding_scaffold/mcp.py:
  mcp policy init      writes .coding-scaffold/mcp-policy.json
  mcp scan             inspects opencode.json + .claude/settings.json;
                       parses servers; flags remote/unpinned/risky-
                       launcher/broad-fs-arg/unapproved/denied/
                       review-required-capability findings
  mcp snapshot         writes .coding-scaffold/mcp-snapshot.json with
                       server fingerprints
  mcp diff             compares scan vs snapshot; exits 1 on drift

  No commands executed. No network. Pure config parsing.

Priority 3 - Reviewable skill packs

src/coding_scaffold/skills.py:
  skills new <name>    scaffolds .coding-scaffold/skills/<name>/ with
                       SKILL.md, manifest.json, README.md, scripts/,
                       tests/
  skills lint          flags broad-usage, hidden-instruction (error),
                       undeclared-capability, missing sections,
                       missing manifest fields, invalid risk_level,
                       placeholder owner, checksum drift
  skills approve <n>   writes CHECKSUM (sha256 of SKILL.md ||
                       manifest.json) so future drift surfaces in lint
  skills export <n>    bundles the skill as tar.gz

Priority 7 - Readiness benchmark

src/coding_scaffold/eval_harness.py:
  eval init   writes .coding-scaffold/eval-config.json with toggles
  eval run    runs deterministic checks for build/test/lint signals,
              agent instructions, policy, denied files, PR template,
              MCP policy (when MCP detected), session trace location,
              context lint clean, context budget acceptable
  eval report prints the most recent report (--cached for last run)

  Reuses lint_context and inspect_context_budget; no model-intelligence
  scoring; no shell execution.

CLI

12 new commands wired through src/coding_scaffold/cli.py with --json
output on every one. Existing commands unchanged.

Tests

61 new tests across 4 test files (test_permissions.py, test_mcp.py,
test_skills.py, test_eval_harness.py). 266 total pass, ruff clean.

Docs

- New "Machine-Readable Permissions Artifact", "MCP Governance", and
  "Skill Pack Governance" sections in Security.md.
- New "Readiness Benchmark" section in Team-Rollout.md.
- CHANGELOG [Unreleased] covers all 12 new commands.

Zero new dependencies, zero LLM calls, zero network calls. Existing
behavior unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@JRS1986
JRS1986 merged commit 8b15b83 into main May 18, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant