Skip to content

v1.2.0 — Procurement-defensibility hardening

Choose a tag to compare

View all tags
@alex-jb alex-jb released this 29 Jun 02:46
v1.2.0
2278374

Single-day cluster (2026-06-28): turn the "we're safe / multi-provider / OAuth-ready" positioning bullets from claims into mechanically-verifiable tests + opt-in production gates.

15 commits · +112 tests (196 → 308) · 0 fail

Added

  • Per-cell benchmark regression gate — CELL_HISTORICAL_FLOORS frozen from n=6 history, 5-point tolerance. The 87 ± 3 aggregate can hide single-persona collapse; this gate trips process.exitCode=2 when any persona × scenario cell drops more than 5 points below its historical min.
  • MCPTox / OX Security 2026 canary suite (test/mcptox-canary.test.js) — 28 contract tests covering 6 attack categories (instruction injection · trade-execution verb injection · echo-back probe · oversize buffer · HTML/script injection · nested-JSON auth bypass) × 4 invariants (verdict enum-bounded OR cleanly rejected · enforceAnalysisOnly passes · canary never leaks · response shape pinned). Also covers MosaicLeaks-class multi-turn leakage.
  • MCP Enterprise OAuth (EMA) scope scaffold + live wiring — lib/auth/oauth-scaffold.js declares 3 frozen scopes (shadow:read / shadow:council / shadow:admin). Opt-in SHADOW_REQUIRE_BEARER=1 makes /api/loan-council return 401 + WWW-Authenticate on missing claims, 403 on insufficient scope, 200 on match. Works with OAuth2 scope strings and Azure AD scp[] arrays.
  • GLM-5.2 contract tests (test/glm-call.test.js) — 12 tests with mock-fetch ($0 GLM credits): Bearer header, snake_case max_tokens wire field, system-then-user order, error paths, base-URL pin.
  • Catalog-as-code install registry — installer/tools.json + scripts/check-tools.mjs + test/tools-catalog.test.js. Pattern adapted from msitarzewski/agency-agents (117k stars). 5 MCP hosts × 6 tools × \$server_contract declared in one JSON file. Bidirectional CI drift gate.
  • bin/install.mjs one-line installer — reads installer/tools.json, detects which MCP hosts are installed, JSON-merges the correct config (preserving sibling MCP servers). --dry-run shows exact bytes, --all installs every detected host.
  • GLM vs Sonnet A/B benchmark harness (eval/glm-vs-sonnet-ab.mjs) — 5 voice-prompts × 2 providers × N runs, deterministic structural scoring matching benchmark/runner.js, envelope-skip built-in.
  • OCR live-smoke envelope-skip — Anthropic / Mistral usage-cap / quota errors are treated as t.skip() not fail. Pins verbatim 2026-06-28 Anthropic wording.
  • Audit-guardrail edge-case pins — all 12 forbidden verbs individually, case-insensitive, word-boundary anti-FP, ViolationError shape, nested-object scan.

Changed

  • README EN + zh-CN: badge from 154 → 308 tests, agentic score auto-computed
  • docs/positioning-vs-anthropic-fs.md: added "Multi-provider isn't sales copy — own-dogfood evidence (2026-06-28)" section with two real same-day fixes shipped in response to hitting our own Anthropic quota cap
  • Live demo URL refreshed to current production deploy (old URL was GC'd by Vercel)

Procurement contract

A bank's procurement team that wants to verify Shadow's 2026 named-threat coverage can grep these files in under 10 minutes:

  1. lib/audit-guardrail.js — 12-pattern regex output gate (Schema-Layer Safety)
  2. lib/run-loan-council.js — if (loan.fico < CREDIT_THRESHOLDS.FICO_FLOOR) return { verdict: "block", ... } (Determinism Floor)
  3. installer/tools.json — frozen install-target × scope catalog (EMA-ready surface)

Plus the four test files for the corresponding mechanical proofs:

  • test/mcptox-canary.test.js (MCPTox / MosaicLeaks)
  • test/oauth-scaffold.test.js + test/oauth-loan-council-wiring.test.js (EMA)
  • test/glm-call.test.js (multi-provider)
  • test/tools-catalog.test.js (catalog drift)

Distribution

Assets 2
Loading