Skip to content

v1.14.0

Choose a tag to compare

@ChanMeng666 ChanMeng666 released this 13 Jul 06:22

v1.14 Tranches 1–2 + 4 — the measurement foundation, then the intent channel it licensed
(roadmap docs/research/2026-07-roadmap-proposal.md). Tranches 1–2 were repo-internal (eval/ and
CI only; the one core exception is the repair() purity fix under Fixed): fix the ruler before
measuring capability
. Gate G1's PASS then cleared Tranche 4, which DOES extend the published
surface — the intent channel below.

Added — Tranche 4: the intent channel (2026-07-12; core + CLI, zero new runtime deps)

  • src/intent.ts — the judge-v2 scoring core, lifted into the core package. A brief's
    checkable expectations as data (Intent), lowered to the shallow predicate kinds
    (room-count / room-exists / room-area / total-area / adjacent / reachable, plus a
    new gating room-windows), checked against describe() facts.
    validateIntent(source, intent){ ok, satisfied, total, violations, subscores, assertions, diagnostics } with typed, catalogued violations and Nickel-style spanless blame
    messages (intent /roomsInclude/1: no room matching concept "bathroom" …);
    intentFromJson (zero-dep pathed shape walker); feedbackForResult (deterministic
    per-violation correction prompts — advisory data, ADR 0005, never auto-applied).
  • src/intent-concepts.ts — the concept vocabulary, now production name resolution. Byte-
    mirrors the eval's table; a known concept resolves exactly as the eval judges (label →
    room_typeuses, token-bounded), an unknown one falls back to a literal
    id → label → uses → room_type match.
  • Eight catalogued codes: E_INTENT_ROOM_MISSING / _ROOM_COUNT / _ROOM_AREA /
    _TOTAL_AREA / _NO_WINDOW gate; E_INTENT_NOT_ADJACENT / _NO_DOOR / _UNREACHABLE are
    advisory (gate: false — scored, never failing ok; reachable blames by cause: no entrance
    NO_DOOR, cut-off rooms → UNREACHABLE). Promoting adjacency/reachability to gating stays
    parked on T3's still-open loop-vs-resampling question.
  • schemas/intent.schema.json (npm run gen:intent-schema, drift-tested, served by the docs
    site). Its field docs make Gate G1's two lessons normative: the area band conventions
    ("about/~/bare N m²" → ±10%; "at least N" → min only; qualitative words → no assertion) and
    the count discipline ("assert a room count only when the brief enumerates it").
  • CLI: arch validate --intent <intent.json> (the gate — exit 2 on a gating violation;
    composes with --graph/--strict; --feedback appends the correction prompts) and
    arch score <file> --brief <intent.json> (the continuous meter — satisfied/total +
    subscores, exit 0 on any successful measurement).
  • describe() windows gain facing: "N"|"S"|"E"|"W" (append-only; the outward normal of the
    window's host wall), and intent windows assertions take an optional facing.
  • The eval now consumes the same implementation (eval/assertions.ts/synonyms.ts are thin
    re-export shims; run.ts's Expect is the production Intent) — one judge, zero eval↔prod
    skew. JUDGE_VERSION stays "2", proven by a pinned fixture (eval/judge-fixture.json +
    test/eval-fixture.test.ts) that every corpus per-assertion judgment is byte-identical across
    the lift; the fixture is regenerated only to record an approved bump, never to green a red suite.

Added — release engineering: tokenless OIDC publishing (npm + MCP registry)

  • .github/workflows/release.yml — a v* tag push (or manual dispatch) publishes the core,
    then the MCP shim, to npm via OIDC trusted publishing with provenance (no npm token exists
    anywhere; each package carries a one-time Trusted Publisher registration on npmjs.com pointing
    at this workflow), then syncs the MCP registry with mcp-publisher login github-oidc
    also tokenless. Every step is idempotent (versions already on a registry are skipped, and the
    registry sync is guarded by the registry's own state), so partial failures re-run safely.
    Replaces the local granular-token publish flow that npm is deprecating through 2026–2027.
  • MCP shim 0.2.0 (@chanmeng666/archlang-mcp, registry entry updated): the validate tool
    takes an optional intent (gating assertions fail it; advisory ones score), a new score
    tool is the continuous satisfaction meter, and intent.schema.json ships as the
    intent-schema resource — the same intentFromJson/validateIntent/feedbackForResult path
    the CLI uses.
  • Provenance gotcha, recorded: npm E422-rejects a publish whose package.json
    repository.url casing differs from the OIDC-attested repo (ChanMeng666, not
    chanmeng666) — fixed in both package.json files + server.json.

Added — Gate G1 verdict + the L2 experiment harness (2026-07-12; still repo-internal)

  • Gate G1: PASS (eval/g1/ — generator harness, generated intents, double-blind scores,
    report). NL→intent-JSON per-assertion faithfulness on all 26 briefs: 154/157 (98.1%) vs
    93.4% per-assertion accuracy of direct .arch generation (one-tailed z = 2.08, p = .019;
    valid-only sensitivity variant below resolution — recorded). The intent channel (roadmap T4:
    src/intent.ts, arch validate --intent, intent.schema.json) is cleared for a future
    release. The generation prompt is oracle-isolated and test/g1.test.ts enforces it.
  • T3 harness: the L2 tier (eval/l2.ts + eval/l2-run.ts + .github/workflows/eval-l2.yml,
    npm run eval:l2). Diagnostic feedback loop (≤2 rounds, fed only compile/lint diagnostics +
    fix --dry-run previews + trimmed describe() — oracle-isolated) against an equal-token-budget
    i.i.d. resampling control
    (Olausson accounting, round-up favours the control), per-metric
    best-of, mean±σ across trials, pass@n/pass^n, retrying author + per-brief error isolation.
    Offline-tested (14 tests). The live experiment has not been run (cost declined) — the
    loop-vs-resampling question remains open and no loop-gain claim is made.

Added — judge v2: brief-grounded intent scoring

  • Intent-assertion scoring core (eval/assertions.ts, JUDGE_VERSION = "2"). scoreSource no
    longer greps the goldens for label substrings and golden-derived area bands; it lowers each brief
    to a small intent-assertion data structure — the shallow five kinds room-count /
    room-exists / room-area / total-area / adjacent / reachable — and checks the model's
    plan against those. The five-kind boundary is deliberately the one a future src/intent.ts can
    lift wholesale (Tranche 4 hook). Score gains append-only subscores / assertions /
    judgeVersion fields.
  • Oracle-isolated synonym table (eval/synonyms.ts, SYNONYMS_VERSION = 1). Room-label matching
    runs through a versioned, never-shown-to-the-model concept table with token-bounded,
    one-room-one-concept greedy assignment — so "wc"/"toilet"/"bath" resolve to one concept without
    leaking the answer into the prompt.
  • Brief-grounded area checks. Area is verified only where the brief states a number, in a
    ±10–15% band around the brief's number; all 20 golden-derived bands were deleted. Qualitative
    size words ("compact", "generous") carry no cap yet (a documented tier-b hook, added the day a
    real "oversized compact" instance appears).
  • Frozen corpus-review rubric (eval/rubric.md). Blind-drafted by an isolated agent, then frozen
    with the approver's decisions: room-count policy B (a ±1 surplus passes the gate only when the
    extra room is pure circulation, operationalized as planCirc >= expectedCirc + 1); one-room-one-concept
    greedy assignment; qualitative size words carry no cap. Adjacency and reachability score as
    subscores only, never a gate (Tranche 4 hook).
  • Corpus 22 → 26. Three prompts amended so every room count is brief-derivable
    (two-bath-flat, against-wall-bath, accessible-bath), plus a new per-room-area slice
    (sized-kitchen-flat, sized-bedrooms, sized-wet-room, sized-office-mix) so the area dimension
    is no longer total-only (H5) — every band carries the brief-source quote it came from.
  • L1 deterministic-tool gate (eval/faults/, eval/l1.ts, test/fault-injection.test.ts, in CI
    via npm test). Six single-defect fixtures (off-wall door/window/opening, furniture-through-wall,
    blocked-doorway, and a combined case) prove the l1Pipeline — a bounded machine-applicable-fix
    fixpoint (mirroring arch fix) followed by repair(), in the ADR 0011 → ADR 0006 order — heals
    every defect class deterministically and idempotently
    , and is a byte no-op on a clean golden.
  • --l1 live overlay (eval/run.ts, live runs only). Reports the deterministic dividend
    ΔL0→L1 (what fix+repair recover for free, zero extra API calls) with a per-row heal column;
    the committed baseline delta stays L0-only so cross-run comparisons don't silently fold the tool
    tier into a model score.
  • eval-live workflow inputs (.github/workflows/eval-live.yml): a --l1 toggle (default on) and
    the corpus-covering max default of 26.

Changed — live-harness integrity

  • Token budget & determinism. Anthropic max_tokens 2048 → 16384 (reasoning models spend
    thinking tokens out of the completion cap — the 2048 ceiling truncated output into false
    invalidity) with temperature: 0 and ephemeral prompt caching; the OpenAI path pins
    seed = 20260711 and records system_fingerprint (temperature deliberately not sent).
  • --budget <n>tok|<n>usd circuit breaker — a pre-call estimate halts the run before it
    overspends; skipped briefs are excluded from the denominators (over-estimating direction, verified
    price map).
  • Cross-judge guard. Baseline now carries a judge field and renderDelta flags any delta
    taken across a judge-version change as non-comparable — a judge change is never a capability
    result.
  • Calibrated judge-v2 baseline (eval/live-baseline.json; 26 briefs, gpt-5.5-2026-04-23,
    seed-pinned, GitHub Actions): L0 valid 25/26 (96%) · intent 13/26 (50%) · sound 4/26 (15%);
    the --l1 overlay lifts it to intent 18/26 (69%, ΔL0→L1 +5) · sound 6/26 (+2) — 7 briefs
    healed by 47 repair moves, 0 fix edits. The old 9% one-shot intent was ~55–65% measurement
    artifact
    (deep-dive H2); the calibrated 50% sits inside the roadmap's predicted 45–60% band.
    Residual true failures are dominated by physical violations (which L1 clears), with a few
    room-count and placeholder-label misses and one compile failure (the model inventing a label
    statement). Judge-v1 numbers are kept only as history.

Fixed

  • repair() is pure again across repeated calls. It mutated the shared parse-stage
    memo's AST in place (moving furniture at nodes), so a second repair() of the
    byte-identical source saw already-moved pieces and reported zero changes — same input,
    history-dependent output, violating ADR 0006's determinism promise. repair now works
    on a private deep clone of the parsed plan; compile() output was never affected
    (regression-tested in test/repair.test.ts). Found by the new fault-injection L1 gate.