chore: version packages by github-actions[bot] · Pull Request #2 · doidor/agentrig

github-actions · 2026-06-08T15:56:29Z

This PR was opened by the Changesets release GitHub action. When you're ready to do a release, you can merge this and the packages will be published to npm automatically. If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated.

Releases

@doidor/agentrig@0.6.0

Minor Changes

#1 dc7c740 Thanks @doidor! - First public release as the scoped package @doidor/agentrig: a meta-harness CLI that installs
best-practice agent harnesses into any repo and projects them to every agent surface (local +
remote). Includes automated Changesets releases with npm provenance, Node >= 22, and CI/release
status badges.

The eval found that the producer agent ignored AGENTS.md's wiki-logging rule (memory=0 on fix-failing-test). Root cause: the rule was at the bottom of the Critical Rules list ('Every mistake is a prompt bug...') and the action ('log to .agents/wiki/') was buried in the second sentence. Fixes applied: - Moved from position #5 to position #2 (right after 'instructions are the source of truth'). - Rephrased to lead with the imperative action: 'Log every gotcha to .agents/wiki/ the moment you hit it — not at the end, not in passing.' - Kept the skill-improver pointer for cases where a skill/rule should have prevented the gotcha. - Re-ran agentrig compile to re-project into CLAUDE.md + .github/copilot-instructions.md. New Critical Rules order: 1. Instructions are the source of truth (meta) 2. Log every gotcha to .agents/wiki/ immediately (NEW: was #5) 3. Self-verify before handoff 4. Never skip a state-machine gate 5. Respect hard limits Reads as a per-task discipline checklist: log gotchas → self-verify → honor gates → stay within limits. Honest follow-up (not in scope): the deepest fix would be a glob-scoped reflex rule in .agents/rules/ that fires whenever the agent edits source code, prompting 'did you discover anything non-obvious? log to wiki'. Position-bump alone is necessary but probably not sufficient — will only know for sure after the next eval run. Verified: - npm test -> 59/59 green - eval --static -> Install Completeness 100% / Quality Probes 100% - compile -> 12 surfaces re-projected - CLAUDE.md + copilot-instructions.md now lead Critical Rules with the wiki-logging imperative. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Three follow-ups from the markbook eval run. 1. self-verify skill — explicit baseline -> after evidence The skill now requires the agent to (a) capture a baseline output BEFORE editing related code, (b) capture the after output, and (c) surface the transition in the transcript. Added an explicit handoff checklist + a "did you hit any non-obvious behavior?" step pointing at the new log-gotcha skill. 2. log-gotcha skill — wiki-logging is part of the task, not a "good-to-have" New .agents/skills/log-gotcha/SKILL.md walks the agent through when to log, how to log (check wiki first, sharpen vs duplicate), the entry template, common failure modes, and explicit verification (git diff --cached --stat). The self-verify checklist now references it so the agent reflexively considers logging at handoff time. Manifest bumped to 0.5.0; new skill ships to target repos. The Critical Rule #2 position bump alone wasn't sufficient — confirmed by memory=0 reproducing on both agentrig AND markbook. 3. Bundled scenarios excluded from default eval (per user feedback) Generic bundled scenarios (add-small-feature, fix-failing-test, review-catches-bug) now carry "bundled: true" in scenario.yml. The dynamic eval orchestrator filters them out by default and prints an actionable message pointing at agentrig eval --scaffold. New flag: --include-bundled (added to BOOLEAN_FLAGS so it doesn't eat the positional path). Explicit --scenario <id> always runs the named scenario, bundled or not. Verified on agentrig: default run skips the 3 bundled and runs only agentrig-init-on-empty-repo + fix-checks-json-drift. Tests: 66/66 green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ss variant) The markbook re-run showed memory + self_verification only partially improved from skill-level rewrites — the producer (claude-opus-4.8) acknowledged lessons in summaries but didn't commit wiki entries, and reported final-green tests without surfacing a baseline. Pure skill-body prompting wasn't enough. Per the discussion, shipping option 1: an explicit checklist rendered as text at the END of the producer prompt, where LLMs weight instructions most heavily. Checklist enforces three items the eval scores on: 1. Baseline test output captured + surfaced in transcript 2. After test output captured + surfaced 3. Wiki entry committed in same diff if anything non-obvious 4. Diff is on-target (git diff --stat) Each item explains what a half-credit vs zero answer looks like, with concrete bad/good examples for the most common slips ("All tests pass." vs "baseline: 1 fail → after: 0 fails"). Critical design constraint: the checklist is ONLY rendered for the harness variant (when `variant !== "baseline"`). The baseline variant deliberately runs without this nudge so the harness-vs-baseline A/B still measures something real. If baseline got the checklist too, the "harness lift" delta would shrink to nothing. Also worth noting: this isn't "teaching to the test" — the rules already exist in AGENTS.md (Critical Rule #2 promotes wiki-logging, self-verify skill requires baseline→after). The checklist just makes those rules unmissable at the moment of action. Tests: 66/66 still green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…l lift (#11) * eval: P0+P1+P6 — schema validation, layer split, veto axes - P0: schemaVersion=2 on results, read-time validation, rename 'Harness Score' to 'Install Completeness', demote pre_merge hook from harness-eval to a cheap install-completeness check (nightly: harness-eval), capture producer model in run meta. - P1: model-family classifier (catches claude-sonnet-4.5 vs claude-sonnet-4.6), YAML-parsed state-machine DAG check (caught real invalid-YAML inline form, fixed the schema), frontmatter-keys-all across every skill, content quality probes (axes/checks coherence, unfilled placeholders, context.md present), split renderer into Layer A1 (completeness) + Layer A2 (quality probes). - P6: per-axis weight + veto in axes.json (new v2 schema, back-compat with v1); pass = aggregate >= threshold AND no observed=0 AND no veto<1.0; report prints failReason. Layer A1 stays dep-free (mjs has its own regex YAML reader + inline family classifier). Added 'yaml' as a real dep for src/core/audit.ts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * eval: P2 — fixture-based scenarios + deterministic oracle Replaces the 3 single-file 'invent your own task' scenarios with proper fixture-based benchmarks. New scenario layout: .agentrig/eval/scenarios/<id>/ scenario.yml — id, type, scope, oracle_axes, judge_axes prompt.md — exact task handed to producer fixture/ — mini-repo seeded into throwaway worktree (or fixture/baseline + fixture/change for review scenarios) oracle.yml — deterministic checks (cmd/diff_stats/diff_files/file_contains) judge_brief.md — post-scoring reveal for calibration (P5) src/core/scenario-runner.ts: locate, load, seed, captureDiff, parseDiffStats, runOracle (with env override + NODE_TEST_CONTEXT scrub), oracleAxisScores (min() per axis), oracleAxesToFlags (back to score.mjs). Four scenarios: - fix-failing-test — broken integer division, oracle checks suite green - add-small-feature — slugify SPEC + skipped acceptance tests, oracle checks un-skipped AND all 6 actually ran - review-catches-bug — baseline+change fixture, planted off-by-one in src/pagination.js:5, reviewer writes review.json which oracle scores for finding + blocking + false-positive - agentrig-init-on-empty-repo — DOGFOOD scenario (no LLM), tests AgentRig the product: init succeeds, audit >=80%, compile idempotent Knowledge version bumped to 0.4.0. Templates ship the 3 generic scenarios; the dogfood scenario stays AgentRig-only. Tests added: test/scenario-runner.test.mjs (12 tests). Updated existing tests for renamed JSON fields + weighted aggregation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * eval: P3+P4 — producer/judge isolation + paired sign-test compare P3 — producer and judge are now separate provider.startConversation() calls, not one conversation with two hats: - new src/commands/eval-dynamic.ts: orchestrator that for each scenario seeds a worktree, runs the producer there, captures diff+transcript, runs the deterministic oracle, then spawns the judge in a DEDICATED cwd containing only prompt/diff/transcript/oracle/judge_brief (NOT the producer worktree). - new prompt builders: buildProducerPrompt (handed to producer in the worktree) and buildJudgePrompt (handed to judge with explicit 'don't defer to producer' instructions). The legacy buildDynamicEvalPrompt is kept as a deprecated stub. - judge submits scores by writing <artifactsDir>/<scenario>.trial<N>.judge.json (provider-agnostic — no tool-calling dependency); orchestrator reads + maps to score.mjs --axis flags. - --producer-model / --judge-model / --allow-same-family CLI flags. - score.mjs save enforces producer/judge family divergence (rejects same family unless override is set; override is recorded). New record fields: producerModel, judgeModel, producerFamily, judgeFamily, allowSameFamily, trialIndex. Inline model-family classifier keeps score.mjs dep-free. P4 — single-trial HELPS/HURTS coin flip is gone: - --n flag (default 5 in baseline mode, 1 otherwise) drives per-scenario trials. - score.mjs compare now pairs trial i of variant with trial i of baseline, computes median delta + binomial sign-test p-value, prints verdict: HELPS / HURTS / INCONCLUSIVE (n<3 or p>=0.05 or |median|<0.05). - Per-trial records carry trialIndex; filenames disambiguate. - Test suite: 7 new sign-test/divergence cases including the n=7-wins/0-losses case that produces a stable HELPS (n=5 was borderline at p=0.0625). - Existing compare test updated for new {medianDelta, n, pValue, verdict} shape. Knowledge version still 0.4.0 (the eval/score.mjs is overwrite-policy already). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * eval: P5+P7 — judge calibration + CI workflow + doc rewrite P5 — judge calibration: - new .agentrig/eval/calibration/ directory ships hand-labeled rubric instances (transcript + diff + ground-truth axes) per type. Seed set: 3 instances (run x2, review x1) so the calibrate command isn't a no-op on day one; release-gate threshold is 10/type, documented in README. - score.mjs gains 'calibrate' subcommand: calibrate --instance <path> --judge-scores <path> --judge <model> calibrate --report reports % agreement (within ±0.5 tier) + signed bias per judge. Inline mini-YAML reader keeps the script dep-free. - src/commands/doctor.ts surfaces the calibration rollup and FAILS doctor (exit 1) when any judge averages below the 80% threshold. - Tested: perfect judge -> 100% / 0 bias; lazy 'all 1.0' judge on the 'no-verify' instance -> 33% / +0.83 bias (correctly penalizes over-leniency); good judge -> 100% / 0. - New manifest entry installs calibration/ into target repos (refresh: preserve so users' own labels aren't clobbered on update). P7 — CI + docs: - new .github/workflows/agentrig-eval.yml: nightly + workflow_dispatch. Runs harness + baseline variants with n=5 each, computes per-scenario lift via 'score.mjs compare --baseline baseline', uploads JSON artifacts, and posts/updates a tracking issue with the verdict table. CI's ci.yml continues to gate on Layer A only (cheap, deterministic). - RUBRIC.md rewritten around the three layers (A1 install completeness, A2 quality probes, B fixture-based dynamic eval). Explicit about what each does and does NOT prove. Documents producer/judge isolation, family-divergence enforcement, weighted veto aggregation, paired sign-test verdicts (HELPS / HURTS / INCONCLUSIVE), and the judge calibration gate. - .agents/skills/harness-eval/SKILL.md updated for the new flow. - README 'Evaluating the harness itself' section rewritten: three layers (was two), honest scope ('synthetic fixtures, not your PR workload'), shows the paired sign-test compare command, mentions calibration as the answer to 'how do you trust the judge'. Final state: 37/37 todos done across phases 0-7. Test suite 56/56 green. Static audit 100/100. Dogfood scenario (the only one that tests AgentRig the product, not generic agent behavior) passes deterministically with zero model calls in seconds. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: bring README + docsite in line with the rebuilt eval The eval rebuild (P0-P7 on this branch) renamed Harness Score -> Install Completeness, added Quality Probes as a distinct layer, made scenarios fixture-based, split producer and judge into different families with enforcement, and added a paired sign-test compare. The README + docsite were still describing the pre-rebuild eval — corrected here. - docs/evals.md: full rewrite around the three layers (A1 install completeness, A2 quality probes, B fixture-based dynamic eval). Documents producer/judge isolation, family-divergence enforcement, --n + paired sign test (HELPS / HURTS / INCONCLUSIVE), judge calibration + doctor gate, per-repo customization, when-to-run-what. - docs/commands.md: eval flag table now includes --producer-model, --judge-model, --allow-same-family, --n, --seed. Doctor + dashboard wording updated for the new score names. - docs/getting-started.md: command output references updated; explains what the full agentic run actually does (deterministic oracle + independent judge in a different family). - docs/index.md: feature card + guide card link sharpened. - README.md: dashboard bullet now mentions both scores; command-table eval row exposes the new flags. - knowledge/PRINCIPLES.md: 'harness audit' phrasing updated to match the install-completeness + quality-probes split. docs/principles.md regenerated. Verified: docs:build succeeds, rendered site/evals.html contains 26+ hits for the new concepts, zero stale 'Harness Score' / --compare / 35-checks mentions in the built HTML. 56/56 tests still green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(agents-md): correct broken https://github.com/ link The intro line in AGENTS.md said '[AgentRig](https://github.com/)' — a bare URL pointing at the github.com home page instead of the repo. Fixed in both: - AGENTS.md (this repo's own) - knowledge/templates/AGENTS.md (the canonical template that ships to every target repo via agentrig init) Re-ran 'agentrig compile' to refresh the projected surfaces (CLAUDE.md and .github/copilot-instructions.md); compile also picked up the full AGENTS.md body that previously hadn't been mirrored to those surfaces. Verified with a fresh init on a temp repo: the installed AGENTS.md now correctly links to https://github.com/doidor/agentrig. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * eval: default producer/judge models from the role roster `agentrig eval --dynamic` used to require both --producer-model and --judge-model on every invocation, which is verbose. Now it resolves them from the existing role roster by default — no extra config: Producer ← .agentrig/agents/developer.yml model: Judge ← .agentrig/agents/reviewer.yml model: The install-completeness audit already enforces that developer and reviewer use different model families (P1 'roles-distinct-families' check), so the default pair is guaranteed to clear the producer/judge family-divergence check at save time. Resolution chain (highest precedence first): 1. --producer-model / --judge-model CLI flags 2. AGENTRIG_PRODUCER_MODEL / AGENTRIG_JUDGE_MODEL env vars 3. --model (legacy, producer only) 4. roster yml files (new default) 5. provider default The resolved model + source ('← --producer-model' / '← .agentrig/agents/ developer.yml') is logged at run start and recorded in meta.json so anyone debugging a result can see exactly where the model came from. Common path is now just `agentrig eval --dynamic`. README, docs/evals.md, docs/commands.md, and the CLI help text all updated to reflect the defaults. Three new tests in test/eval-defaults.test.mjs verify (a) the roster files are read, (b) the default pair clears family divergence, and (c) missing roster files don't crash the resolver. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(eval): valid Copilot model id + pre-flight validation + monitor leak Three bugs surfaced by the first real eval run. 1. "Model gpt-5 is not available" The Copilot SDK exposes ids like gpt-5.4, gpt-5-mini, gpt-5.4-mini — there is NO bare 'gpt-5'. The default judge model in reviewer.yml was wrong. Fixed in both the local roster and the canonical template, plus every docs/skill that quoted the bad id. 2. Pre-flight model validation Added optional validateModel(id) to AgentProvider. CopilotProvider implements it via client.listModels() and returns a 'did you mean' list of same-prefix ids. runDynamicEval calls it for both producer and judge BEFORE any scenario starts, so a bad model id fails in ~2 seconds with an actionable hint, not after the producer burns 30 seconds of real model spend on the first scenario. 3. ActivityMonitor leak on judge spawn failure monitor.start() was called BEFORE the try/finally that owned monitor.stop(). When provider.startConversation() threw (because of bug #1), the monitor's setInterval was never cleared and it flooded the terminal with 'still working — 0 tool calls so far' lines for the user even after the orchestrator had returned and the report had printed. Fixed by moving monitor.start() and startConversation() into the same try/finally block, in both the producer and judge sections, and adding catch-all-and-warn on the judge side so a single failed judge doesn't kill the whole run (soft axes just get marked na for that trial). Verified end-to-end: - npm test -> 59/59 green - eval --dynamic --scenario agentrig-init-on-empty-repo -> oracle-passing - eval --dynamic --judge-model gpt-5 -> fails in ~2s with 'Did you mean: gpt-5-mini, gpt-5.3-codex, gpt-5.4, gpt-5.4-mini, gpt-5.5?' - Install completeness 100%, quality probes 100%. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore(models): upgrade roster to latest Copilot model ids | role | before | after | | --------- | ------------------- | -------------------- | | triager | gpt-5-mini | gpt-5-mini (kept) | | developer | claude-sonnet-4.5 | claude-sonnet-4.6 | | reviewer | gpt-5.4 | gpt-5.5 | | judge | claude-opus-4.5 | claude-opus-4.8 | state-machine.yml model_tiers also bumped: standard: [claude-sonnet-4.5, gpt-5.4] -> [claude-sonnet-4.6, gpt-5.4] premium: [claude-opus-4.5, gpt-5] -> [claude-opus-4.8, gpt-5.5] (The premium tier's bare 'gpt-5' was a leftover bug from the same root cause as the reviewer.yml fix — the previous sed missed state-machine.yml.) Developer (anthropic-claude) and reviewer (openai-gpt) remain in different families, so the roles-distinct-families quality probe still passes and the dynamic eval's producer/judge family-divergence check is still satisfied by the defaults. Updated example invocations everywhere they appear (README, docs/evals.md, docs/commands.md, RUBRIC.md, harness-eval skill) to claude-sonnet-4.6 + gpt-5.5. Mirrored every change to knowledge/templates/. Verified: - npm test -> 59/59 green - install-completeness + quality-probes -> 100% / 100% - eval --dynamic --scenario agentrig-init-on-empty-repo resolves new defaults from the YAMLs and dogfood oracle passes 1.00 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(eval): fold staged harness into baseline + readable report Two issues from the user's real eval run. 1. Harness staging was polluting the producer's diff. `stageHarnessInto` was copying AGENTS.md + .agents/ + .agentrig/ into the fixture worktree AFTER `seedWorktree` had already taken the baseline commit. Every staged file then showed up in `captureDiff()` as part of the producer's change → for a one-line src/math.js fix, the oracle saw '5081 added lines, 108 files touched' and the judge correctly (per its inputs) scored scope = maintainability = gate_compliance = 0. Fix: `stageHarnessInto` now runs `git add -A && git commit --amend --no-edit` after copying, so the staged harness becomes part of the baseline. captureDiff then only shows what the producer actually edited. Verified: same +1 line src/math.js edit now reports '+1/-1' on src/math.js only (was '+12770/-34 files' before). 2. Report was unreadable. The old human-readable report printed only scenario pass/fail + per-axis means — no failure reason, no failing axes, no evidence, no guidance on how to investigate. The user couldn't tell if 0.78 was good or bad. Rewrote score.mjs's report renderer to: - lead with 'X/N PASS, Y FAIL' summary - state the pass rule + veto axes inline - for each FAIL: show failReason next to the row, then list every failing axis with its issue code + evidence - flag weak axes (mean<0.8) and weakest axes (mean<0.5) in the per-axis rollup - close with a 'How to read this' block pointing at where to look in run artifacts to investigate a FAIL - explicitly note the overall aggregate is NOT harness lift (need baseline comparison for that) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore(models): bump every role to its family's top model + swap to keep families split User requested: developer -> opus latest judge -> gpt latest reviewer -> opus latest triager -> gpt latest The literal spec puts opus on BOTH developer and reviewer, which collides with the roles-distinct-families quality probe (P1) and the dynamic eval's default producer/judge family-divergence enforcement. To honor the spirit (top opus + top gpt across all 4 roles) without breaking either invariant, applied the swap variant — same 4 model ids, role assignment shuffled: | role | new | source | | --------- | -------------------- | --------------------- | | triager | gpt-5.5 | (was gpt-5-mini) | | developer | claude-opus-4.8 | (was claude-sonnet-4.6) | | reviewer | gpt-5.5 | (was gpt-5.5; kept) | | judge | claude-opus-4.8 | (was claude-opus-4.8; kept) | So developer (anthropic-claude) vs reviewer (openai-gpt) still diverges, and the dynamic eval's default producer/judge pair (read from those YAMLs) still passes the family check with no override. Honest expectations: bumping the producer from sonnet to opus will *probably* help slightly with the self_verification=0.5 finding (smarter agent more likely to show red->green explicitly), but is unlikely to fix the memory=0 finding by itself — the wiki-logging Critical Rule is just buried in AGENTS.md and a smarter model alone isn't enough to surface it reflexively. Persistent fix for that would be a dedicated 'log-gotcha' skill or rule trigger. Also bumped triager's comment to reflect the tier inversion (gpt-5.5 is premium, not cheap) — left a note that gpt-5-mini / claude-haiku-4.5 would be a sensible cost optimization once triage volume scales up. Verified: - npm test -> 59/59 green - eval --static -> Install Completeness 100% / Quality Probes 100% - eval --dynamic --scenario agentrig-init-on-empty-repo resolves 'Producer: claude-opus-4.8 ← developer.yml' + 'Judge: gpt-5.5 ← reviewer.yml', dogfood oracle passes 1.00. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs(agents): promote wiki-logging rule to Critical Rule #2 The eval found that the producer agent ignored AGENTS.md's wiki-logging rule (memory=0 on fix-failing-test). Root cause: the rule was at the bottom of the Critical Rules list ('Every mistake is a prompt bug...') and the action ('log to .agents/wiki/') was buried in the second sentence. Fixes applied: - Moved from position #5 to position #2 (right after 'instructions are the source of truth'). - Rephrased to lead with the imperative action: 'Log every gotcha to .agents/wiki/ the moment you hit it — not at the end, not in passing.' - Kept the skill-improver pointer for cases where a skill/rule should have prevented the gotcha. - Re-ran agentrig compile to re-project into CLAUDE.md + .github/copilot-instructions.md. New Critical Rules order: 1. Instructions are the source of truth (meta) 2. Log every gotcha to .agents/wiki/ immediately (NEW: was #5) 3. Self-verify before handoff 4. Never skip a state-machine gate 5. Respect hard limits Reads as a per-task discipline checklist: log gotchas → self-verify → honor gates → stay within limits. Honest follow-up (not in scope): the deepest fix would be a glob-scoped reflex rule in .agents/rules/ that fires whenever the agent edits source code, prompting 'did you discover anything non-obvious? log to wiki'. Position-bump alone is necessary but probably not sufficient — will only know for sure after the next eval run. Verified: - npm test -> 59/59 green - eval --static -> Install Completeness 100% / Quality Probes 100% - compile -> 12 surfaces re-projected - CLAUDE.md + copilot-instructions.md now lead Critical Rules with the wiki-logging imperative. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(eval): agentrig eval --scaffold generates repo-specific scenarios Closes the gap raised in user feedback: the 3 bundled scenarios are language-agnostic JS micro-fixtures, so the eval was measuring 'agent + harness on JS', not 'agent + harness on THIS repo'. New command: `agentrig eval --scaffold [--scaffold-count N]` (default 2). For each new scenario the agent must produce: .agentrig/eval/scenarios/<id>/scenario.yml (frontmatter) .agentrig/eval/scenarios/<id>/prompt.md (exact task) .agentrig/eval/scenarios/<id>/fixture/ (mini-repo, ≤10 files, REPO STACK) .agentrig/eval/scenarios/<id>/oracle.yml (deterministic checks) .agentrig/eval/scenarios/<id>/README.md .agentrig/eval/scenarios/<id>/judge_brief.md (optional, soft-axis hints) Hard constraints enforced via prompt + post-run validator: - May only touch .agentrig/eval/scenarios/ (no other files) - May only reference axis names already in axes.json - Must use the repo's actual test runner / package manager - Existing bundled scenarios are NEVER modified - Every new scenario must parse + locate-fixture cleanly before being marked valid Files: - src/prompts/index.ts: buildScaffoldScenariosPrompt with example scenarios + repo context + axis whitelist inlined. - src/commands/eval-scaffold.ts: orchestrator that discovers existing scenarios as templates, runs the agent in a single conversation, then walks new directories with the scenario-runner load logic + flags invalid output (without auto-deleting — operator can fix by hand). - src/commands/eval.ts: new --scaffold / --scaffold-count options in EvalOptions + branch in evalCommand. - src/cli.ts: --scaffold + --scaffold-count flags wired through, help text updated. Bug fix found while testing: --scaffold (and --allow-same-family) were NOT in cli.ts BOOLEAN_FLAGS, so 'eval --scaffold /some/path' would swallow /some/path as the flag's value and operate on cwd instead. Added both to the boolean set. Tests (5 new in test/eval-scaffold.test.mjs, total 64): - bundled scenarios still parse + locate cleanly - validator rejects unknown axes - validator rejects missing fixture/ - validator rejects malformed YAML - empty-repo --scaffold fails fast with 'No harness installed' Live verification: ran 'eval --scaffold --scaffold-count 1' against AgentRig itself. Agent produced 'fix-checks-json-drift' — a repo-specific scenario that tests the actual gotcha documented in AGENTS.md (manifest/checks.json coupling: a new artifact in manifest.json must have a matching check entry in checks.json or the audit silently skips it). Oracle correctly distinguishes broken state (3 FAILs on correctness/tool_discipline) from a hand-applied fix (all 7 PASS). Validates that the scaffold flow produces real, repo-specific eval signal — not just generic agent behavior on JS fixtures. Docs: docs/evals.md gains a 'Generating repo-specific scenarios' section; docs/commands.md flag table includes --scaffold. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * eval: fix memory/self-verify findings + filter bundled scenarios Three follow-ups from the markbook eval run. 1. self-verify skill — explicit baseline -> after evidence The skill now requires the agent to (a) capture a baseline output BEFORE editing related code, (b) capture the after output, and (c) surface the transition in the transcript. Added an explicit handoff checklist + a "did you hit any non-obvious behavior?" step pointing at the new log-gotcha skill. 2. log-gotcha skill — wiki-logging is part of the task, not a "good-to-have" New .agents/skills/log-gotcha/SKILL.md walks the agent through when to log, how to log (check wiki first, sharpen vs duplicate), the entry template, common failure modes, and explicit verification (git diff --cached --stat). The self-verify checklist now references it so the agent reflexively considers logging at handoff time. Manifest bumped to 0.5.0; new skill ships to target repos. The Critical Rule #2 position bump alone wasn't sufficient — confirmed by memory=0 reproducing on both agentrig AND markbook. 3. Bundled scenarios excluded from default eval (per user feedback) Generic bundled scenarios (add-small-feature, fix-failing-test, review-catches-bug) now carry "bundled: true" in scenario.yml. The dynamic eval orchestrator filters them out by default and prints an actionable message pointing at agentrig eval --scaffold. New flag: --include-bundled (added to BOOLEAN_FLAGS so it doesn't eat the positional path). Explicit --scenario <id> always runs the named scenario, bundled or not. Verified on agentrig: default run skips the 3 bundled and runs only agentrig-init-on-empty-repo + fix-checks-json-drift. Tests: 66/66 green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * prompts: inline pre-handoff checklist into the producer prompt (harness variant) The markbook re-run showed memory + self_verification only partially improved from skill-level rewrites — the producer (claude-opus-4.8) acknowledged lessons in summaries but didn't commit wiki entries, and reported final-green tests without surfacing a baseline. Pure skill-body prompting wasn't enough. Per the discussion, shipping option 1: an explicit checklist rendered as text at the END of the producer prompt, where LLMs weight instructions most heavily. Checklist enforces three items the eval scores on: 1. Baseline test output captured + surfaced in transcript 2. After test output captured + surfaced 3. Wiki entry committed in same diff if anything non-obvious 4. Diff is on-target (git diff --stat) Each item explains what a half-credit vs zero answer looks like, with concrete bad/good examples for the most common slips ("All tests pass." vs "baseline: 1 fail → after: 0 fails"). Critical design constraint: the checklist is ONLY rendered for the harness variant (when `variant !== "baseline"`). The baseline variant deliberately runs without this nudge so the harness-vs-baseline A/B still measures something real. If baseline got the checklist too, the "harness lift" delta would shrink to nothing. Also worth noting: this isn't "teaching to the test" — the rules already exist in AGENTS.md (Critical Rule #2 promotes wiki-logging, self-verify skill requires baseline→after). The checklist just makes those rules unmissable at the moment of action. Tests: 66/66 still green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(eval): no-hang on session.create failure + scope report to current run Three issues surfaced when the user ran eval --dynamic on agentrig: 1. Process hung after the report printed. When client.createSession() inside CopilotProvider.startConversation threw (because the model id worked in listModels but failed at session create — see Copilot SDK quirk note below), the CopilotClient that had already been start()'d was never stop()'d. Its child runtime kept node's event loop alive, so the CLI completed all the user-visible work, printed the report, and then sat there forever. Fix: wrap the createSession() call in a try/catch that calls client.stop() before re-throwing. Verified: scenario with a bogus model now exits in 5s with a clear "did you mean…" message. 2. Report showed scenarios that didn't run in this invocation. The user's run output listed 5 PASS/FAIL rows even though "Excluded 3 bundled scenarios" was logged and only 2 trials were actually scheduled. Root cause: renderSavedDynamicResults() shells to score.mjs report with no filter, which loads every JSON file under .agentrig/eval/results/ — including stale records from earlier invocations. So a default-filter run (which excludes bundled) still showed the bundled scenarios' last- known scores from a previous --include-bundled run. Fix: score.mjs report now accepts --run <id>. eval.ts passes the current runId so the post-run report only shows what this run produced. The bundled rows are gone from default runs. 3. Roster note: claude-opus-4.8 is upstream-flaky as of 2026-06-10. Probed every model via createSession() and found opus-4.8 fails with "Authentication failed: Failed to fetch OAuth user login" while listModels() still reports it. Reverted developer.yml to claude-sonnet-4.6 (works reliably) with a comment to revert to opus when the upstream issue clears. The eval's pre-flight validateModel() uses listModels(), so it can't catch this — that's a Copilot SDK quirk worth knowing about but not something agentrig can paper over. Verified end-to-end: dogfood scenario now completes in 9.4s with "PASS 1.00", clean exit, and report shows only the current run's row. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore(models): fall back to claude-opus-4.7-high (plain 4.7 also unsessionable) User asked to fall back to claude-opus-4.7 instead of sonnet-4.6. Probed every opus variant via createSession() and found that plain claude-opus-4.7 is ALSO not sessionable for this account today (same class of failure as 4.8). Sessionable opus variants right now: OK claude-opus-4.7-high OK claude-opus-4.7-xhigh OK claude-opus-4.7-1m-internal (flagged 'internal only' in id) OK claude-opus-4.6 OK claude-opus-4.6-1m Picked claude-opus-4.7-high — honors the user's 4.7 intent, public-looking name (no '-internal'), high-reasoning variant. Documented xhigh + 4.6 as alternatives in the role comment if -high ever degrades. Family invariant preserved (anthropic-claude developer vs openai-gpt reviewer); roles-distinct-families quality probe still passes. Updated .agentrig/agents/developer.yml, knowledge/templates/agents/ developer.yml, plus the model column in both agents/README.md tables. Verified live: dogfood scenario completes in 11s with Producer: claude-opus-4.7-high resolved from developer.yml, PASS 1.00. Static audit 100/100, 66/66 tests green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert "chore(models): fall back to claude-opus-4.7-high" The "opus is unavailable" diagnosis was wrong. Earlier in the session I had switched the active gh account to 'doidor' (to bypass a stale env-token shadowing issue when editing PR descriptions), and my subsequent createSession probes ran under that account. doidor doesn't have premium-model entitlements; tudorpopams does. Verified under the correct account: claude-opus-4.8, claude-opus-4.7, claude-sonnet-4.6, gpt-5.5 all sessionable. Restored developer.yml to claude-opus-4.8 (the originally-intended top opus). Removed the fallback comment. Lesson stored as user memory: for Copilot SDK / Copilot CLI access to premium models, the active gh account MUST be tudorpopams. Don't probe under any other account. Verified live: dogfood eval runs under claude-opus-4.8 + gpt-5.5, PASS 1.00, clean 9.5s exit. Static audit 100/100, 66/66 tests green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ds, add agentrig fix + --auto-fix (#15) Closes the entire feedback loop from the markbook update-experience report — fewer surgical edits after `agentrig update` on any target repo. What changed ============ 1. Marker populator (src/core/markers.ts). The  ... :end --> block in AGENTS.md is now rewritten deterministically by both `compile` AND `update`. `skillsInventory()` walks .agents/skills/ on disk (with manifest fallback) so user-added skills appear alongside AgentRig-bundled ones — surfaced by running on this repo's own 6 markbook-* skills. 2. YAML validation (src/core/validate.ts). `validateYaml()` parses every .agentrig/**/*.yml using `parseAllDocuments` so multi-doc scenario files are tolerated. `update` runs it post-install and exits non-zero on syntax errors. 3. Model-ID validation. `validateModelIds()` matches every .agentrig/agents/*.yml's `model:` against the Copilot SDK registry (live, when reachable) or a curated allowlist (offline fallback), with "did you mean…" suggestions. 4. `agentrig fix` + `agentrig update --auto-fix`. New deterministic repair engine (src/core/fix.ts): restores broken YAML from canonical, replaces unknown model ids with the safe `auto` fallback. Writes .bak backups; no agent / network needed. 5. `--diff` drift classification. Preserved files tagged 🔴 broken / 🟡 stale / 🟢 enhancement / ⚪ mixed with a per-severity summary at the top. 6. `update` enumerates added + preserved files inline (no more "11 files preserved" without naming them). 7. New files grouped + summarized by parent dir. 8. `doctor` install provenance + model-ID validation. Detects when the global `agentrig` binary resolves to a linked checkout vs the registry, and surfaces YAML / model-id validation findings; exits non-zero on any blocker. 9. Reconciliation history (state.json.reconciled[]). Records every preserved-file decision with a canonical hash so a future `update` skips re-prompting on files the user has consciously diverged on — unless canonical has drifted past the recorded hash. Audit changes ============= - New `marker-populated` check type in BOTH src/core/audit.ts and knowledge/templates/eval/static-audit.mjs (the in-repo audit) so they stay in lock-step. - `agents-skills-inventory` upgraded from a loose `file-contains` substring match to `marker-populated` — the audit now fails (correctly) if the block is empty, has unfilled placeholders, or omits any skill under .agents/skills/. - Knowledge bundle bumped 0.5.0 → 0.6.0; Changeset at .changeset/update-workflow-improvements.md. Docs ==== - README: new `fix` row + `--auto-fix` flag in Commands table; new "Self-healing the install" subsection. - docs/commands.md: extended `update`, `compile`, `doctor`; new `fix` section. - docs/getting-started.md: iterate block includes `--auto-fix` + `fix`. - knowledge/templates/AGENTS.md: notes that the skills-inventory marker is auto-populated. - .agents/wiki/skills-inventory-populator-enumerates-disk.md (new): gotcha log for the disk-enumeration fix (per critical rule #2). Verification ============ - `npm run build` clean under TypeScript strict. - CLI audit + in-repo `static-audit.mjs` both 100% / 100%. - Round-trip: broke state-machine.yml + injected `model: gpt-5` → both audits caught it → `agentrig fix` restored both → 100% recovered. - Doctor on this dev checkout correctly reports `linked checkout` + `npm latest: 0.10.0`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

chore: version packages

5318841

doidor merged commit 9ffb66f into main Jun 8, 2026

doidor mentioned this pull request Jun 10, 2026

Rebuild the eval setup: 3 honest layers + isolated judge + statistical lift #11

Merged

github-actions Bot mentioned this pull request Jun 10, 2026

chore: version packages #13

Merged

This was referenced Jun 10, 2026

feat(update,fix,doctor): auto-populate markers, validate YAML/model ids, add agentrig fix + --auto-fix #14

Closed

feat(update,fix,doctor): auto-populate markers, validate YAML/model ids, add agentrig fix + --auto-fix #15

Merged

Navigation Menu

chore: version packages - #2

chore: version packages#2
doidor merged 1 commit into
mainfrom
changeset-release/main

github-actions Bot commented Jun 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant