Skip to content

v6.6.0

Latest

Choose a tag to compare

@github-actions github-actions released this 23 Oct 07:39
· 2 commits to main since this release
0ac54d5

An A/B's treatment can be given a second skill's directory, lint_vault reads four more Markdown shapes as a reader does, and one measurement round read two ceilings. A minor because it adds an arcforge eval ab flag, --also-stage. Four decisions govern this release: D-066, which plans it; D-067, which designs --also-stage; D-068, which fixed the round before any session ran; and D-069, which records it. The benchmark snapshot is new (evals/benchmarks/2026-10-22.json, generated 2026-10-22T09:48:04Z) and is the release's only benchmark change. The round's trials ran at commit 61b19e46, and nothing under skills/, evals/scenarios/ or evals/fixtures/ changed after it. The round ran two preflights and no A/B, and preflight rows are not part of the benchmark, so the snapshot carries no row of either scenario the round measured.

Known limits of what is new. --also-stage has not run in a live trial: the round's one A/B that would have used it was not started, so the flag's evidence is its contract tests. The line that names the second skill imitates the host's listing of an available skill and is not one, so a treatment may read the second skill more or less often than a real session would. The second directory's digest is recorded, not pooled, so two runs of one scenario Version given different --also-stage directories, or a second skill edited between them, land in one pool, told apart only by each row's alsoStagedFilesHash. No A/B between two skill texts has run past its preflight. lint_vault still reads one shape differently from commonmark.js: a lazy line inside a list item whose content starts at column 5 or later; tabs are still not expanded as indentation, and HTML blocks are not modelled.

Closed, and not by a measured improvement: executing's handoff to dispatching (#184) and diagramming-obsidian's layout reference (#185) both close as findings at ceiling, so the handoff sentence and the current layout reference ship on their text, unmeasured against an alternative. lint_vault's block-model gaps (#256), which the 6.5.1 entry listed as still open, are fixed below.

Fixed

  • lint_vault read four rare Markdown shapes differently from a reader (#310, #256, D-066). maintaining-obsidian's audit script decides which [[wikilinks]] are links and which are examples inside code, and its model of a note's blocks had four gaps, each recorded as still open when #210 was fixed in 6.3.0. A block indented four spaces or more was not treated as code, so its example links entered the link graph; a fence opened inside a blockquote did not close when the quote ended, so real links after it were hidden; a lazy continuation line ended its list item; and an inline code span's search for its closing backtick ran across a heading, the start of a blockquote or list item, or a quoted blank line, so a span could swallow links in the next block. The script now keeps one model of the note's containers, read by both its fence walker and its span search, so the two cannot disagree: indented code is code, a quoted fence closes where its quote does, a lazy line stays in its item, and a span is bounded by the block it opens in. The audit's reference now says a link inside indented code is an example. The evidence is a contract test per shape under tests/skills/, each failing before the fix, and a differential fuzz against commonmark.js 0.31.2: over 150,000 generated notes, about 14,600 went from a wrong link set to CommonMark's and 1 went the other way; over 40,000 notes that include 2019.-style list markers, about 4,070 and 10. Every regression sampled is the column-5 limit above. If an audit report's link counts change after you upgrade, this is why: the new counts are the ones a Markdown reader would give.

Changed

  • New eval scenarios (#314, #315). Contributor-only: eval-executing-dispatch-acceptance Version 1 is new: both arms are injected with the same executing text, and the treatment alone gets dispatching staged with --also-stage, so the arms differ only in whether dispatching is within reach; its one assertion is that the subagent writing the task carries the spec's three acceptance criteria word for word. eval-diagramming-obsidian-layout-grid Version 2 asks for six equal services in one row without naming a width, so the baseline reaches the 180 px Primary width that Version 1 never did.
  • Product and plan records. Contributor-only: D-066 and D-067 open 6.6.0 with the spec sentences for eval B-19, B-20 and B-22 and obsidian B-5 (#308); the two offline design gates, D1 for the handoff and D2 for the layout grid, both ending MEASURABLE (#313, #312); D-068, the round's pre-registration, which also writes --also-stage's refusals into eval B-22 (#316); the round's ledger, its per-row audits, its preflight records and the regenerated benchmark (#318); and D-069, the result (#319).
  • Contributor rules and post-release records. Contributor-only: the 6.5.1 post-release check (#306) and its addendum recording that the website rebuilt only on the next push to main (#307); and a section of the contributor rules on running a live eval from the checkout: write nothing into the checkout while a trial runs, disable commit signing in a fixture Setup that commits, and a node_modules directory in the host's install cache is the host's (#320).

Added

  • --also-stage gives an A/B's treatment a second skill's directory (#309, #316, D-067). A trial runs with slash commands disabled, so no skill loads by name, and --stage-skill-files copies only the directory the injected SKILL.md sits in; a skill that hands work to another skill could therefore never reach it in a trial, which is why 6.5.0 found executing's handoff to dispatching not measurable. arcforge eval ab --also-stage <skill-dir> stages a second skill directory for the treatment arm alone, a copy of its own per trial under the same drift check and write guard as the first, and appends one line after the injected skill text: Skill `<name>` (<frontmatter description>) is available; its base directory is `<copy>` — read `<copy>/SKILL.md` to use it. The second skill's body is not injected; whether to read it is the agent's choice, as it is with the plugin loaded. The instruction to stay inside the trial directory names both staged copies as exceptions, and with one staged directory it is the sentence it was before, word for word. Every row of such a run, both arms, records alsoStaged: true, which is a pooling condition, so a run with the flag never pools or pairs with one without it, and alsoStagedFilesHash, the second directory's digest, which is recorded and not pooled. skillTextHash, skillFilesHash and the preflight key do not change, so an A/B with the flag is gated by the same preflight as one without, and every pool and record written before 6.6.0 keeps its identity. The flag needs --stage-skill-files and a skill-scope scenario; it is refused on eval preflight, which runs the baseline; and it is refused, before any trial, for a bare or empty value, an empty treatment skill text, a directory with no SKILL.md, and a SKILL.md whose frontmatter description is missing or is not a one-line plain or quoted value without an inline comment. docs/guide/eval-system.md gives the exact rules and docs/guide/cli-invocation.md lists the flag.
  • The 6.6.0 measurement round (D-066, D-068, D-069, #312–#319). The round spent 7 trial sessions of a cap of about 45, on opus[1m] / xhigh, Node v24.13.1, isolated, one run at a time, and read two ceilings. The handoff (#184) read baseline 3/3 in preflight: every baseline, with no dispatching body in reach, handed the task to a subagent whose brief carried the three criteria byte for byte with the spec's path, so on this task the handoff sentence has no delta to show. The layout grid (#185) read baseline 3/3: every baseline, given the old reference, chose 180 px boxes, noticed that the old 200 px grid then leaves 20 px against the 40 px minimum the same text states, and widened the pitch on its own. Both are BLOCK at ceiling, no A/B ran, and both issues close as findings, on k=3, one fixture and one model, for injected skill text. The finding says nothing about acceptance criteria that are long, scattered or not named in the task, or about a build, render or save of a diagram. Two infrastructure reruns were spent: the handoff scenario's fixture tried to sign its commit with the operator's git config, which the contained trial environment cannot reach, fixed by disabling signing in its Setup (#317); and an operator write into the checkout during a trial, which the write guard attributed to the trial. Of 16 scenario files that contain git commit, 2 disable signing, so others carry the same exposure (the eval-trial-no-commit-signing wish). The diagramming-obsidian verify-exit sentence stayed out of the round. Contributor-only: the operator audited every completed transcript against its row (docs/plans/v6.6/wp-d/audit.d1.md, audit.d2.md).