Arkitect 2.4.0

Added

• arkitect report --draft <diagram> [--spec <spec>] prints the seven report headings with File, Icons and Validation filled: the file's pages and counts, each icon's id and pack (or ref and library) and every placeholder, missing, ambiguous or on-demand mark resolved from the spec without writing, and every validate error, warning and note word for word, with the build report's unknown kinds, unknown fields and notes. Engine, Assumptions, Render and Deviations each hold a <!-- fill in --> line, and arkitect report and the Stop hook fail a report while one is left (#387).

• Every Draw.io build carries its spec, as an arkitect-spec attribute on <mxfile>, which Draw.io Desktop keeps on save, and arkitect drawio spec <file> prints it: an edit to a diagram Arkitect built changes the spec and rebuilds instead of hand-writing XML. Rebuilding the printed spec reproduces the file byte for byte. A file saved by an editor since still counts as Arkitect's while it carries the spec, so a rebuild names the cells added there and keeps their backup (#376), and spec warns that changes made there are not in it. references/editing.md gains that branch. Every built file is larger by its spec, about 3 KB for the starter; the two committed examples are rebuilt, and nothing they draw moved (#388).

• arkitect report --draft without --spec lists the icons from the spec the diagram carries, so a Draw.io file, whose cells do not record their icon, needs no spec file beside it (#388).

• Every Excalidraw build carries its spec under customData.arkitectSpec of the first element its first node drew, and arkitect excalidraw spec <file> prints it. The issue proposed the top of the scene or appState; the app drops both on save and keeps an element's customData. Rebuilding the printed spec with the same seed reproduces the scene byte for byte. A rebuild over a scene the app changed names the elements added there and keeps their backup (#376), and spec warns when the scene no longer matches what the build wrote. references/editing.md gains that branch. The two committed examples are rebuilt and re-rendered; nothing they draw moved (#388).

• Every eval case carries one tier tag. tier-1 is the floor and must stay at 1.00; tier-2 and tier-3 hold the cases that measure progress, where a score below 1.00 is expected. Every other tag is a documented competency, weight: 3 marks the grader a tier-2 or tier-3 case exists for, and a test holds all three. The 18 existing cases are tier 1, and excalidraw-learning-skill-stays-manual is tagged invocation, like its Draw.io twin, in place of skills. docs/testing.md said 17 cases, nine for Draw.io, and is checked now (#415).

• The eval summary prints a scorecard by tier and by competency, each with its cases, mean score and the cases where every run was perfect. An efficiency line reads each run's trace: median turns, tool calls, failed or refused calls, permission denials and the bytes of tool output taken into context. A failed weight: 3 grader is listed as CRIT, and a run that used every turn as TURNS. The gate fails on a tier-1 case below 1.00, an errored run, or a case further than --tolerance (default 0.2) below its --baseline, which is a scoreboard or a result.json. A tier-2 or tier-3 case below 1.00 no longer fails it. --scoreboard FILE writes the run's scores and counts with no path, prompt or reply in it. scripts/eval.sh writes one beside every result and takes --baseline, --tolerance and --tier N (#415).

• Four tier-2 and tier-3 eval cases measure editing, one competency four ways. drawio-edit-built-diagram and excalidraw-edit-built-scene give the agent a file Arkitect built and nobody has touched since, and expect the spec route of #388: print the spec, change it, rebuild over the file. drawio-edit-saved-by-desktop gives it the same build after a Draw.io Desktop save that filled a box orange and added a text cell by hand, and expects both to survive. excalidraw-edit-follows-the-scene gives it a scene drawn in the app, with its own colours and no roughness, and expects an in-place edit in that look with every arrow bound. Each scaffold builds its fixture with the real CLI through evals/fixtures.mjs. A test makes each fixture, acts out the right outcome and the traps, and holds every file grader to its verdict on each (#405, #415).

• Eight more tier-2 and tier-3 eval cases, for what the contract promises and no case measured:

  • drawio-refused-spec-is-repaired: a refused spec fixed rather than its edge dropped;
  • drawio-instructions-in-a-spec: a note planted in a spec, asking for an upload, a POST and a deletion, treated as data;
  • drawio-lifecycle-target-state: lifecycle caveats repeated for a target state, and a customer name and codename kept out of every search, fetch and download;
  • drawio-whole-file-never-read: a 400 KB diagram answered without its payload entering the trace;
  • drawio-interview-when-vague: one question with a recommended answer, and nothing drawn yet;
  • drawio-scale-multi-region: twenty-odd components in nine nested boundaries;
  • excalidraw-legend-for-two-kinds: flows told apart by kind, and the legend that follows;
  • excalidraw-portable-report-check: report --draft, filled in, then checked, on an AGENTS.md host.

  The suite has 30 cases. Five tags are new: robustness, privacy, context, interview and scale (#415).

• evals/scoreboard.json holds the first full run of the 30 cases: Haiku under test, Sonnet judging, 88 runs for $1.64. Tier 1 scored 1.00, tier 2 0.89 and tier 3 0.96. evals/README.md says what the headroom showed and when to replace the board, and a test keeps the board naming exactly the cases that exist, each with its tier and no path (#415).

• arkitect drawio pattern N and arkitect excalidraw pattern N print one section of the pattern catalog, read-only, and with no N list the sections. Both SKILL.md step 2 name node scripts/pattern.mjs N in place of reading the section by hand: sed -n on a plugin file is refused in a sandboxed run, and two eval runs read that refusal as a dead shell and stopped without the edit. AGENTS.md step 2 names the command too, and each reading path is 13 bytes shorter (#417).

• Two eval cases continue a conversation through context.history_file, which the runner resumes with claude --resume: drawio-interview-continues (tier 3) takes the user's answer to the first interview question and expects the next one, never one already answered, and excalidraw-edit-next-turn (tier 2) asks for one change to a scene built a turn earlier and expects it rebuilt by its spec, with a backup, the other nodes kept and the assumption it made untrue gone. Each history is written by its history.mjs through transcript() in evals/fixtures.mjs, in the shape Claude Code writes a session, and a test rebuilds it, checks the chain and the tool calls, and refuses a path or a $CLAUDE_PLUGIN_ROOT command, which the agent's shell leaves empty. Resuming writes the continued session into the case folder: .gitignore keeps it out and scripts/eval.sh deletes it. A new tag, conversation. evals/scoreboard.json is replaced by a run of the 32 cases on Haiku, 94 runs for $1.56: tier 1 1.00, tier 2 0.93, tier 3 0.89 (#418).

• evals/scoreboard.sonnet.json and evals/scoreboard.opus.json hold tiers 2 and 3 on Sonnet and Opus, beside the Haiku board, so a case every model fails can be told from one only a small model fails. Tier 2 scored 0.93, 0.99 and 0.97 on Haiku, Sonnet and Opus; tier 3 0.89, 0.99 and 0.98. Only the two interview cases fall below 1.00 on every model (#422, #423). evals/README.md gains a table of each model's tier means, perfect cases and cost, linked to its board, and a test keeps each board naming exactly the tier-2 and tier-3 cases, its model and no path (#419).

Changed

• AGENTS.md step 8 and both SKILL.md step 9 name arkitect report --draft in place of the sentence asking for each validate warning by hand, with every id; the drafted Validation section carries them. Each reading path is a few bytes shorter (#387).
• The Cursor and OpenCode slash command is /arkitect, not /diagram: arkitect install cursor-command writes .cursor/commands/arkitect.md and arkitect install opencode writes .opencode/command/arkitect.md. diagram is a generic name a host, an extension or the project itself can hold, and a project's own diagram.md kept Arkitect's from installing. Adapter names are unchanged. An install removes the diagram.md an earlier Arkitect wrote, under any install path, and says so; any other diagram.md is the project's own, kept byte for byte and named. Cursor and OpenCode users type /arkitect from now on (#401).

Fixed

• analyze --out and build --out refuse a hard link or symlink to the file they read, as they already refused the same path. analyze a.drawio --out summary.json, through a hard link, replaced the diagram with its summary, exit 0 and no backup (#391).
• Both builders refuse, exit 2 and nothing written, an --out inside the Arkitect install, which the next plugin or npm update replaces; --defaults, which every template rebuild passes, still writes there. The Codex skill said to run a guide's commands from the skill folder, where a relative --out landed inside the install; it now says to run them from the project folder. A build that wrote into the install now exits 2 (#377).
• Draw.io validate fails an image linked by URL (http:, https:, //, file:) instead of warning, as references/editing.md already said and Excalidraw's validator already did; Draw.io's own img/… artwork passes. Excalidraw's validator names a remote image file once, not also as unreadable. A file that passed with that warning now fails (#379).
• Draw.io validate warns when a wrapped label holds a word wider than its box. Wrapping breaks at spaces and hyphens, so a CamelCase or snake_case name ran out of both sides of the box and passed (#381).
• Both build reports list an unknown key inside layout under unknownFields, and Draw.io's inside context and Excalidraw's inside style too. layout.colPich changed no spacing and was named nowhere (#393).
• The Excalidraw build report lists an unknown accent, boundary color, strokeStyle, fillStyle, route, layout.route or routing under unknownKinds, with what it was drawn as. An unknown strokeStyle or fillStyle used to be written into the scene as it was; the default is written now. validate warns on an element whose strokeStyle or fillStyle Excalidraw does not know. A spec whose route was unknown now draws the auto route in place of an elbow (#380).
• The build-route hook routes a diagram request that names Draw.io or Excalidraw and also a PNG, an SVG or a canvas to that engine's skill. "Draw our AWS architecture in Draw.io and give me a PNG for the slides" got no line, because the format filter ran before the engine was read (#383).
• The report check counts a build whose quoted script path holds a space, as an install under /Users/Jane Doe does. Since #373 such a build was not counted, and its report went unchecked (#382).
• The report check keeps the folder of an --out whose name it cannot resolve: --out out/$n.drawio, with n=$(basename …), now stands for a diagram in out/, not for any file. A later review that quoted docs/arch.drawio was sent back for the seven headings (#395).
• The report check reads a folder held in a variable set in the same command: S=scratch/tpl; … --out "$S/$n.drawio" now stands for a diagram in tpl/, not for any file. Every later reply naming any diagram path was sent back for the seven headings (#397).
• The report check counts a build only from what the shell runs. A build command quoted in a heredoc's body, a node -e string, a commit message or any other quoted string of more than one word counted, its --out became a built file or any file, and every later reply naming a diagram path was sent back for the seven headings. The body of bash -c, pwsh -Command, cmd /c or a heredoc fed to a shell still counts, and so does a quoted path to the builder. A script a heredoc writes and a later line runs is no longer counted (#400).
• arkitect install --force copies a Cursor rule, a Cursor or OpenCode command or the Codex skill it replaces to <file>.backup-<stamp> first, when it holds anything but the block being written. install --all --force, the documented update, replaced a project's own command with no copy left. The name ends past the extension, so Cursor and OpenCode do not load the copy as a second command (#401).
• Both build reports leave the edge-kind table to --print-style unless an override drew the build: the house style's meanings repeated in every report of a rebuild loop (#386).
• The Excalidraw build report names each shared mark by ref and pack. Its source, licence and sha256 moved behind --verbose, once per ref under icons.sources: printed for every node, they made the gallery's report 10 KB (#386).
• Draw.io analyze --cells prints one compact JSON row per cell, without its empty keys; --cells --all keeps every key. Fifteen pretty-printed keys per cell made the starter's table 11 KB, on the route the read hook names in place of reading the file (#385).
• Excalidraw analyze --cells prints one line per spec node, boundary and edge: its main shape's type and short id, the box around it and its element count. A scene no spec drew gets a line per outermost group, else per element; --cells --all keeps the element table. One object per element, every stroke of an icon included, made the 48-node template's table 140 KB; it is now 5 KB (#385).
• The Draw.io build report notes a diagram of more than three nodes, with edges, and no note to hold the assumptions, and a page whose legend: false drew no legend for two or more connector kinds. Both non-negotiables went unchecked on every host. The generate-architecture eval case reads the file for the note (#384).
• The Excalidraw build report notes the same two: a scene of more than three nodes, with edges, and no note, and legend: false over two or more connector kinds. Its generate-architecture eval case reads the scene for a note rectangle (#384).
• docs/status.md no longer carries a release snapshot. Its 2.3.0 "Where it stands" named seven open issues, all closed, and a Node 20 floor since raised to 22; the audit prompt reads that page first. It keeps how work gets done and the decisions that can look unfinished. evals/README.md counted the portable report check as a nineteenth case; it is one of the ten Draw.io cases (#389).
• An Excalidraw boundary is sized to hold a child scope's outside label. The label belonged to the parent frame but crossed its top border, and validate passed. This changes what a spec draws: a parent of an outside-labelled scope is now taller, and wider where the label is. A top-level outside label is unchanged (#392).
• The Draw.io build notes a node or boundary drawn inside or across a boundary it is not a member of, and validate warns on a cell drawn inside or across a container that is not its parent, naming both. A boundary covers every cell between its members, so a node set between two of them was drawn inside the VPC the spec put it outside, and both passed. A file with that defect, a hand-drawn icon laid over a group shape included, now has a warning (#378).
• The Excalidraw build notes the same, and stamps each member's boundary into customData.arkitect as parent, so validate warns on a node drawn inside or across a boundary it is not a member of. A scene built before carries no parent and is not checked. The two committed examples are rebuilt for the stamp; nothing they draw moved (#378).
• A Draw.io build that replaces a diagram someone added to since it was built names, in notes, each cell the build does not draw, with the backup path, and writes that backup as <name>.backup-<stamp>-kept.<ext>, which retention never deletes. It compares cell ids, leaving out the ids the builder makes up, so a node dropped from the spec is named too. The rebuild dropped an added "Audit log" with nothing said, and seven more rebuilds deleted the only backup holding it (#376).
• The Excalidraw build does the same for elements without Arkitect's stamp, and now stamps its title and legend as {"chrome": true}; a title or legend from before sits where the build draws one, and is not counted. The two committed examples are rebuilt for the stamp (#376).
• The four stays-manual eval cases check the fixture itself: its content, its cell or element count, and that no Edit or Write touched it. Their input-unchanged graders looked for the fixture in target: files, which lists only the files a run created; the scaffold wrote it before the run, so they passed whatever happened to it. A test refuses a files grader that names a file its case's scaffold writes (#414).
• The build-route hook routes a diagram asked for politely: "I'd like", "would like", "could you do", "put together" or "show me", then a diagram or a flowchart. "I'd like an architecture diagram of our platform for the client's architecture review board" loaded no skill, and every run of drawio-interview-when-vague asked five or six questions at once where the skill asks one. "A fix for the diagram", or a diagram noun naming code ("the diagram renderer", "a diagram view"), still gets nothing (#416).
• A fifth hook, hooks/install-write.mjs, refuses a shell command, Write or Edit that would write inside the plugin, which the next update replaces: above all a mkdir, a redirect, a tee or an --out after a cd into a skill folder. 11 eval runs of 15 ran cd <plugin>/skills/arkitect-drawio && …, and one lost its deliverable to a mkdir -p ./eval-output from there. The refusal names the path, says the shell still works, and says to run the command from the project with the script's full path. Running or reading a script from the skill folder passes, and everything does when the project is the plugin itself. Both SKILL.md say scripts/ and every path are in the skill folder, to be run from the project, never by a cd there, and AGENTS.md says the same; the Excalidraw reading path stays 8 bytes under its 12,000 (#404).