Pin the product Compile CLI contract - #18
Conversation
Replace the monolithic successor check with focused modules that exercise the zero-flag journey, retained artifact safety, and the packed executable.\n\nBind the eval corpus and evidence summary to generation-aware analysis, Publication recovery, corrected digest provenance, and the controlled local service rehearsal so contract drift fails CI.
ReviewThe checker work is good and I verified it against the pinned CLI. One finding: the packaged Skill still instructs three commands the CLI stack removes, and the summary's first bullet reads as though that were fixed here. Verification
The last row is the sanity check that matters: a checker that exits 0 silently could be doing nothing, and this one fails when pointed at the wrong tree. So "the exact checker already passed locally against the pinned worktree" reproduces. The packaged Skill still names removed commandsThis branch does not touch What the shipped
I confirmed the first is genuinely unavailable by running the new CLI: So once the CLI stack releases, an agent following the shipped instructions fails at three separate points, each with a bare "Unknown command" and no hint about what to use instead. This matters more than a stale doc because of what skills#16 built. The Skill is installable as a marketplace plugin, so this is not internal notes; it is text that reaches other people's agents and gets executed. Instructions that name a command are part of the interface, and this half of the interface has not moved. The wording invites misreading
Read cold, that says the obsolete references were replaced. What was replaced is skills#17 says the renewal is deliberately later work: "This branch deliberately does not alter the packaged Skill yet ... before the workflow and evidence stack is renewed." So this is sequencing rather than an oversight, and I am not asking for the Skill rewrite here. What is worth adding is the release constraint, stated plainly: the packaged Skill must not reach users between the CLI stack landing and the instructions being renewed. Right now that ordering lives only in the reader's head, and the two repositories release independently. A line in this PR's description, or better an entry in the README's boundary section, would make it something a person can check rather than remember. The blocked gate is handled honestly
Good. An ancestry gate is the right mechanism, since it prevents pinning a commit that could be force-pushed away or that never merges, and saying up front that it will be red avoids the alternative failure mode where somebody sees red CI and assumes the branch is broken. Worth confirming the gate distinguishes "not yet an ancestor" from "not a valid commit," since those need different responses and both would fail an ancestry check. The module split reads wellBreaking one checker into The description's claim that adversarial artifact, transport, state, status, Publication, provenance, materialization, and leak coverage is preserved through the split is the sort of thing a diff cannot easily show. The Distinguishing the two digests
This is the same conflation cli#19 fixed, and pinning it in the contract checker is the right place. It means the Skill's contract cannot silently drift back to requiring the two digests be equal, which is the bug the Movie Catalog journey surfaced. |
When your documentation is an APIHere is a situation that did not exist a few years ago, and it changes how you should think about docs. This repository ships a Skill: a folder of Markdown that gets installed into someone's Claude Code and tells an agent how to use the First Draft CLI. Instructions like: Meanwhile, in a different repository, the CLI removed So the shipped instructions name a command that no longer exists. An agent following them runs it, gets "Unknown command," and has to work out what to do with that. Why this is different from a stale READMEYou have shipped stale documentation before. Everybody has. Usually the consequence is that a human reads it, tries the thing, sees it fail, mutters, and finds the right answer another way. Annoying, low stakes, self-correcting. An agent reading instructions is different in three ways. It follows them literally. A human sees It has no other source. The human opens the CLI's The failure is not obviously a documentation problem. "Unknown command" looks like a broken installation or a missing dependency. The agent may retry, or try to install something, or report a confusing failure to the user. It will not usually conclude "my instructions are out of date." So instructions consumed by an agent behave less like documentation and more like a client library. And you already know the rule for client libraries: you cannot remove a command without coordinating with the things that call it. The version-skew problem, againIf you have shipped an API, this shape is familiar:
Same problem here, with prose in the client position:
The window between those rows is the problem, and it opens the moment the CLI is released. Note what makes it worse than an ordinary API break: the two live in separate repositories with separate release cycles. Nothing mechanical connects them. In one repository somebody deletes a command, tests pass, ship. In the other, a Markdown file still names it, tests pass, ship. Both repositories are individually green and the combination is broken. What this PR does about it, and what it does notThe PR updates the contract checker: a script that runs the real CLI and asserts it behaves as the Skill expects. That is genuinely valuable, because it is the mechanical link the two repositories were missing. What it does not do is update the instructions. The That is a deliberate sequencing choice; a sibling PR says the Skill renewal comes later. Worth understanding as a reader, though, because the summary's first bullet reads as if the references had been replaced, and what was replaced is the checker's expectations of the CLI. The technique worth stealingThe interesting idea here is testing your documentation by executing it. Most of us test code and proofread docs. When your docs contain commands, you can do better: [
[["--version"], /\A0\.1\.0-alpha\.2\n\z/],
[%w[plan --help], /compile\s+Compile and publish the current Foundation Plan/],
[%w[compilation status --help], /compilation status <compilation-id> \[--wait\]/],
]Run each command, assert the output matches. Now a removed command fails a check rather than a user. You can approximate this cheaply in a Rails project. Something as simple as extracting every fenced shell command from your README and asserting each one at least exists: test "every documented rake task exists" do
documented = File.read("README.md").scan(/^(?:bundle exec )?rake ([\w:]+)/).flatten
available = `rake -T`.scan(/^rake ([\w:]+)/).flatten
(documented - available).tap do |missing|
assert_empty missing, "README names tasks that do not exist: #{missing.join(", ")}"
end
endCrude, and it would have caught this. The general principle: anything in your docs that a reader will execute should be executed by a test. Ancestry as a coupling mechanismOne more thing in this PR worth knowing about, because it is a neat trick. The Skill pins the CLI commit it was verified against. A digest of a commit hash alone would not be enough, because a commit can exist on a branch, get force-pushed away, and never reach So CI checks that the pinned commit is an ancestor of the CLI's main branch. Not just that it exists; that it actually landed. And the PR says plainly that this gate is expected to fail until the CLI stack merges. That honesty matters: an expected-red gate that nobody documented becomes a red gate everybody ignores, and then it stops being a gate at all. Worth remembering the distinction generally. "This commit exists" is weak. "This commit is reachable from main" is a real guarantee, and it is one line of git. The takeawayAsk who reads your documentation. If the answer includes a program, the documentation is an interface, and interfaces have compatibility rules:
|
|
Follow-on #19 now records the fresh-model evidence that this PR intentionally did not establish. The no-fresh-agent-proof boundary in this PR remains historically correct for its own diff; #19 adds separately pinned Home Inventory opening-interview and Movie Catalog diagnostic-to-Compile evidence after this contract. This PR remains the prerequisite and is not obsolete. |
|
Follow-up from the later stack: #19 now rewrites the packaged The 41→40 count is also now explained: two prior Compilation/Publication guidance cases were consolidated into one product-Compile and retained-Compilation contract. The ancestry check already distinguishes a valid nonancestor from an invalid revision, so no code change is needed there. |
Preserve the reviewed external CLI contract while inheriting the landed interview protocol and evaluation foundation.
1 participant
Summary
This PR renews the deterministic checker and eval contracts, not the packaged
SKILL.md. Skills #19 performs the shipped workflow rewrite and must land before any release.Integration
Skills #17 and the complete CLI stack have landed. The exact CLI contract revision
f55edffc…remains an ancestor of CLImain, so the original reviewed contract and digest pin remain unchanged and both hosted ancestry gates now pass.Verification
PATH="/Users/sandbox2/.asdf/shims:$PATH" node script/check-cli-contract.mjs /Users/sandbox2/code/firstdraft/cli-skills-contract-f55-20260804PATH="/Users/sandbox2/.asdf/shims:$PATH" sh script/check(40 tests)git diff --checkThe 40-test count consolidates two prior Compilation/Publication guidance cases into one product-Compile and retained-Compilation contract. The joined harness remains controlled local evidence: no live GitHub or staging mutation, generated-app execution, deployment, or fresh-agent proof.