Record bounded web and iPhone evidence

[raghubetina]

Summary

• replace the stale pre-journey wording with the current bounded rails-sketch/2026-08 evidence
• distinguish the reproducible installed-CLI smoke, fresh-agent authoring and generated iOS checks, and later operator Rails and Simulator observation
• pin the immutable First Draft field report at 16b056a6f55eb92cb6e5a6e02abd58e84b47abd5
• retain explicit limits around authentication, representative users, physical devices, iPad, deployment, production, cancellation, and unsupported graph breadth
• strengthen repository checks for revision provenance, evidence polarity, agent-versus-operator attribution, and the synthetic 190-file eval versus observed 194-file output

Evidence boundary

The linked field report records one staff-prepared local observation, not a reproducible agent evaluation or production proof. The fresh session authored Movie and Director, reached graph-version-1 valid analysis, compiled once, materialized 194 files, and completed the generated iOS checks. After that session ended, an operator performed Rails setup and the composed Simulator observation using a temporary test-only copy.

The Skill's existing approval boundaries remain unchanged. It still requires separate authorization before sending a Plan, compiling, or executing generated Rails or iPhone code.

Review

Claude Review ran on Opus at high effort, findings were addressed, and two focused re-reviews approved the final evidence attribution and immutable repin with no remaining actionable issues.

Verification

• npm ci --ignore-scripts — 0 vulnerabilities
• sh script/check — 16 tests passed
• node script/check-cli-contract.mjs /Users/sandbox2/code/firstdraft/cli
• uv run --with pyyaml python /Users/sandbox2/.codex/skills/.system/skill-creator/scripts/quick_validate.py skills/create-full-stack-app
• gh skill publish --dry-run — completed; existing advisory warnings remain for optional license metadata and tag-protection rules
• git diff --check
• tracked-source search confirms no provisional c831d505 reference remains

[raghubetina]

Technical review

Cloned the branch and ran the gate rather than reading it:

sh script/check
ℹ tests 16
ℹ pass 16
ℹ fail 0

The repin is complete, and the new pins are durable

This PR exists partly because c831d505 was a pre-rebase branch head that would not survive the merge. That is fixed, and the fix is enforced structurally rather than by discipline:

$ git grep -n "c831d505" -- .
(no matches)

assertRevisionTokens is what makes that hold. It asserts the complete sorted set of 40-hex tokens in each file, so a leftover pin fails the suite rather than lingering unnoticed. assertRevisionTokens(SKILL.md, []) going to an empty list is a nice consequence: SKILL.md now carries no revisions at all and defers to the reference document, so there is one fewer place for a stale SHA to hide.

I checked that the replacements are actually reachable, since a pin that vanishes is the failure being corrected:

Pin	Relationship to firstdraft main
16b056a6 field report	identical to the current tip
5847a349 controlled smoke	behind, so an ancestor

Both are permanent. Worth contrasting with 0bbb6c5 in the field report itself, which compares as diverged, since it is a clean checkout that was never on main. That one is handled correctly on the other side by anchoring it to 5847a349 through tree identity, and both do resolve to tree 6e8cdf67….

The numbers behind the wording check out independently

The prose assertions in repository.test.mjs pin that the README still says "194-file, 542,894-byte artifact". They cannot tell you whether 194 and 542,894 are correct, since the expected values live in the same repository as the claim.

Those two I can confirm from outside. I ran script/compilation_http_cli_smoke myself while reviewing firstdraft#263 and got 194 files and 542,894 bytes. The CLI runtime digest 205e664d… is one I recomputed from the harness algorithm on that same PR, and the iOS Core archive 0807e76c… is one I regenerated byte-identically from upstream on #259. Independent paths, same values.

The 190 versus 194 clarification is worth having

The Compilation eval's 190-file response remains deterministic synthetic transport data; it is not the 194-file output observed by the controlled smoke and dated field report.

I traced the 190 to see whether the characterization is fair. It sits in evals/create-full-stack-app/cases.json:894, inside an eval prompt, next to artifact SHA-256 aaaaaaaa…, manifest SHA-256 bbbbbbbb…, and byte size 12345. Unmistakably synthetic once you see it in context, and easy to misread as a measurement when quoted alone. Naming the difference explicitly is the right call, particularly since the two numbers are close enough to look like a discrepancy rather than two unrelated things.

The two kinds of check in this file are worth distinguishing

The revision-token assertions are structural. They compare a computed set against an expected set and catch real drift, including the exact class of bug this PR fixes.

The prose assertions are different in kind:

assert.match(readme, /staff-prepared local observation[\s\S]*?fresh Claude Code Opus\/high[\s\S]*?194-file, 542,894-byte artifact/);

That is a check on wording, and its value is specific: a caveat cannot be quietly deleted, and the shape of an evidence claim cannot drift without someone editing the test and noticing what they are changing. For a repository whose main product is calibrated claims, that is a reasonable thing to guard.

It is worth being clear-eyed that it proves nothing about accuracy. Change a number in both places and it passes. That is not an argument against the checks, only against reading a green suite as confirmation that the claims are true. The pins and the external reproductions are what carry that.

The description change is now warranted

SKILL.md drops "that path has not completed an end-to-end journey" from its front matter. Given the field report, that clause had become false, and removing it is a narrowing of a negative rather than a widening of a positive. The surrounding limits on authentication, representative users, physical devices, iPad, deployment, and cancellation all survive.

No findings

[raghubetina]

Lesson: the link you just pasted is probably going to break

This PR fixes a bug that has nothing to do with code. A document pointed at commit c831d505, that commit stopped existing, and the reference became a dead end.

Here is how that happens, and it is worth understanding because it will happen to you.

Why the commit disappeared

You open a pull request. Its branch head is c831d50. You paste that SHA into a document as your citation, which feels like the careful thing to do, since a SHA is immutable.

Then the PR gets rebased, perhaps to pick up a change on main, or because the merge is configured as rebase-and-merge. Rebasing does not move commits. It creates new ones with the same changes, different parents, and therefore different hashes. The originals are now unreferenced. Once the branch is deleted and garbage collection runs, c831d50 is gone from the remote, and your link 404s.

A commit on a pull request branch is not a stable address. It looks permanent because SHAs are immutable, but immutable is not the same as reachable, and a link only works if the commit is still reachable from something.

The hierarchy of link stability

Worth internalising, roughly worst to best:

blob/main/path/to/file.rb#L42 is the worst option and the most commonly used. The branch moves, so both the content and the line numbers drift. Six months later it points at whatever line 42 happens to be, which is often something unrelated and occasionally something that reads as though it supports a claim it does not.

A tag is better but not guaranteed. Tags are movable in git, and while retagging is rude, it happens.

A commit SHA on a merged branch is solid. Once the commit is reachable from main, it is kept alive by that reachability, and GitHub will serve it indefinitely.

A commit SHA on an unmerged branch is the trap described above.

This is why GitHub's "copy permalink" (the y keyboard shortcut on any file view) rewrites the branch name to a commit SHA. It is worth building the habit, because the resulting link keeps pointing at the same bytes forever.

Checking that a pin is actually permanent

The useful question is not "does this SHA exist" but "is it reachable from the default branch." One API call answers it:

GET /repos/OWNER/REPO/compare/main...SHA

The status field comes back as identical if it is the current tip, behind if it is an ancestor of main, and diverged if it is not on main at all. The first two are safe to cite. The third is the one that will rot.

When you have to cite something that was never on main

Sometimes the thing you ran is genuinely not a merged commit: a clean checkout, a local build, a scratch branch. You cannot pretend otherwise, and picking a nearby merged commit instead would be a small lie about what you actually ran.

The move used here is worth stealing. Cite the real commit, then anchor it to a durable one by tree hash:

$ git rev-parse 0bbb6c5^{tree}
6e8cdf67751bc08970fb3ab59cc99ea74a9b4349
$ git rev-parse 5847a34^{tree}
6e8cdf67751bc08970fb3ab59cc99ea74a9b4349

A tree hash identifies file content, independent of commit metadata like parents, author, and timestamp. Two commits sharing a tree have byte-identical working directories. So "I ran 0bbb6c5, whose tree is identical to merged commit 5847a349" is both honest about what executed and verifiable against something permanent, even after the scratch commit is collected.

That is a good general property to want from a citation: honest about what actually happened, and anchored to something that will still resolve.

The habit

When you paste a repository link into a document, an ADR, a comment, or a commit message, ask which of these it is. If it is a branch name, replace it with a SHA. If it is a SHA, check that the SHA is reachable from main, or say what it is tied to.

The cost is about ten seconds. The alternative is a document that looks well-sourced and cites nothing, which is worse than one that cites nothing openly.