Skip to content

feat(scripts): assert no bare any in a marked skill example, and scan .claude/skills too - #7490

Merged
os-litant merged 2 commits into
mainfrom
claude/issue-7463-skill-examples-hardening
Sep 3, 2026
Merged

feat(scripts): assert no bare any in a marked skill example, and scan .claude/skills too#7490
os-litant merged 2 commits into
mainfrom
claude/issue-7463-skill-examples-hardening

Conversation

@os-litant

@os-litant os-litant commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Part of #7463

Two of the card's three hardening items on check-skill-examples.mjs are implemented. Item 2 is measured and escalated, not taken: the bound's two candidate homes lead to materially different edits, and the brief for this flight says stop there rather than guess.

Every population below was measured before the assertion was armed, and re-measured at head 2df07b7.


Item 1 — the bare-any assertion (implemented)

Ported from objectstack's packages/spec/scripts/check-skill-examples.ts with its scope intact: the annotation must be any in a position that erases checking — a parameter, a variable / property / return annotation, a type alias, or an as any / satisfies any / angle-bracket assertion. An any nested inside a larger type is deliberately allowed: Record of string to any, any[], Promise of any. That boundary is the zero-false-positive line, and holding it is what keeps a red meaning broken.

One deliberate divergence from objectstack, and why

objectstack picks ScriptKind off the fence label. The harness this gate compiles through — compileSnippets in check-doc-snippet-types.mjs — parses every block as TSX regardless of label, for reasons its own header records. So this walk uses TSX too. A guard that walked a different tree from the one tsc judged would be exactly the dormant checker this file's docblocks warn about.

Visible consequence: an angle-bracket any assertion is JSX under TSX, so it is a parse failure — already red through the syntax leg, one exit code earlier, never reaching this walk. Its arm is kept in the position table so the rule stays whole if the harness's ScriptKind ever changes, and the header says so rather than leaving a reader to discover a dead branch.

Measured population, before arming

--measure over every candidate fence, marked or not:

population findings
every candidate fence (121 ts) 10
MARKED fences only — what this assertion gates 4, in 3 fences
new red on day one 0
.claude/skills fences (item 3's new candidates) 0

The 4 would-be-red rows, each declared verbatim

None is unmarked or re-pointed. All four are carried in KNOWN_BARE_ANY_EXAMPLES, a shrink-only ratchet in the shape KNOWN_UNTAUGHT_EVAL_TOKENS uses in check-skill-eval-tokens.mjs: a row whose red goes away fails as stale, so the list can only shrink.

row position why it is declared debt, not a mechanical unmark
skills/objectui/guides/plugin-development.md:92 property defaultValue The guide is faithful prose, not rot. ComponentInput.defaultValue really is any at packages/types/src/base.ts:550. The honest fix is to the platform type; changing the guide alone would make it lie about the contract it restates.
skills/objectui/guides/testing.md:60 as any assertion validateSchema({} as any) — feeding the validator something invalid on purpose. Which idiom the testing guide should teach for that (as unknown as T, a ts-expect-error) is a question about the guide.
skills/objectui/guides/testing.md:208 variable mockClient The test-double idiom. Same question, same owner.
skills/objectui/guides/testing.md:208 as any assertion Reaching a private field through an adapter cast, in the same block.

Each needs a per-row skills judgement that a gate-hardening change is not the place to make. Declaring them keeps that judgement a visible piece of work instead of a silent exemption — and the stale direction means a row cannot be parked forever.


Item 2 — bounding root-devDependency resolution (MEASURED, ESCALATED, no code)

What was measured

The marked population's unmapped specifiers are exactly @playwright/test and vitest. Both are declared by the repository root and both resolve out of /node_modules, where pnpm symlinks the root's own dependency set. Neither is declared by any package a marked fence imports, so neither is covered by the workspace map or the declared-dependency map. The inherited UNDECLARED control does not close this: it bounds resolution against transitive packages, which pnpm leaves only under .pnpm/ — a different leak.

Under --measure the unmapped set is 12: six root-declared and installed (@playwright/test, @testing-library/react, msw, react, vite, vitest), and six not installed at the root at all (four @objectstack/*, plus @tailwindcss/vite and @vitejs/plugin-react) — those already fail to resolve, which is the existing bound working.

Why this stops here

The harness is shared by exactly two production consumers: check-doc-snippet-types.mjs itself and this gate. Measured blast radius of putting the bound in compileSnippets:

A harness-level bound newly reds exactly ONE of check-doc-snippet-types.mjs's own 432 compiled snippetscontent/docs/guide/objectos-integration.mdx:638, which imports @playwright/test.

That is a verdict move on a gate this flight does not own, over a documentation surface with its own owner. The card's own words are that the harness "carries the same edge" and asks where the bound should live; the two readings produce materially different edits, so per the flight brief this item returns for a decision rather than being guessed.

The options, on the four axes

Option A — bound inside compileSnippets, unconditionally (both gates).

  • Real business need: measured pull is real but small — one leaking snippet in the docs corpus, two specifiers in the skills corpus. The need is genuine (a green that rests on this workspace's devDependencies is not a claim about what a reader installs) but the surface is narrow.
  • Long-term soundness: strongest. One resolution regime, one answer to "how far does resolution reach", stated once in the harness that already owns the syntax/semantics split and the four self-controls. It is the contract-first shape: the harness's guarantee stops being "narrow, except for whatever the root happens to declare".
  • Preventing AI-authored mistakes: strongest. Declared equals enforced — a guide cannot silently rest on a package the reader was never told to install, which is precisely the class an AI copying a fence would propagate.
  • Startup scope discipline: cheapest in surface — no new flag, no second regime, no staged window. Costs one docs snippet, which must be fixed or declared in the same change.

Option B — bound per gate (a parameter compileSnippets accepts, default off).

  • Real business need: same pull, and it lands the skills half today without touching the docs corpus.
  • Long-term soundness: weakest, and the cost should be stated plainly. An opt-in leniency flag in a shared harness is two resolution regimes behind one function, with the docs gate permanently on the lenient one. That is the consumer-tolerance shape the decision frame names — the harness would answer "did resolution stay narrow" with "depends who is asking".
  • Preventing AI-authored mistakes: weaker. A gate whose strictness is a caller-side argument is a gate whose guarantee has to be re-read at every call site.
  • Startup scope discipline: it looks cheaper and is not — it adds a permanent configuration surface to avoid one docs edit, which is sunk-cost reasoning about a snippet nobody has argued for.

Option C — do nothing; keep naming the specifiers on every run.

  • Real business need: the Unmapped specifiers line already stops the leak from being silent, which is what the landing flight chose deliberately.
  • Long-term soundness: weakest — a known-unenforced property that a reader must remember to read.
  • Preventing AI-authored mistakes: weakest; a printed line is not a gate.
  • Startup scope discipline: strongest in the short run, and it is the status quo.

Recommendation: Option A, led by long-term soundness (weighted at least 50%): it is the only option that leaves one answer to one question, and it is the option the frame's contract-first and declared-equals-enforced axes both select. The one docs snippet it reds is a real finding, not collateral — that snippet's green rests on this workspace's devDependencies rather than on anything its reader installs. Option A does need a call this flight cannot make on its own: whether that snippet is fixed, or declared, and by whom. Option B is not recommended at any weighting — it buys one avoided docs edit with a permanent second resolution regime in shared code.


Item 3 — SCAN_ROOTS widened to .claude/skills (implemented)

.claude/skills does exist in objectui (4 markdown files), so this is code, not a note. Widened the way check-skills-paths.mjs was in #7358, and for the same measured reason: when #7251 moved the two contributor-only guides out of skills/, that gate stopped looking at them and nothing turned red.

Widening a root is not arming it. Opt-in is the design, so this adds candidates, not coverage:

before after
guides scanned 18 20
ts / tsx / typescript fences 112 121
json / jsonc fences 56 56
MARKED fences 56 56

All 9 new candidates are typescript fences in .claude/skills/objectui-contributor/ — 5 in guides/console-development.md, 4 in rules/no-touch-zones.md. The two SKILL.md files carry none. No marker was added to any file under .claude/skills in this PR; that is the surface owner's step. None of the 9 carries a bare any either, so item 1 gains nothing to declare from item 3.


Reverse verification

Each leg: commit first, mutate under a trap ... EXIT INT TERM restoring an absolute path, prove the mutation on disk by grep count before reading any verdict, then prove the restore by git hash-object equal to the HEAD blob plus an empty git diff HEAD.

Leg A — an undeclared bare any in a marked fence must red

Mutated skills/objectui/guides/i18n.md (marked fence at line 39), injecting a bare any variable.

  • mutation proof: injected-text count 0 -> 1; worktree blob 7c2f01fe… differs from HEAD blob fc0d4ad3…
  • verdict: gate exit 1, [bare-any] skills/objectui/guides/i18n.md:44:35 variable ... is annotated any, summary line 5 finding(s) ... 1 NOT declared
  • restore proof: blob back to fc0d4ad3… (equal to HEAD), injected-text count 0, git diff HEAD empty

Leg B — the ratchet's shrink-only direction

Mutated skills/objectui/guides/testing.md so a declared row's red goes away ({} as any becomes {} as unknown as never).

  • mutation proof: removed-text count 1 -> 0, injected-text count 0 -> 1; blob a88336f9… differs from HEAD 601d98fd…
  • verdict: gate exit 1, skills/objectui/guides/testing.md:60 as any assertion [stale-baseline] this row is no longer red — delete its line
  • restore proof: blob back to 601d98fd…, removed-text count back to 1, git diff HEAD empty

Leg C — the widened root is real coverage, not a cosmetic list

Appended a marked fence carrying a bare any to .claude/skills/verify/SKILL.md.

  • mutation proof: injected-text count 0 -> 1; blob 38816807… differs from HEAD 68b48b7e…
  • verdict: gate exit 1, [bare-any] .claude/skills/verify/SKILL.md:50:42 parameter v, and Marked: 18 ts fence(s) (up from 17) — the widened root both scans and gates the file
  • control, the load-bearing half: the same probe run against a copy of the script with SCAN_ROOTS narrowed back to skills alone sees 0 .claude/skills guides and 0 findings. Without this control the leg would only show the gate working, not that the widening is what made it reach.
  • restore proof: blob back to 68b48b7e…, injected-text count 0, git diff HEAD empty

Leg D — classifying three unrelated red tests

Three failures in scripts/__tests__ are pre-existing and independent of this change. Proven rather than asserted: reverted all three of my changed files to the merge-base e307c98 (proof on disk: the narrow SCAN_ROOTS line present, findBareAny absent), re-ran both suites, and the same 3 tests failed identically. Neither suite references check-skill-examples. Restore proof: all three blobs equal their HEAD blobs, git diff HEAD and git status --porcelain both empty.

They are built-tree-sensitive: check-sdui-registration-pins expects packages/app-shell/src/...tsx and gets dist/...js, and the two check-readme-exports legs are explicitly about "BOTH build states". This gate requires a built tree, so running its suite after the build surfaces them. Reported as out-of-scope findings, not fixed here.


Gate table

Every exit code captured by redirect before any pipe. Verdicts quoted from each gate's own printed line.

gate exit its own verdict line
node scripts/check-skill-examples.mjs (after the filtered build it derives) 0 Every marked skill example holds up against the built types.Bare any: 4 finding(s) across 3 selected fence(s); 4 declared ... 0 NOT declared, 0 declared row(s) no longer red.
--self-test 0 ✓ check-skill-examples self-test: 42 cases pass (... the bare-any guard in both directions with its shrink-only baseline ...) (21 new legs)
--measure 0 Starting population — ts: 18/121 pass; json: 39/56 pass.Bare any would-be population — 10 finding(s) over every candidate, of which 4 sit in a MARKED fence.
--build-filter 0 output byte-identical to the pre-change run (diffed) — the workflow's build step is unmoved
pnpm exec vitest run --maxWorkers=2 scripts/__tests__ 1 Test Files 2 failed, 98 passed (100) / Tests 3 failed, 2883 passed (2886) — the only failures are the 3 pre-existing ones classified in Leg D; check-skill-examples.test.ts alone: 65 passed, exit 0
pnpm exec tsc -p tsconfig.scripts.json --noEmit 0 clean
eslint scripts/check-skill-examples.mjs --format json 0 1 file linted, 0 errors, 0 warnings
pnpm lint (repo-wide) 0 Tasks: 47 successful, 47 total — 4220 files linted repo-wide, 0 errors (11832 pre-existing warnings, unchanged in kind)
node scripts/check-pre-install-import-graph.mjs 0 ✅ check-pre-install-import-graph: OK — 23 pre-install step(s) in 18 job(s) run 21 scripts/ gate(s); 23 module(s) walked, every non-relative leaf a node builtin.
node scripts/check-changeset-presence.mjs 0 ✅ No source or published contract of a released package changed in this range, so no changeset is owed.
node scripts/check-governed-queue-guard.mjs --test over the final path list 0 ✅ NOT GOVERNED — 2 path(s) checked against 5 governed surface(s); none matched.

The governed-queue verdict is NOT GOVERNED and that is the expected answer: this change set touches only scripts/, its tests, and content/docs/ — no file under skills/ or .claude/skills/ is modified. The PR stays draft either way.

Control-byte scan over the changed files (grep -naP for C0/C1 and DEL) returned no matches.


Per-file before and after

file before after delta
scripts/check-skill-examples.mjs 1055 1397 +342
scripts/__tests__/check-skill-examples.test.ts 476 557 +81
content/docs/guide/ci-cd-pipeline.md 2079 2101 +22

No file under the published skills bundle or under .claude/skills is touched, so the published-skill line budget does not apply.

Invariants held

  • exit 2 PRECONDITION NOT MET semantics unchanged — unbuilt tree, failed harness control and empty marked population all still leave through couldNotRun. A bare any and a stale row are verdicts about a guide, so they leave through exit 1, alongside the orphan marker.
  • The marked-population shrink-only ratchet (No gate reads inside a code fence in skills/objectui/ — 123 TypeScript fences and 63 JSON fences are checked by nothing #7359 step 3) is not taken here.
  • No behaviour change for the existing 17 ts and 39 json marked fences: all still pass, and the only new findings are the 4 declared rows.
  • --build-filter output byte-identical, so REQUIRED_CONTEXTS and the workflow are unchanged.

NOT MEASURED, and why

  • The body of check-skill-examples.mjs is not type-checked. tsconfig.scripts.json sets allowJs: true with checkJs: false on purpose (its own comment records the measurement behind that choice). Confirmed with --listFiles: the file is in the program, so the .ts tests' use of the new exports (findBareAny, bareAnyRowKey, KNOWN_BARE_ANY_EXAMPLES) is checked — but errors inside the .mjs are not reported. A clean tsc here is a statement about the callers, not about the module body.
  • Whether the 4 declared rows should be fixed, and how. That is a per-row skills judgement, deliberately deferred to a card rather than made by a gate-hardening PR. The shrink-only direction is what stops the deferral from becoming permanent.
  • Item 2's bound is not implemented, so nothing here proves a bound works in either direction. The blast radius is measured; the bound is not written.
  • CI convergence. Reported at draft-PR time per the dispatch contract; the merge-queue and workflow results are the reviewer's read, not this body's.

Out-of-scope findings

Recorded for the seat to dedup and file — not fixed here, and no cards filed by me.

  1. scripts/__tests__/check-sdui-registration-pins.test.ts:146 — the pin expects the source module packages/app-shell/src/console/connect/ConnectAgentWidget.tsx and gets packages/app-shell/dist/console/connect/ConnectAgentWidget.js on a built tree. The derivation prefers dist once it exists, so this pin passes only on an unbuilt checkout. Proven pre-existing at e307c98 (Leg D).
  2. scripts/__tests__/check-readme-exports.test.ts:899 and :950 — both legs of "hides ONLY omissions, and the tree says so in BOTH build states" / "finds no fabricated or wrong-path import when built" fail on a built tree (excerptsNotJudged is 1, expected 3). Same class as (1), same proof.
  3. packages/types/src/base.ts:550defaultValue?: any in the exported ComponentInput interface. Not a guide defect: it is why row 1 of the baseline exists. Tightening it to unknown is the contract-first fix, and it is a platform-type change with its own consumers, so it wants its own card.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LraLgQVGq8egUwfYZpbYt1

…an `.claude/skills` too

Two of the three hardening items objectui#7463 records against
`check-skill-examples.mjs`. The third — bounding root-devDependency
resolution — is measured here and escalated rather than taken, because the
harness that would carry the bound is shared.

The bare-`any` assertion is ported from objectstack's
`packages/spec/scripts/check-skill-examples.ts` with its scope intact: the
annotation must BE `any` in a position that erases checking (parameter,
variable / property / return annotation, type alias, `as any` /
`satisfies any` / angle-bracket assertion). A nested `any`
(`Record<string, any>`, `any[]`, `Promise<any>`) is deliberately allowed —
that boundary is the zero-false-positive line. One deliberate divergence: the
walk parses every block as TSX because `compileSnippets` does, so the guard
reads the same tree `tsc` judged.

Measured under `--measure` before arming: 4 findings in 3 marked fences, 10
over the whole candidate population. All four are declared verbatim in
`KNOWN_BARE_ANY_EXAMPLES`, a shrink-only ratchet whose rows fail as STALE when
their red goes away. None is unmarked or re-pointed: one of them faithfully
restates `ComponentInput.defaultValue`, which really is `any` in
`packages/types/src/base.ts`, and the other three are testing idiom. Zero new
red on day one.

`SCAN_ROOTS` gains `.claude/skills`, the widening `check-skills-paths.mjs`
took in objectui#7358. Opt-in stays opt-in: 18 to 20 guides and 112 to 121 ts
fences, and zero new MARKED fences — no marker is added under
`.claude/skills/` here.

Exit-2 PRECONDITION NOT MET semantics are unchanged, `--build-filter` output
is byte-identical, and the marked-population shrink-only ratchet is still not
taken. `--self-test` grows 21 legs: ten in-scope positions found, six nested
`any`s not found, JSX not mis-parsed, and the baseline pinned in both
directions.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LraLgQVGq8egUwfYZpbYt1
…ed scan roots

The two `SCAN_ROOTS` pins asserted the pre-widening root list; they now assert
the widened one, and the non-vacuity leg additionally requires BOTH roots to be
non-empty in this checkout, so a root list nothing reads cannot pass.

New pins for the bare-`any` assertion. The negative half is the load-bearing
one: ten in-scope positions must be found and six nested `any`s must not be,
because that boundary is the reason a red from this assertion means something.
Plus the TSX ScriptKind (a JSX example must not be mis-read as a type
assertion), the no-throw contract on an unparseable block, the baseline row-key
shape, and a check that every declared row is shaped like a key the gate can
actually produce and sits under a scanned root.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LraLgQVGq8egUwfYZpbYt1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants