Skip to content

fix(spec): scan planned and experimental evidence, not live alone - #13081

Merged
os-trump merged 1 commit into
mainfrom
claude/issue-13041-planned-evidence-scan
Aug 29, 2026
Merged

fix(spec): scan planned and experimental evidence, not live alone#13081
os-trump merged 1 commit into
mainfrom
claude/issue-13041-planned-evidence-scan

Conversation

@os-trump

Copy link
Copy Markdown
Collaborator

Fixes #13041

Disposition option 1 as ruled at dispatch (claim comment 5460051202): extend the evidence scan to planned and experimental. Option 2 (the exclusion written down as a commented, argued exemption) is done alongside it, because the triage said it is worth doing whichever way option 1 went.

The population change

packages/spec/scripts/liveness/check-liveness.mts, the one condition that gated all four evidence checks (existence, line bound, symbol anchor, key mention):

// before
if (status === 'live' && led?.evidence) {
// after
if (EVIDENCE_SCANNED_STATUSES.has(status) && led?.evidence) {

with the partition declared, argued and held to the published vocabulary:

const EVIDENCE_SCANNED_STATUSES = new Set(['live', 'planned', 'experimental']);
const EVIDENCE_UNSCANNED_STATUSES = new Set(['dead']);
for (const s of STATUS_COLUMNS) { /* a status in NEITHER set throws */ }

(Both Set constructions carry an explicit string type parameter in the source; it is elided here because an angle-bracket fragment does not survive this body's sanitizer, in a fence or out of one.)

producer was already scanned at any status; that asymmetry is what made the gap visible. The status vocabulary is exactly four — read from STATUS_COLUMNS in readme-table.mts and confirmed against the ledgers, which use no fifth value — so three are in and one is out, with no third category to decide about.

The exhaustiveness loop is a deliberate, named addition rather than a rider: a fifth status added later would otherwise fall out of both sets and land silently unscanned, which is this card refiled. It compares two code constants, so it can only fire when someone edits STATUS_COLUMNS.

The output label is derived from the set (EVIDENCE_SCANNED_LABEL), so the numbers and the population they describe cannot drift apart — the class of bug this whole file is about.

Why dead stays out, measured

Across every ledger at this commit: 80 dead entries, all 80 carry a note, only 6 carry an evidence string. A dead row's pointer is the retirement story in note — which sweep, which ADR, which tombstone rejects the key now — and no check scans note, by design. Scanning dead.evidence would hold 6 rows to a standard, read nothing of the other 74, and publish that as coverage of the class: the same shape as the defect being fixed, one status over. Measured for completeness: those 6 carry 3 repo-local paths and all 3 resolve today, so the exclusion is not hiding a red.

Survey first — the explosion radius, before the switch was flipped

Counted before the code changed, per the triage comment's dispatch prerequisite.

Population: live 819 · planned 10 · dead 80 · experimental 5 (914 classified). Of the 15 newly-in-scope entries, 7 carry an evidence string at all:

entry status evidence disposition
api/inputMapping.transform planned packages/runtime/src/api-mapping.ts#mappingDeclarationRejection clean — file exists, symbol at line 250, file names transform (8 sites)
api/outputMapping.transform planned same anchor, same file clean — same three readings
agent/lifecycle experimental no runtime reader (StateMachine) clean — prose, extracts no path
agent/memory experimental no runtime reader clean — prose
agent/guardrails experimental no runtime reader clean — prose
agent/structuredOutput experimental no runtime reader clean — prose
tool/outputSchema experimental packages/services/service-ai/src/tools/action-tools.ts:437 clean by attributionpackages/services/service-ai/ is a declared FOREIGN_PATH_PREFIXES entry, so it counts as foreign and is never resolved or line-bounded. The file is genuinely absent here; that is the boundary working, not a rot

The other 8 (action/onSuccess.navigate, action/onSuccess.openIn, field/useGrouping, field/relatedListFilter, object/externalSharingModel, page/requires, translation/flows.label, translation/flows.screens) carry no evidence key, so they add nothing to any counter.

Newly flagged: zero. No mechanical repair was needed and none was made — no ledger byte changed in this PR. The prediction from the survey was written before the switch was flipped and the run below matched it exactly, counter for counter.

Counter movement (the gate's own output, before → after)

evidence paths:        465 declared / 465 resolved / 153 foreign
                   →   467 declared / 467 resolved / 154 foreign
line citations:          0 written / 0 in range          (unchanged)
                   →     0 written / 0 in range
symbol anchors:        499 written / 499 resolving
                   →   501 written / 501 resolving
key-mention anchoring: 465 pairs asked / 464 anchored / 1 exempt
                   →   467 pairs asked / 466 anchored / 1 exempt

+2 local paths and +2 anchors are the two planned api.json refusal pointers — the ones the card measured as "counted by the census, verified by nothing". +1 foreign is tool/outputSchema. Line citations do not move: the only newly-scanned path:NNN citation is the foreign one, which is never bounded. key-mention.baseline.json needed no new row — the two new pairs anchor.

byStatus is untouched by this change, so liveness/state-counts.md and the README table do not move.

Ablation — the new red direction is real

Implementation committed first; the widened condition then reverted to the pre-fix status === 'live' form and the gate's own test suite read on the mutated tree. Absolute paths, trap restore.

On-disk confirmation, both directions (anchored at the text being replaced, not a bare --stat):

BEFORE   new-form: 1   old-form: 0     blob 28bacc5bb988a2b049e983d5a3210c12ac3dfce0 (== HEAD blob)
AFTER    new-form: 0   old-form: 1     blob 3b9d59a74bfcf364c8fd547930827939cb78d85c
RESTORED new-form: 1   old-form: 0     blob 28bacc5bb988a2b049e983d5a3210c12ac3dfce0 (== HEAD blob)
git diff HEAD  →  empty

The mutated blob under which the run below was taken is the same 3b9d59a7 hash, so the reading and the byte proof are about one tree.

No build/dist leg, stated rather than skipped: the gate runs from source via tsx and the test spawns that same source file, so there is no dist/ artefact a stale build could mask.

Prediction, written before the run: exactly 3 RED — the three new population cases — and the dead-boundary case and the partition pin stay GREEN, because the mutation touches neither the dead exclusion nor the two Sets.

Observed (pnpm exec vitest run scripts/liveness/check-liveness.test.ts on the mutated tree, Tests 3 failed | 29 passed (32)):

× FAILS when a `planned` entry's anchored REFUSER is gone from the file
× FAILS when a `planned` entry cites a repo-local file that is gone
× FAILS when an `experimental` entry cites a repo-local file that is gone
AssertionError: … expected +0 to be 1   (the gate exits 0 where the fixed gate exits 1)

stays GREEN when a dead entry carries the SAME rotted pointer and declares every status either scanned or explicitly unscanned both passed on the mutated tree, as predicted — so the three reds are attributable to the population change and to nothing else about the fixtures.

Honest note on the transcript: the first ablation pass also printed two grep -c lines that read 0 in both directions — a shell-quoting error in the harness, not a reading. They were vacuous and are not quoted as evidence; the byte proof above is a clean second pass at the same blob hashes, and the first pass's real proofs (an anchor count that refused any hit-count but 1, and MUT_BLOB != HEAD_BLOB) stand on their own.

Tests

  • check-liveness.test.ts — new describe('check:liveness — the evidence-scan population (#13041)'): the card's own instance (a planned entry whose anchored refuser is renamed away must red), a planned missing-file case, an experimental missing-file case, the dead boundary green on the same rotted string, and a source pin on both Sets plus the published label. Four existing assertions moved off the hardcoded 'live' wording onto one SCANNED_LABEL constant.
  • evidence.test.ts — the shipped-ledger contract test's population genuinely moves: its status !== 'live' filter mirrored the gate's, and leaving it narrow would have re-created the counted-but-unverified split one layer down. The anchor contract test beside it already visited every status and is unchanged.
  • key-mention.mts — prescription wording, which said "a live entry" and now covers the refusal statuses.

Verification

All exit codes captured before any pipe; verdicts quoted from each gate's own output. Union re-run after the final commit, at 348b3d1f1.

  • pnpm --filter @objectstack/spec check:liveness — exit 0, ✓ every governed-type property … every 'live' / 'planned' / 'experimental' entry's repo-local evidence path resolves …
  • pnpm --filter @objectstack/spec exec vitest run --maxWorkers=2 scripts/liveness/ — exit 0, Test Files 10 passed (10) · Tests 258 passed (258)
  • tsc --noEmit -p tsconfig.scripts.json --listFiles — exit 0, 0 lines matching error TS; coverage proven rather than assumed: all four edited files appear in the program's --listFiles output (1 hit each), the two .test.ts files included
  • pnpm --filter @objectstack/spec check:test-typecheck — exit 0, 55 file(s) / 263 error(s) held in test-typecheck-debt.json — unchanged, so this diff drifts the debt ledger up by nothing
  • pnpm check:nul-bytes — exit 0, scanned 7275 text file(s) … no raw ASCII control bytes; plus a direct control-byte scan of the four edited files (no hits)

Gate family derived with node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack (26 path-derived families, plus the convention triggers for a test-file edit and a gate-script edit). Run locally:

gate verdict
node scripts/pm/bare-root-worklist.mjs --self-test exit 0 — OK self-test: 47 live row(s) … none stale, none missing, none contradicted
pnpm check:pm-dispatch-gates exit 0 — ✓ dispatch-gates self-test: 872 cases pass.
pnpm check:engine-double-contract exit 0 — OK — 704 pinned, 134 in the DEBT ledger, 3 exempt
pnpm check:where-matcher exit 0 — 313 matcher(s) discovered … 0 silently-wrong … none new
pnpm check:query-options-erasure exit 0 — ratchet holds: 67 unswept non-test site(s) … none new
pnpm check:cross-package-test-inputs exit 0 — OK: 23 package(s) read outside themselves, all declared
pnpm check:type-check-coverage exit 0
pnpm --filter @objectstack/spec check:empty-state exit 0 — ✓ all classified (1 closed, 2 open, 4 output, 9 scope)
pnpm check:published-files exit 0 — 69 publishable package(s) … admits no test, test-harness config or build script

Declared narrowing: the repo-wide scans in the derived list — pnpm lint, the docs-drift pair, the CI shard/attestation gates and check:type-check-debt --re-measure (which refuses an unbuilt worktree and needs the whole workspace closure built) — were not run locally. CI runs the farm exactly once regardless. The one with a real chance of moving is the test-typecheck ratchet, and its per-package half for the only package this diff touches is the check:test-typecheck run above, green with the ledger count unchanged.

Changeset — measured, not assumed

packages/spec's files array is ["dist", "json-schema", "liveness", "prompts", "llms.txt", "README.md", "src/**/*.zod.ts", "CHANGELOG.md", "api-surface", "spec-changes.json"]. scripts is not in it, and check:published-files independently reports that the whitelists admit "no test, test-harness config or build script". This diff is entirely inside packages/spec/scripts/liveness/ — four files, none of which reach a consumer's node_modules — and no ledger byte under the shipping liveness/ directory changed. Nothing is released by this PR, so it takes the skip-changeset label route rather than a patch changeset.

Honest residual


Generated by Claude Code

The liveness gate's four evidence checks (existence, line bound, symbol
anchor, key mention) ran under `status === 'live'` while `producer` was
scanned at any status. An entry whose whole content is a REFUSAL therefore
carried evidence the census COUNTED and no check READ.

Measured on api.json: `inputMapping.transform` and `outputMapping.transform`
are `planned` and were migrated to `path#symbol` anchors precisely because
the refusal disappearing is what should go red -- and renaming
`mappingDeclarationRejection` moved no verdict.

`dead` stays out, now as a declared exclusion rather than an omission: all 80
dead rows carry a `note` and only 6 carry `evidence`, so a dead row's pointer
lives in prose no check scans by design.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LpRNHxWZgSUgVnFT9mQQo4
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

Nothing in this diff resolved to a documentable surface (no symbol, route or SDK anchor derived from 0 changed package(s)), so this run has no opinion about the docs.

What this run could not see
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 0 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json f887e5249373ccc2d9af36b5c9b02cc99a691c40packageMentionDocs.

@os-trump os-trump added the skip-changeset PR has no user-facing published change; bypasses the changeset gate label Aug 29, 2026 — with Claude
@os-trump
os-trump marked this pull request as ready for review August 29, 2026 04:11
@os-trump
os-trump enabled auto-merge August 29, 2026 04:11
@os-trump
os-trump added this pull request to the merge queue Aug 29, 2026
Merged via the queue into main with commit 7c0cd41 Aug 29, 2026
38 checks passed
@os-trump
os-trump deleted the claude/issue-13041-planned-evidence-scan branch August 29, 2026 04:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/m skip-changeset PR has no user-facing published change; bypasses the changeset gate tests tooling

Projects

None yet

2 participants