feat(conformance): add the executable four-path producer contract - #9
Conversation
The four stable read paths had a prose contract in docs/conformance.md and no executable one. This adds the standard-owned suite that measures a producer candidate against it, plus red tests proving each protected behavior is actually protected. check:conformance a candidate satisfies the contract check:conformance:test breaking each protected behavior makes that fail The suite runs against a candidate supplied by WORKSPACEJSON_CLI_CANDIDATE, because the producer lives in another repository. It does NOT skip when the candidate is absent — it exits non-zero with instructions. A conformance gate that goes green because it could not find the implementation reports conformance it never measured. Asserted per path: fileIndex populated from repository evidence with repository-root-relative POSIX keys that name real files, deterministically ordered; frameworkManifest publishing only dependency-corroborated frameworks at the documented confidence floor, and omitting uncorroborated tokens; manual.fragileFiles and manual.coChangePatterns preserved verbatim, and left absent rather than fabricated when a maintainer declared nothing. Beyond the paths: invalid artifacts refused rather than overwritten, --force moving them aside recoverably, producer identity not attributable to an invoker, output validating against the package-owned schema, a second run on an unchanged repository byte-identical, and mediated invocation producing the same artifact as direct after removing only generated.generatedAt. Deliberately NOT asserted: per-file values inside fileIndex. Every value field is optional in the published type, so an empty entry is conformant. Those values are behavioral and git-derived, and whether that source may enter the stable contract is an open determination made outside this repository. Requiring them would fail a producer that is behaving correctly. The scope boundary is stated in the suite so a later reader does not mistake it for an oversight. The red tests carry two honesty guards: each mutation asserts it changed bytes, so a stale anchor cannot leave a case silently testing nothing; and each names the check it expects to break, so reddening the suite for an unrelated reason does not count. A baseline case asserts a clean candidate is accepted, so the suite cannot pass by rejecting everything. Running this against the current producer found a real defect on the first attempt: the producer indexed its own artifact, so run 1 and run 2 disagreed and generate --check would fail on a repository's first CI run after adoption. Fixed in the producer repository; the case that catches it is retained here.
…blic The conformance job checked out the producer with a fine-grained PAT because that repository was private. It is public, so the checkout needs no token and no secret. This is strictly better than the token arrangement it replaces: - the gate runs on forks and on first-time contributors' pull requests, which a secret-dependent job cannot; - there is no credential to provision, rotate or leak; - nothing has to be configured before the gate can enforce anything, so the job does not sit red waiting on an out-of-band step. Verified with an unauthenticated clone against a disabled global git config, so the result does not depend on a credential helper that happens to be present locally. Also ignores .candidate/, the checkout path. A conformance run leaves no untracked noise, and a vendored copy of the producer cannot be committed by accident — `packages/cli` under a tracked path would trip the repository-boundary guard. The job continues to track the producer's default branch rather than a pinned SHA. Noticing when the implementation drifts from the contract is the point; a pin would only re-prove a known-good pairing.
The CI decision dissolved —
|
The red CI is the gate working — merge order matters
$ gh api ".../evidence.ts?ref=main" | grep -c producerOutputs
0 # defect present
$ gh api ".../evidence.ts?ref=fix/meta-198-fileindex-self-inclusion" | grep -c producerOutputs
2 # fixedThe four failures are all one cause — the producer indexing its own artifact, so run 1 and run 2 disagree: The last two are downstream of the first: an unconverged artifact means the mediated run rewrites where it should skip. Merge orderNothing to change here. The job tracks the producer's default branch deliberately, so this red is a true statement about
On Dependabot #8 (
|
The four stable read paths had a prose contract in
docs/conformance.mdand no executable one. This adds the standard-owned suite that measures a producer candidate against it, plus red tests proving each protected behavior is actually protected.Follows the existing
check:architecture/check:architecture:testidiom.It runs against a candidate, and does not skip
The producer lives in another repository, so the suite takes
WORKSPACEJSON_CLI_CANDIDATE— a built package directory.It does not skip when the candidate is absent; it exits non-zero with instructions. A conformance gate that goes green because it could not find the implementation reports conformance it never measured — the same principle as
check:examplesfailing on an empty examples directory.What is asserted
generated.fileIndexgenerated.frameworkManifest>= 0.7floor; uncorroboratedAGENTS.mdtokens not published; deterministically orderedmanual.fragileFilesmanual.coChangePatternsBeyond the paths: invalid artifacts refused rather than overwritten,
--forcemoving them aside recoverably,generated.by.namenot attributable to an invoker, output validating against the package-owned schema, a second run on an unchanged repository byte-identical, and mediated invocation producing the same artifact as direct after removing onlygenerated.generatedAt.What is deliberately NOT asserted
Per-file values inside
fileIndex. Every value field is optional in the published type, so{}is a conformant entry. Those values are behavioral, git-derived, and whether that source may enter the stable contract is an open determination made outside this repository. Requiring them would fail a producer that is behaving correctly.The scope boundary is stated in the suite header so a later reader does not mistake it for an oversight. Also unchanged: no requirement on non-empty human-owned fields, and no git-derived co-change on the acceptance surface.
The red tests carry two honesty guards
Plus a baseline asserting a clean candidate is accepted, so the suite cannot pass by rejecting everything.
It found a real defect immediately
First run against the current producer: 24/28, three failures. The producer indexed its own artifact, so run 1 and run 2 disagreed and
generate --checkwould fail on a repository's first CI run after adoption. Fixed in workspacejson/cli#12; the case that catches it is retained here. That is the watched-red → watched-green receipt.workspacejson/cliis private, so the new job needsWORKSPACEJSON_CLI_READ_TOKEN— a fine-grained PAT scoped tocontents: readon that repository and nothing else.Until that secret exists this job will fail, and that is deliberate — the alternative is a gate that skips and reports nothing. If you would rather it not block
mainbefore the token is configured, say so and I will gate the job on the secret's presence with a loud warning instead. I did not make that call unilaterally because it trades enforcement for greenness.Local run against a built candidate is green either way.
Verification
check:docsflagged internal tracker identifiers in my first draft of the workflow comment and script headers; all ten are removed and the work is described instead, per this repository's public-prose rule.