[CLI] Make the parity harnesses gate, and wire them into CI - #6
Merged
Conversation
The harnesses reported; they did not gate. `parity-agents-audit-runtime.sh`
exited 0 with two failing checks, so adding it to CI as-is would have created a
check that can never fail — worse than no check, because it reads as coverage.
Gating mechanism
migration/parity-expected-differences.txt records the differences that a human
ratified, each citing the deciding issue. The runtime harness now compares the
observed difference set against that baseline and fails when the set CHANGES,
in either direction:
a check differs and is not listed -> REGRESSION
a listed check starts matching -> STALE BASELINE (a ratified decision was
reverted, or the entry is now a lie)
The second direction matters as much as the first. Without it, quietly undoing
the META-236 vendor-notice ruling would have made CI greener, not redder.
parity-agents-audit-pack.sh now gates on the eleven packed identity fields —
name, version, bin, main, module, types, exports, files, engines, type,
publishConfig. Those are how a consumer resolves the package. Its other
differences (content-hashed tsup chunk name, description, the added
@workspacejson/cli dependency) are deliberate META-247 consequences and stay
informational. parity-datahub-shim.mjs already exited non-zero correctly.
Flake found and fixed while building the gate
The runtime harness was intermittently reporting a third difference, roughly 1
run in 7. Reproduced and diagnosed: `temporalWeight` is a time-decayed float in
@workspacejson/rules, so a sub-millisecond gap between the old and new runs
yields 1 vs 0.9999999998842592. That is engine nondeterminism, not a migration
difference — the same class as `durationMs` and timestamps — so it is now
normalized alongside them. 10 consecutive runs clean afterwards.
This flake predates the gate and would have made CI fail at random. It only
surfaced because the gate made the difference count meaningful.
Red-tested, both directions
unexpected difference (renamed a command) -> exit 1, REGRESSION
stale baseline (bogus expected entry) -> exit 1, STALE BASELINE
restored -> exit 0
CI
Separate `parity` job: it clones and builds a second repository, which the
unit-test matrix should not pay for, and it needs no Node matrix. The frozen
source is pinned to an immutable commit, so that SHA is the cache key — it can
never go stale for the wrong reason.
This was referenced Jul 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The harnesses reported; they did not gate.
parity-agents-audit-runtime.shexited0with two failing checks — verified. Adding it to CI as-is would have created a check that can never fail, which is worse than no check because it reads as coverage.Gating mechanism
migration/parity-expected-differences.txtrecords the differences a human ratified, each citing the deciding issue. The runtime harness compares the observed set against that baseline and fails when the set changes, in either direction:The second direction matters as much as the first. Without it, quietly undoing META-236's vendor-notice ruling would have made CI greener, not redder.
parity-agents-audit-pack.shgates on the eleven packed identity fields —name,version,bin,main,module,types,exports,files,engines,type,publishConfig. Those are how a consumer resolves the package. Its other differences (content-hashed tsup chunk name, description, the added@workspacejson/clidependency) are deliberate META-247 consequences and stay informational.parity-datahub-shim.mjsalready exited non-zero correctly — no change needed.A flake found while building the gate
The runtime harness was intermittently reporting a third difference, roughly 1 run in 7. Reproduced and diagnosed:
temporalWeightis a time-decayed float in@workspacejson/rules. The old and new sides cannot execute at the same instant, so a sub-millisecond gap produces that drift. It is engine nondeterminism, not a migration difference — the same class asdurationMsand timestamps — and is now normalized alongside them. 10 consecutive runs clean afterwards.This flake predates the gate and would have made CI fail at random. It only surfaced because the gate made the difference count meaningful — which is a fair argument that the gate was worth building before wiring anything up.
Red-tested, both directions
CI
A separate
parityjob. It clones and builds a second repository, which the unit-test matrix should not pay for, and it needs no Node matrix. The frozen source is pinned to an immutable commit, so that SHA is the cache key — it can never go stale for the wrong reason.Expected steady-state cost: cache miss builds the old side once; cache hits run in well under a minute.
Scope
Harness and CI only. No package, producer or guard behavior changed. No parity assertion was weakened — one volatile field was added to the existing normalization list, and the count of substantive compared behaviors is unchanged at 29.