Skip to content

[CLI] Make the parity harnesses gate, and wire them into CI - #6

Merged
qmarcelle merged 1 commit into
mainfrom
feat/parity-harness-ci-gate
Jul 26, 2026
Merged

[CLI] Make the parity harnesses gate, and wire them into CI#6
qmarcelle merged 1 commit into
mainfrom
feat/parity-harness-ci-gate

Conversation

@qmarcelle

Copy link
Copy Markdown
Contributor

The harnesses reported; they did not gate. parity-agents-audit-runtime.sh exited 0 with two failing checks — verified. Adding it to CI as-is would have created a check that can never fail, which is worse than no check because it reads as coverage.

Gating mechanism

migration/parity-expected-differences.txt records the differences a human ratified, each citing the deciding issue. The runtime harness compares the observed set against that baseline and fails when the set changes, in either direction:

Condition Result
a check differs and is not listed REGRESSION — something meant to stay compatible changed
a listed check starts matching STALE BASELINE — a ratified decision was reverted, or the entry is now a lie

The second direction matters as much as the first. Without it, quietly undoing META-236's vendor-notice ruling would have made CI greener, not redder.

parity-agents-audit-pack.sh gates on the eleven packed identity fields — name, version, bin, main, module, types, exports, files, engines, type, publishConfig. Those are how a consumer resolves the package. Its other differences (content-hashed tsup chunk name, description, the added @workspacejson/cli dependency) are deliberate META-247 consequences and stay informational.

parity-datahub-shim.mjs already exited non-zero correctly — no change needed.

A flake found while building the gate

The runtime harness was intermittently reporting a third difference, roughly 1 run in 7. Reproduced and diagnosed:

9c9
<       "temporalWeight": 1,
---
>       "temporalWeight": 0.9999999998842592,

temporalWeight is a time-decayed float in @workspacejson/rules. The old and new sides cannot execute at the same instant, so a sub-millisecond gap produces that drift. It is engine nondeterminism, not a migration difference — the same class as durationMs and timestamps — and is now normalized alongside them. 10 consecutive runs clean afterwards.

This flake predates the gate and would have made CI fail at random. It only surfaced because the gate made the difference count meaningful — which is a fair argument that the gate was worth building before wiring anything up.

Red-tested, both directions

unexpected difference   (renamed a command)      -> exit 1, REGRESSION
stale baseline          (bogus expected entry)   -> exit 1, STALE BASELINE
restored                                         -> exit 0

CI

A separate parity job. It clones and builds a second repository, which the unit-test matrix should not pay for, and it needs no Node matrix. The frozen source is pinned to an immutable commit, so that SHA is the cache key — it can never go stale for the wrong reason.

Expected steady-state cost: cache miss builds the old side once; cache hits run in well under a minute.

Scope

Harness and CI only. No package, producer or guard behavior changed. No parity assertion was weakened — one volatile field was added to the existing normalization list, and the count of substantive compared behaviors is unchanged at 29.

The harnesses reported; they did not gate. `parity-agents-audit-runtime.sh`
exited 0 with two failing checks, so adding it to CI as-is would have created a
check that can never fail — worse than no check, because it reads as coverage.

Gating mechanism
migration/parity-expected-differences.txt records the differences that a human
ratified, each citing the deciding issue. The runtime harness now compares the
observed difference set against that baseline and fails when the set CHANGES,
in either direction:

  a check differs and is not listed  -> REGRESSION
  a listed check starts matching     -> STALE BASELINE (a ratified decision was
                                        reverted, or the entry is now a lie)

The second direction matters as much as the first. Without it, quietly undoing
the META-236 vendor-notice ruling would have made CI greener, not redder.

parity-agents-audit-pack.sh now gates on the eleven packed identity fields —
name, version, bin, main, module, types, exports, files, engines, type,
publishConfig. Those are how a consumer resolves the package. Its other
differences (content-hashed tsup chunk name, description, the added
@workspacejson/cli dependency) are deliberate META-247 consequences and stay
informational. parity-datahub-shim.mjs already exited non-zero correctly.

Flake found and fixed while building the gate
The runtime harness was intermittently reporting a third difference, roughly 1
run in 7. Reproduced and diagnosed: `temporalWeight` is a time-decayed float in
@workspacejson/rules, so a sub-millisecond gap between the old and new runs
yields 1 vs 0.9999999998842592. That is engine nondeterminism, not a migration
difference — the same class as `durationMs` and timestamps — so it is now
normalized alongside them. 10 consecutive runs clean afterwards.

This flake predates the gate and would have made CI fail at random. It only
surfaced because the gate made the difference count meaningful.

Red-tested, both directions
  unexpected difference  (renamed a command)      -> exit 1, REGRESSION
  stale baseline         (bogus expected entry)   -> exit 1, STALE BASELINE
  restored                                        -> exit 0

CI
Separate `parity` job: it clones and builds a second repository, which the
unit-test matrix should not pay for, and it needs no Node matrix. The frozen
source is pinned to an immutable commit, so that SHA is the cache key — it can
never go stale for the wrong reason.
Copilot AI review requested due to automatic review settings July 26, 2026 12:18

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@qmarcelle
qmarcelle merged commit 0334796 into main Jul 26, 2026
3 checks passed
@qmarcelle
qmarcelle deleted the feat/parity-harness-ci-gate branch July 27, 2026 03:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants