feat(catalog): feature 010 Phase F — post-output comparison, PASS at zero FP/FN (T086a, T087–T092) - #92
Merged
Conversation
…mparison (T086a, T087–T092) ADR-0020 clause 5 requires two distinct steps. Step (a) — the pre-output freeze and its independent audit — landed with Barrier B. This is step (b): run the assembled generator over the frozen accept corpus and diff derived ownership against the frozen expectations at zero false positives and zero false negatives. Result: PASS. 24 expected entities, 0 false positives, 0 false negatives, 0 other mismatches. The frozen trees were not touched, and their hashes are unchanged from the values T024 recorded before any generator existed. T086a (new, prerequisite) — vendor the accept corpus. Phase E discovered the corpus was not materialized: the freeze records metadata, never descriptor files, and evidence/README.md §4 requires it stay that way. The 24 selected descriptors are vendored verbatim from backstage/community-plugins at the pinned commit into specs/010-catalog-backstage/corpus/, each verified by recomputing its git blob id from the bytes that arrived and matching it against the id the pinned tree records, with the pin cross-checked against the freeze's own 156-descriptor-file figure. Any mismatch aborts and writes nothing. The vendored bytes are pristine: no adrkit.io/owned-paths anywhere in the tree. The maintainer-authored overlay stays in overlay.json and is applied at generation time into a temporary directory that is deleted, so clause 5's "otherwise unmodified" stays provable by digest and data-model.md §10's upstream/maintainer boundary stays legible by inspection. Acquisition is a one-time step and is not reachable from the generator or the harness, asserted by scanning import specifiers rather than promised in prose. T087 — scripts/compare-accept-corpus.ts, authored after the freeze and its audit (R5 mechanism 3), with the provenance record that carries the ordering claim. T088 — the diff itself. The comparison reads both frozen artifacts and refuses to run if they disagree. Entities outside the expectation set are findings only if they derive a non-empty path set: the 24 files hold 25 entity documents, and a descriptor file count is not an entity document count. T089 — observed failing. One character in one overlay value (cli → clx) yields exactly one false negative, one false positive, and the corpus-wide union mismatch; restoring returns PASS. Retained at negative-cases/comparison-mismatch/. T090 — step (b) recomputes the frozen hashes rather than trusting them, records hashes that are its own (corpus, overlaid inputs, envelope), and renders its own verdict. It inherits nothing from step (a). T091 — the prohibition guard. Every frozen hash is compared against the value the Barrier B checkpoint recorded, and the amended-then-re-signed case is driven to show why self-consistency alone would not catch it. T092 — reporting honesty. A digest establishes integrity; this comparison establishes agreement with a maintainer-authored expectation set. Neither establishes that the adapter is correct, and every Phase F artifact is scanned for affirmative correctness and rung overclaims. T086 stays unchecked: its SC-009 limb-2 record is corrected to the new fact, but wiring a corpus pass into that close-out is T086's own remaining work. ADR-0014 rung 1 only. No release is scheduled, prepared, or implied. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
🟡 Changes recommended
Hash-guard failures can still record and exit as PASS, and task dependencies and correctness claims are inconsistent.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.
Pull request overview
Adds Phase F’s frozen-corpus comparison evidence, validating generated ownership against 24 audited expectations.
Changes:
- Vendors and verifies 24 upstream descriptors.
- Adds comparison, mutation, hash-drift, and honesty checks.
- Records zero false positives/negatives and updates phase tracking.
File summaries
| File | Description |
|---|---|
specs/010-catalog-backstage/tasks.md |
Updates Phase F tasks. |
specs/010-catalog-backstage/evidence/README.md |
Documents comparison evidence. |
specs/010-catalog-backstage/evidence/negative-cases/comparison-mismatch/README.md |
Documents the negative case. |
specs/010-catalog-backstage/evidence/negative-cases/comparison-mismatch/observed-fail.txt |
Captures observed failure. |
specs/010-catalog-backstage/evidence/negative-cases/comparison-mismatch/restored.observed.txt |
Captures restored pass. |
specs/010-catalog-backstage/evidence/comparison/README.md |
Summarizes comparison evidence. |
specs/010-catalog-backstage/evidence/comparison/diff-report.json |
Records ownership comparison. |
specs/010-catalog-backstage/evidence/comparison/expectations-unchanged.json |
Records freeze immutability. |
specs/010-catalog-backstage/evidence/comparison/harness-provenance.md |
Records harness ordering. |
specs/010-catalog-backstage/evidence/comparison/reporting-honesty.md |
Defines evidence limitations. |
specs/010-catalog-backstage/evidence/comparison/step-b-record.json |
Records step-B hashes and verdict. |
specs/010-catalog-backstage/corpus/README.md |
Documents vendored corpus. |
specs/010-catalog-backstage/corpus/VENDOR-MANIFEST.json |
Records corpus digests. |
specs/010-catalog-backstage/corpus/workspaces/acr/plugins/acr/catalog-info.yaml |
Vendors upstream descriptor. |
specs/010-catalog-backstage/corpus/workspaces/adr/plugins/adr-backend/catalog-info.yaml |
Vendors upstream descriptor. |
specs/010-catalog-backstage/corpus/workspaces/agent-forge/plugins/agent-forge/catalog-info.yaml |
Vendors upstream descriptor. |
specs/010-catalog-backstage/corpus/workspaces/airbrake/plugins/airbrake-backend/catalog-info.yaml |
Vendors upstream descriptor. |
specs/010-catalog-backstage/corpus/workspaces/allure/plugins/allure/catalog-info.yaml |
Vendors upstream descriptor. |
specs/010-catalog-backstage/corpus/workspaces/analytics/plugins/analytics-module-ga4/catalog-info.yaml |
Vendors upstream descriptor. |
specs/010-catalog-backstage/corpus/workspaces/apache-airflow/plugins/apache-airflow/catalog-info.yaml |
Vendors upstream descriptor. |
specs/010-catalog-backstage/corpus/workspaces/apollo-explorer/plugins/apollo-explorer/catalog-info.yaml |
Vendors upstream descriptor. |
specs/010-catalog-backstage/corpus/workspaces/azure-devops/plugins/azure-devops-common/catalog-info.yaml |
Vendors upstream descriptor. |
specs/010-catalog-backstage/corpus/workspaces/azure-sites/plugins/azure-sites-backend/catalog-info.yaml |
Vendors upstream descriptor. |
specs/010-catalog-backstage/corpus/workspaces/badges/plugins/badges-backend/catalog-info.yaml |
Vendors upstream descriptor. |
specs/010-catalog-backstage/corpus/workspaces/bazaar/plugins/bazaar-backend/catalog-info.yaml |
Vendors upstream descriptor. |
specs/010-catalog-backstage/corpus/workspaces/bitbucket-pull-requests/catalog-info.yaml |
Vendors upstream descriptor. |
specs/010-catalog-backstage/corpus/workspaces/bitrise/plugins/bitrise/catalog-info.yaml |
Vendors upstream descriptor. |
specs/010-catalog-backstage/corpus/workspaces/bookmarks/plugins/bookmarks/examples/component/catalog-info.yaml |
Vendors upstream descriptor. |
specs/010-catalog-backstage/corpus/workspaces/catalog/plugins/catalog-backend-module-codeowners/catalog-info.yaml |
Vendors upstream descriptor. |
specs/010-catalog-backstage/corpus/workspaces/checkmarx/plugins/checkmarx-backend/catalog-info.yaml |
Vendors upstream descriptor. |
specs/010-catalog-backstage/corpus/workspaces/cicd-statistics/plugins/cicd-statistics-module-buildkite/catalog-info.yaml |
Vendors upstream descriptor. |
specs/010-catalog-backstage/corpus/workspaces/cloudbuild/plugins/cloudbuild/catalog-info.yaml |
Vendors upstream descriptor. |
specs/010-catalog-backstage/corpus/workspaces/code-climate/plugins/code-climate/catalog-info.yaml |
Vendors upstream descriptor. |
specs/010-catalog-backstage/corpus/workspaces/code-coverage/plugins/code-coverage-backend/catalog-info.yaml |
Vendors upstream descriptor. |
specs/010-catalog-backstage/corpus/workspaces/codescene/plugins/codescene/catalog-info.yaml |
Vendors upstream descriptor. |
specs/010-catalog-backstage/corpus/workspaces/copilot/plugins/copilot-backend/catalog-info.yaml |
Vendors upstream descriptor. |
specs/010-catalog-backstage/corpus/workspaces/cost-insights/plugins/cost-insights-common/catalog-info.yaml |
Vendors upstream descriptor. |
scripts/vendor-accept-corpus.ts |
Implements verified acquisition. |
scripts/vendor-accept-corpus.test.ts |
Tests corpus verification. |
scripts/compare-accept-corpus.ts |
Implements comparison harness. |
scripts/compare-accept-corpus.test.ts |
Tests comparison and honesty. |
scripts/check-freeze-hashes.test.ts |
Extends immutability checks. |
packages/adapters/catalog-backstage/test/sc-009.test.ts |
Updates corpus availability status. |
Review details
- Files reviewed: 43/43 changed files
- Comments generated: 7
- Review effort level: Balanced
We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.
| @@ -1094,7 +1126,7 @@ the barrier exists to prevent. | |||
| Discharges: none — implements R5 mechanism 3 | |||
| Depends: T024, T086 | |||
| ownership result, and its content is fixed by an upstream commit rather than by | ||
| anything derived from the frozen expectations. | ||
| Discharges: none — unblocks T088, and unblocks SC-009 limb 2 for T086 | ||
| Depends: T024, T086 |
| * names: it must contain the forbidden phrases in order to forbid them. | ||
| */ | ||
| const SCANNED = [ | ||
| 'specs/010-catalog-backstage/evidence/comparison/harness-provenance.md', |
| // ── T088: the real run, and the report it wrote ─────────────────────────────────────── | ||
|
|
||
| describe('T088 / SC-011 \u2014 the real comparison over the frozen accept corpus', () => { | ||
| test('zero false positives and zero false negatives over all 24 annotated entities', async () => { |
| barrierSide: 'BEHIND', | ||
| discharges: ['FR-057 (step (b) half)'], | ||
| step: 'ADR-0020 clause 5 step (b) — post-output comparison', | ||
| verdict: report.verdict, |
| for (const finding of [...falseNegatives, ...falsePositives, ...otherMismatches]) { | ||
| console.error(` ${finding.kind} ${finding.canonicalId}: ${finding.reason} — ${finding.detail}`); | ||
| } | ||
| if (report.verdict === 'FAIL') process.exitCode = 1; |
Comment on lines
+290
to
+294
| "whatThisDoesAndDoesNotEstablish": { | ||
| "establishes": "That the generator’s derived ownership over this frozen corpus agrees, exactly and in order, with a maintainer-authored expectation set frozen before any generator output existed.", | ||
| "doesNotEstablish": [ | ||
| "It does not establish correctness in any absolute sense. The expectations are the maintainer’s own, hand-derived from frozen contracts; agreement between our implementation and our specification is not independent evidence that either is right.", | ||
| "The envelope’s self-digest establishes INTEGRITY, not correctness. A semantically wrong envelope can carry a perfectly valid self-digest (ADR-0020 clause 5, FR-058, SC-012).", |
CodeQL flagged `text.includes('https://api.github.com')` under
js/incomplete-url-substring-sanitization at high severity. The security
concern does not apply here - this scans our own source text to check that
exactly one script names the corpus host, and sanitizes nothing - but the
rule is right that substring-matching a URL is a poor way to reason about
hosts, and the guard had a real weakness because of it.
The old check missed `http://api.github.com` and protocol-relative
`//api.github.com`. Matching the hostname as a token catches every scheme,
so this is the stronger guard rather than a differently-spelled one.
The trailing (?![\w.-]) is what keeps it honest: it rejects
`api.github.com.evil.com`, which is the exact defect the CodeQL rule exists
to name and which a substring test would have accepted.
Verified against six cases: the real usage, http, protocol-relative, the
attack shape, an unrelated near-miss, and a clean file - all correct.
Gates: bun test 1791 pass / 0 fail, typecheck clean, check:freeze-hashes ok,
and compare-accept-corpus still PASS at 0 false positives / 0 false
negatives.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ADR-0020 clause 5 step (b) — the post-output comparison. This is the first evidence that the adapter is correct, not merely well-formed: every prior phase proved structure, and a digest-verified envelope proves integrity rather than correctness.
Result: PASS
24 expected entities · 0 false positives · 0 false negatives · 0 other mismatches.
Derived ownership diffed against expectations that were frozen, hashed, and independently audited before the generator existed. No mismatch was found, so nothing had to be reported and nothing was amended.
Independently re-verified by the coordinating session
Not accepted on report:
PASS — 24 expected entities, 0 false positive(s), 0 false negative(s), 0 other mismatch(es).check:freeze-hashesgreen throughout, and the frozen trees show an emptygit statusafter every run.The envelope carries 25 entities, not 24
One selected file (
workspaces/acr/plugins/acr/catalog-info.yaml) holds a second, unselected document. It carries no overlay, derives nothing, and is recorded as an out-of-set entity rather than a finding — it would have been a false positive had it derived any path. This is the file-count vs entity-document-count distinction showing up in live data.All three ownership states exercised in the frozen proportions: 22
explicit-paths, 1explicit-empty, 1annotation-absent. Every entityprovenance: maintainer-overlay.Frozen trees untouched — confirmed three ways
git status --porcelainover both trees is empty.bun run check:freeze-hashesgreen.expectations-unchanged.json(T091): each frozen hash recomputed now equals the value recorded in the Barrier B checkpoint — written by the independent auditor session, not by the freeze's author.e641ae5e…andf98e6d46…, unchanged.T086a — vendoring, and a defect in the instruction it was given
24 descriptors vendored verbatim and pristine to
specs/010-catalog-backstage/corpus/. Noadrkit.io/owned-pathsin any descriptor. The overlay is applied in memory into an ephemeral run dir, never on disk — preserving the separation the freeze already maintains, so clause 5's "authored upstream and otherwise unmodified" stays provable by digest and the upstream/maintainer boundary stays legible.The instruction said "verify each against the digest the freeze records". That digest does not exist. The freeze's only hashes are the two artifact-level
contentHashvalues; there is no per-descriptor digest. Rather than skip the check or invent a hash, verification was done against what the freeze does fix — the commit pin — three independent ways, all aborting:corpusFacts.descriptorFilesExactBasename— the strongest freeze-recorded cross-check available;git hash-object.Stated plainly in the vendor manifest and corpus README rather than glossed.
Offline posture (FR-018) preserved: fetching is one-time acquisition, not part of a generation run. The acquisition path is not reachable from the generator.
Observed failing (ADR-0016)
--observe-failingmutates one overlay value,cli→clx, one character:Restored → PASS, exit 0. Retained at
negative-cases/comparison-mismatch/. Every kernel failure mode is separately driven by an input built to trip it. T091 additionally drives the amended-then-re-signed case — self-consistent, and so would slip past the drift check alone.Two judgement calls
- [ ]. Vendoring closed its blocker and its limb-2 record was asserting something now false, so the fact and the assertion were corrected — but discharging SC-009 limb 2 means wiring a corpus pass into that close-out, which is T086's work, not T086a's.evidence/README.md§4 amended additively. Its "no generator output / no comparison harness" absences stopped being true of the directory the moment Phase F deposited. A §4.1 scopes them to the freeze trees, where they still hold verbatim, and records whatcomparison/now holds. No original sentence deleted. Flagged because it touches a Barrier B document.Checks
bun test1791 pass / 0 fail (was 1721) ·typecheckclean ·check:depsok ·check:freeze-hashesok ·adr lint20 records, 0 errors.Standing
ADR-0014 rung 1 only. This PASS does not assert rung 2 or rung 3, does not ratify 24 as a production sample size, and does not clear ADR-0012 gate 3 as an external claim. Gate 4 remains unmet and not yet testable. No release scheduled, implied, or prepared.