fix(conformance): measure the package fields over packages only - #23
Merged
Merged
Conversation
A file on disk carries no package version and no PURL type — purl defines none for one — so counting file entries against the package fields reports a document as short on what its entries cannot hold. Upstream measured a firmware SBOM whose packages were all identified at 10% PURL coverage because fourteen file entries sat in the denominator; 18% without them. Excluding a type from that denominator is also how a shell SBOM slips past the mandatory checks, which is why the all-data guard exists. It widens with the exclusion: whatever empties the package denominator now warns rather than passing. Upstream passes both shapes — a report-only tool can afford that, a governance badge cannot. Files are asked instead for the identifier they can carry, and the coverage details name how many components were measured against how many were left out, since a percentage alone cannot say whether it was measured over two or fifty. Security review caught two defects in the first cut: set membership on an unhashable `type` raised where the `!=` it replaced could not, taking the whole ingest down, and a non-string component name reached the JSONB sanitiser, which iterates characters. Both are fixed with the adversarial cases pinned.
This was referenced Aug 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The two mandatory component checks — 100% name+version, PURL coverage ≥ 90% — count a denominator of "package components". A file on disk cannot answer either: it has no package version, and purl defines no type for one. Counting file entries against those fields reports a document as short on what its entries cannot hold. Upstream measured a firmware SBOM whose packages were all identified at 10% PURL coverage because fourteen file entries sat in the denominator, and 18% once they did not.
Excluding a type from that denominator is also exactly how a shell SBOM slips past the mandatory checks, which is why this repository already carries an anti-evasion guard for
data: type everythingdata, the denominator hits zero, and the empty-denominator handling passes the document. That guard downgrades the shape to a warning instead, with the reason in the code — a report-only CLI can afford to pass it, a governance badge cannot. Addingfileto the exclusion without widening that guard would have moved the evasion from one type to another rather than closing it, so the guard now fires on whatever empties the denominator. This is a deliberate divergence from upstream, which passes both shapes.Files are asked instead for the identifier they CAN carry, by a new recommended
file-hashcheck. It counts towardn_warnlike license and hash coverage do — a file entry with no hash carries no identifier at all — which is stated in the code because the neighbouring regulatory checks are deliberately verdict-neutral and the difference is easy to misread.The fixtures are captured supplier output rather than hand-built JSON, per the hardening rule that says the density of real documents is where these defects live: one with two libraries and six file entries (the shape that gains), one with file entries alone (the shape that must not pass).
Security review found two defects in the first cut, both introduced here. The exclusion was written as
type not in {frozenset}, and set membership raisesTypeErroron an unhashable value —{"type": {}}— where the!=comparison it replaced could not. The module promises never to raise on malformed input, and the promise is kept at the persist boundary of a Celery task, so this failed the whole ingest rather than one row. Separately,_namecould return a non-string that reached the JSONB sanitiser, which iterates characters; that path predates this change but the new check added a third caller of it. Both are fixed at the root with the adversarial shapes parametrized.The review also raised a dilution channel that this change does not close: leave one real package and re-type the rest as files, and the badge passes on a denominator of one. That is not new — the same is possible by re-typing to
data, or by simply omitting the components — and closing it properly means deciding whether this verdict is a quality bar or evidence against tampering. It is a quality bar, so the coverage details now name how many components were measured against how many sat outside, and the guide says plainly that the badge cannot see a component that was left out or re-typed. Two further findings are recorded as separate work:missing[]entries have no per-item length cap on the conformance persist path (pre-existing), and Trivy may match nested components that the persisted graph does not carry (out of scope).Verified: backend unit suite matches
mainon this machine (39 failures either way, list-compared, zero new), ruff clean, mypy clean over 616 files, frontend 1984 passing with the mirror contract extended for the new check id,tools/ko-style/lint.mjs --all --fail-on S2clean. The captured verdict baselines were re-recorded and the summary counters did not move on any fixture.