Skip to content

fix(conformance): measure the package fields over packages only - #23

Merged
haksungjang merged 1 commit into
mainfrom
fix/file-components-coverage-denominator
Aug 8, 2026
Merged

fix(conformance): measure the package fields over packages only#23
haksungjang merged 1 commit into
mainfrom
fix/file-components-coverage-denominator

Conversation

@haksungjang

Copy link
Copy Markdown
Contributor

The two mandatory component checks — 100% name+version, PURL coverage ≥ 90% — count a denominator of "package components". A file on disk cannot answer either: it has no package version, and purl defines no type for one. Counting file entries against those fields reports a document as short on what its entries cannot hold. Upstream measured a firmware SBOM whose packages were all identified at 10% PURL coverage because fourteen file entries sat in the denominator, and 18% once they did not.

Excluding a type from that denominator is also exactly how a shell SBOM slips past the mandatory checks, which is why this repository already carries an anti-evasion guard for data: type everything data, the denominator hits zero, and the empty-denominator handling passes the document. That guard downgrades the shape to a warning instead, with the reason in the code — a report-only CLI can afford to pass it, a governance badge cannot. Adding file to the exclusion without widening that guard would have moved the evasion from one type to another rather than closing it, so the guard now fires on whatever empties the denominator. This is a deliberate divergence from upstream, which passes both shapes.

Files are asked instead for the identifier they CAN carry, by a new recommended file-hash check. It counts toward n_warn like license and hash coverage do — a file entry with no hash carries no identifier at all — which is stated in the code because the neighbouring regulatory checks are deliberately verdict-neutral and the difference is easy to misread.

The fixtures are captured supplier output rather than hand-built JSON, per the hardening rule that says the density of real documents is where these defects live: one with two libraries and six file entries (the shape that gains), one with file entries alone (the shape that must not pass).

Security review found two defects in the first cut, both introduced here. The exclusion was written as type not in {frozenset}, and set membership raises TypeError on an unhashable value — {"type": {}} — where the != comparison it replaced could not. The module promises never to raise on malformed input, and the promise is kept at the persist boundary of a Celery task, so this failed the whole ingest rather than one row. Separately, _name could return a non-string that reached the JSONB sanitiser, which iterates characters; that path predates this change but the new check added a third caller of it. Both are fixed at the root with the adversarial shapes parametrized.

The review also raised a dilution channel that this change does not close: leave one real package and re-type the rest as files, and the badge passes on a denominator of one. That is not new — the same is possible by re-typing to data, or by simply omitting the components — and closing it properly means deciding whether this verdict is a quality bar or evidence against tampering. It is a quality bar, so the coverage details now name how many components were measured against how many sat outside, and the guide says plainly that the badge cannot see a component that was left out or re-typed. Two further findings are recorded as separate work: missing[] entries have no per-item length cap on the conformance persist path (pre-existing), and Trivy may match nested components that the persisted graph does not carry (out of scope).

Verified: backend unit suite matches main on this machine (39 failures either way, list-compared, zero new), ruff clean, mypy clean over 616 files, frontend 1984 passing with the mirror contract extended for the new check id, tools/ko-style/lint.mjs --all --fail-on S2 clean. The captured verdict baselines were re-recorded and the summary counters did not move on any fixture.

A file on disk carries no package version and no PURL type — purl defines
none for one — so counting file entries against the package fields reports a
document as short on what its entries cannot hold. Upstream measured a
firmware SBOM whose packages were all identified at 10% PURL coverage
because fourteen file entries sat in the denominator; 18% without them.

Excluding a type from that denominator is also how a shell SBOM slips past
the mandatory checks, which is why the all-data guard exists. It widens with
the exclusion: whatever empties the package denominator now warns rather
than passing. Upstream passes both shapes — a report-only tool can afford
that, a governance badge cannot.

Files are asked instead for the identifier they can carry, and the coverage
details name how many components were measured against how many were left
out, since a percentage alone cannot say whether it was measured over two or
fifty.

Security review caught two defects in the first cut: set membership on an
unhashable `type` raised where the `!=` it replaced could not, taking the
whole ingest down, and a non-string component name reached the JSONB
sanitiser, which iterates characters. Both are fixed with the adversarial
cases pinned.
@haksungjang
haksungjang merged commit 4caeaa2 into main Aug 8, 2026
24 checks passed
@haksungjang
haksungjang deleted the fix/file-components-coverage-denominator branch August 8, 2026 18:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant