SBOM: emit spec-valid purl types instead of '???' placeholders

[villelaitila]

Problem

bom_ref() in the CycloneDX generator falls back to pkg:???/<name>@<version> when it
cannot resolve an ecosystem, and to pkg:??Java/... for elements under a parent named
Java.

Neither is a valid package URL. The purl spec
requires the type to begin with a letter and hold only letters, digits, . and -, and to
be canonically lowercase; the spec's own test vectors
reject a ? in the type. Validating consumers (Dependency-Track, for example) reject such
components outright, so they are never matched against vulnerability databases — the
affected components are invisible in exactly the tooling an SBOM exists to feed.

In a large real-world model (674 SBOMs, ~12.5k components) this affected 115 components,
several of them packages with known critical CVEs.

Cause

The affected elements all share one shape: a binary committed into the repository and
referenced from source, with

• either no repotype attribute, or one that names no ecosystem — the analyzer's own
  record that it could not identify one,
• no ecosystem-named ancestor (NPM, PIP, Maven, Assemblies),
• incoming ref associations from t="file" elements — typically .dll, sometimes .exe.

These are legacy .NET <Reference><HintPath> pointers at checked-in assemblies, not
package-manager dependencies. Both existing inference paths (attribute, ancestor name) have
nothing to read. The one remaining signal is the extension of the referencing file.

Change

This is two things, and it is worth separating them because only one is spec-backed:

• A conformance fix. ??? and ??Java are invalid types; generic is the correct
  fallback. No judgement is involved and the spec settles it.
• A labelled ecosystem inference. Guessing nuget from a referencing .dll is an
  inference the spec neither blesses nor forbids — nuget identifies packages from
  nuget.org, not .NET assemblies in general, and purl has no type for a bare assembly. It
  is justified empirically, by assembly names routinely equalling NuGet package ids.

The purlTypeResolution property below is what keeps the two distinguishable in the
output, so a consumer can filter the second out and keep the first.

1. infer_pkgtype_from_referencing_files(elem) derives a purl type from the extensions of
   the files referencing the element, via a documented mapping (dll/exe/nupkg →
   nuget, whl/egg → pypi, gem → gem). A type is inferred only where the
   inferred name is by itself a complete package identifier for that type — see
   "Why dll/exe map to nuget but .jar does not" below. Extensions are counted
   rather than taken first-hit, since a binary can be referenced from several file kinds;
   ties break alphabetically so output stays byte-stable across runs.
2. The unresolved fallback becomes generic — the purl type for packages that fit no
   other type, and the only registered type with no repository behind it — instead of
   ???. The spec asks that another type be used where possible; inference runs first and
   generic is reached only when it finds nothing, which satisfies that constraint. (A
   sid type for binary-only software with no registry behind it is proposed but not
   adopted, so generic is correct today and there is a named successor to watch.)
3. The ??Java branch is removed. Elements under a Java parent go through the same
   inference, which for JVM artifacts resolves to generic (see limitation 2).
4. analyze_3rdparty() now skips a component whose bom-ref it has already emitted,
   mirroring the seen_refs guard that _collect_3rdparty_for_subtree() has always had.
   Without it, making previously-distinct purls agree would make them collide as
   bom-refs — and CycloneDX requires those to be unique within a BOM. JSON Schema cannot
   express that constraint, but the XML schema can and does, at document scope: a duplicate
   rejects the whole document rather than one component. Fixing purl types while opening
   that hole would trade a component-scoped defect for a document-scoped one.
5. bom_ref() becomes a thin wrapper over a new purl_for() that additionally returns
   provenance properties. The signature, return type and behaviour of bom_ref() are
   unchanged.

Provenance: guesses are labelled

Any component whose type was inferred or fell back carries a purlTypeResolution property:

"purl": "pkg:nuget/Example.Web.UI@2019.1.115.45",
"properties": [
  { "name": "purlTypeResolution",
    "value": "inferred from referencing file extension: dll" }
]

A type read from repotype or from an ecosystem-named ancestor gets no such property. Read
that absence precisely: it means the type was not inferred by this mechanism, not that it
is certainly right. The pre-existing ancestor-name branches are themselves heuristics — they
are simply older and unlabelled, and relabelling them would change output for every existing
component, so it is left alone here. The value cites only the extensions
that voted for the winning type, so it never lists evidence that argued against the type
actually chosen.

component.properties is CycloneDX's sanctioned place for this and is what the module
already uses for sourceCodeReferences. component.evidence.identity (technique
filename) would be the standards-idiomatic home for identity provenance specifically, and
is the natural follow-up; it is not adopted here because it would make this one field
inconsistent with the module's existing property-based convention for no gain in validity.

Migration note: anyone currently grepping for pkg:??? as an "unresolved ecosystem"
marker should switch to the purlTypeResolution property with the value
ecosystem unresolved, which is now the discriminator. This matters more than a rename:
??? announced its own failure, whereas pkg:generic/... looks plausible while matching
nothing. The property is what keeps unresolved components findable (see limitation 7).

Why dll/exe map to nuget but .jar does not

The rule governing the map: infer a type only when the inferred name is by itself a
complete identifier for that type. Whether that holds depends on the purl type definition,
and the two ecosystems differ.

nuget prohibits a namespace — the package id is the whole identity. A HintPath assembly
reference is not proof of a NuGet package, but assembly names routinely match real package
ids, so the inferred purl carries a complete identifier and stands a real chance of
matching. An in-house
assembly gets a purl that matches nothing in any database — a harmless miss rather than a
false alarm.

maven requires a groupId namespace, and Maven Central identity is groupId:artifactId.
A file extension supplies no groupId and no way to derive one, so pkg:maven/<artifact>@<v>
would be malformed and unmatchable — an artifact name alone is ambiguous across groups.
Mapping jar/war/aar to maven would therefore trade one class of invalid purl for
another, which is exactly what this change exists to stop. JVM artifacts fall through to
generic instead.

The purlTypeResolution property keeps every inference auditable either way. Note that it
cannot rescue a malformed purl: a validating consumer rejects the component before any
property is read, which is why the completeness rule gates the map rather than provenance
alone.

Effect on a real model (reported, not reproduced here)

These figures come from the environment where the problem was found: same model, same
command, this change the only variable, measured on the per-element path
(generate_multi_from_sgraph). That environment is not reachable from this repository, so
they are reported rather than independently reproduced. "What was verified here" below
separates the two.

	before	after
??? types	115	0
purls with a spec-invalid type	115	0
nuget	6457	6539
generic	0	2
npm / docker / pypi	unchanged	unchanged
components total	12490	12459

The component total drops by 31 because those formerly-??? components now resolve to
purls that already existed as NuGet components in the same SBOM. The same package had been
counted twice — once as an assembly_ref from a csproj/vbproj, once as the committed DLL —
and now collapses to one entry. This is a correction, not lost data.

Reduced to the smallest case that shows it, with bom_ref as the only variable:

before : 4 components   pkg:nuget/Ionic.Zip@1.9.1.8
                        pkg:???/Ionic.Zip@1.9.1.8      <- same package, counted twice
after  : 3 components   pkg:nuget/Ionic.Zip@1.9.1.8    <- one entry

Both output paths now collapse such a pair, because change 4 gives the single-SBOM path
the same bom-ref guard the per-element path already had. Before that guard the two
components would have survived side by side there, sharing one bom-ref.

What was verified here

Measured over eight models available to this change, single-SBOM path, before and after:

                            before   after
components                    1761     1729
unique bom-refs               1729     1729
purls with an invalid type       6        0
duplicate bom-refs              32        0

The six become generic, each carrying a purlTypeResolution property.

The conformance half is therefore reproduced rather than reported: real models emit invalid
purl types today and stop after the change. The last row is the bom-ref guard on its own,
and it is the clearer statement of what that guard does — those 32 duplicates existed
beforehand and CycloneDX does not permit them, so the guard repairs a live document-level
defect rather than only preventing one this change would otherwise have introduced. The
32-component drop is the same guard, concentrated entirely in one of the eight models
(limitation 5).

Note what those six became — generic, every one of them, and no mapped ecosystem type.
The inference half never fired here, because no model available to this change contains a
single .dll, .exe, .nupkg, .whl, .egg or .gem; there is one .jar in total.

Also reproduced, against the new fixtures: the generic fallback, the presence and absence
of the provenance property, the alphabetical tie-break, the removal of the ??Java branch,
and the deduplication pair shown above.

The map's entries do not all rest on the same kind of evidence, and it is worth being
explicit about which:

entry	evidence
nupkg → nuget	true by definition — a .nupkg is a NuGet package
dll/exe → nuget	structural validity from the nuget type definition; field support reported from the environment above, not reproduced here
whl/egg → pypi, gem → gem	structural validity only; no field evidence of any kind, in either environment

The reported extension distribution and component counts therefore rest on the reference
environment alone.

The separation that matters for review: the change's justification is the conformance
argument — measured above, six invalid types to zero — plus the structural validity of each
mapped type, which is checkable from the purl type definitions. Neither needs the reported
figures. Those are the change's motivation: they say the problem was worth fixing and at
what scale, not whether the fix is correct. Nothing in the design depends on them, which is
why they are attributed rather than asserted.

Known limitations

1. In-house assemblies are not distinguished from third-party ones. An in-house DLL that
   happens to share a name with a real NuGet package produces a false-positive purl. A
   precise fix needs model-root access inside purl_for(); out of scope here.
2. JVM binaries (.jar/.war/.aar) resolve to generic and so get no vulnerability
   matching, for the reason given above: a maven purl needs a groupId that no file
   extension can supply. Recovering coordinates from a JAR's META-INF/maven/*/pom.properties,
   or from a pom.xml elsewhere in the model, would make the inference possible later. Out
   of scope here, and deliberately left as an honest gap rather than a plausible-looking
   guess that would match nothing anyway.
3. Inference reads the element's own incoming references and its parent's, matching what
   produce_source_code_references() already does. Where several versioned children share
   one name directory, a reference landing on that directory votes for every unresolved
   child, so the recorded evidence can be broader than the evidence for one specific child.
4. Where an inferred component collapses into a pre-existing equivalent on the per-element
   path, the surviving sourceCodeReferences point at the DLL rather than at the csproj.
   Because seen_refs keeps whichever component it met first, the survivor can also lose a
   purlTypeResolution label its twin carried, or keep one when the discarded twin was
   authoritatively typed. That is at stake only where a collision pairs an inferred
   component with an ancestor-resolved one: there exactly one twin carries the label, so
   traversal order decides which survives. A collision can equally pair two
   ancestor-resolved components, with no label on either side and nothing to lose — which
   describes every one of the 32 collisions measured across the models available here. So
   where this bites is the mixed class, and no mixed collision occurs in any model
   available to this change. It is inferred rather than measured that the reference
   environment's merges were of that kind: they are reported there as formerly-???
   components meeting the assembly_ref twins they had been double-counted against, and
   that pairing is one inferred component and one ancestor-resolved one. This is existing
   _collect_3rdparty_for_subtree behaviour, not introduced here, but this change makes
   more components collide and so makes it more visible.
5. The bom-ref guard on the single-SBOM path also collapses duplicates that predate this
   change — any two externals resolving to the same purl already collided there. Measured
   across eight models, seven were unaffected and one lost 32 of 261 components, so the
   effect is real but concentrated. That model's single-SBOM output already carried 32
   duplicate bom-refs and was invalid under the XML schema before any change here, so
   the guard repairs it rather than degrading it. No distinct package is lost: collapse is
   by identical purl, every unique purl survives, and vulnerability matching keys on purl.
   What narrows is sourceCodeReferences — the guard discards rather than merges, and the
   survivor is whichever the traversal reaches first, so it may be the twin with the
   shorter reference list. Merging the two lists instead would preserve them, but that is
   a different semantic than the per-element path has ever applied and belongs in its own
   change. One class this closes was already invalid under JSON Schema too: two
   indistinguishable unresolved elements produced byte-identical component objects, which
   bom.components forbids via uniqueItems.
6. Only the purl type is fixed, so the regression test guards the type only. Two
   independent conformance gaps remain and are deliberately not touched: package names
   are emitted unencoded, so pre-existing components whose names contain literal spaces
   stay non-conformant; and the existing Maven ancestor branch emits pkg:maven/<name>
   without the groupId namespace that the maven type requires — the same namespace problem
   that keeps jar out of the inference map, in a branch this change does not touch.
   Widening the fix to those would change output well beyond the paths measured above.
7. Ecosystem buckets are matched at inconsistent nesting depths. NPM, APT, PIP and
   Assemblies match a parent or grandparent; Python, Go and Maven match only a
   direct parent, although the documented External/ layout nests packages one level
   deeper. Packages nested under Python, Go or Maven therefore miss their branch and
   fall to inference, which has no mapping for .py or .go and so yields generic.
   Before this change they yielded ???. Both are wrong, but ??? was loud and greppable
   where generic looks plausible, so this change makes an existing failure quieter — the
   purlTypeResolution property is what keeps it detectable. Aligning those three branches
   is a small, obvious follow-up; it is excluded here because it changes branches the
   measurements above did not exercise.
8. Licenses remain empty for these components — a separate concern.

Tests

New fixture tests/converters/modelfile_for_sbom_binary_refs_tests.xml reproduces the
production shape: both attribute variants — absent, and a repotype naming no ecosystem —
with the version in the ' of version ' name suffix and a ref association from a
t="file" element. New tests cover each mapped extension family, the
generic fallback, presence and absence of the provenance property, the alphabetical
tie-break that keeps output byte-stable, and the removal of the ??Java branch.
A .jar-referenced binary is asserted to yield generic rather than maven, so
the deliberate exclusion is pinned by a test and cannot be "fixed" back into a defect. One
regression test asserts the spec invariant itself over every generated purl:

PURL_TYPE_PATTERN = re.compile(r'^pkg:[a-z][a-z0-9.-]*/')

encoding the rule rather than a list of expected strings, so future ecosystem additions
cannot silently reintroduce an invalid type.

The guard is deliberately type-scoped, and the fixture carries a witness for why: one
component emits a maven purl whose name contains a literal space. That component is
asserted as-is rather than corrected, so the name-encoding gap in limitation 6 is recorded
as executable characterization rather than left as prose — and so a future full-purl
validator has a known case to fail against.

Existing tests are unchanged and pass.

[softagram-bot]

Softagram Impact Report for pull/165 (head commit: fc659fc)

TL;DR Changed code files: 3 | Directly impacted code files: 1

⭐ Change Overview

(Open in Softagram Desktop for full details)

⭐ Details of Dependency Changes

(Open in Softagram Desktop for full details)

[]

📄 Full report

• Permalink: Full report for pull/165

Impact Report explained. Give feedback on this report to support@softagram.com