Group non-package files together for analysis and reporting purposes

[DennisClark]

Scan results are currently focused on packages, but in a typical product codebase there are lots of files that are not (or no longer) associated with packages, and these files may be very important from a license compliance perspective. Some cases include:

• first-party files that could be reported together as belonging to the product copyright holder under the product license expression
• files that have been extracted from third-party archives and used individually, often containing copyright or license statements or both

These miscellaneous files are currently difficult to analyze in the product codebase scan results, especially when there are lots of them, and they are not always being included in attribution and SBOM outputs.

Consider grouping these miscellaneous files into "local" packages (or "custom", "virtual", "logical", "file_set" packages), perhaps combining them by license_expression. It may also make sense to define these using the PURL convention; consider:

Type: "local" (or some other descriptive label that distinguishes them from standard packages)
Namespace: the name of the product codebase
Name: the common license_expression for the group of files
Version: the version of the product codebase
Qualifiers: optional, most likely null
Subpath: optional, most likely null

The primary advantage of creating "local" packages in the scan results are:

• organize the data for review and analysis, where the assigned PURL makes it clear that these "packages" are a special case
• take advantage of the existing ScanCode.io UI
• take advantage of existing code that generates attribution and SBOMs

I prefer the term "local" for the PURL Type value, but this is of course open to discussion.

[tdruez]

@DennisClark Could you clarify the following:

• "the name of the product codebase"
• "the version of the product codebase"

There's no concept of Product in the ScanCode.io context, the Pipelines/Scans are regrouped by "Analysis projects".

[DennisClark]

@tdruez I guess that we would have to use the scancode.io project name for the Name, and we don't have any value for the version field.

[DennisClark]

but the best solution would be to add the concept of Product (product name + product version) to SCIO to improve integration, ultimately, with other applications.

[mjherzog]

In the big picture we need to be sure that we fully support reporting of first-party code, especially first-party code that has a copyright or license notice.

[DennisClark]

let's go with this:
Type: "local" (or some other descriptive label that distinguishes them from standard packages)
Namespace: the name of the SCIO Project
Name: the common license_expression for the group of files

[DennisClark]

alternative value for "Name": a UUID

[DennisClark]

Let's go with this:
Type: "local-files"
Namespace: a generated UUID
Name: the common license_expression for the group of files

the rest of the PURL is probably null

[DennisClark]

Let's go with this:
Type: "local-files"
Namespace: the SCIO project as a slug
Name: generated UUID

the rest of the PURL is probably null

[tdruez]

PR #927 merged in main.

The "local-files" packages are now created as part of the d2d pipeline.

[DennisClark]

If possible and practical, I think that this new feature should be expanded to include these pipelines (and possibly others):

• scan_codebase
• scan_single_package