purl-next: Create PURL accuracy benchmark - ScanCode Automation

[pombredanne]

Create a pipeline in ScanCode.io that takes as an input:

• an SBOM for a PURL benchmark for a given input
• the expected list of PURLs

And creates as an output a report to validate and compare the SBOM to the expected output list of PURLs

[pombredanne]

The process could work this way:

1. For a given code input, say a Docker image or a package download URL, we run scans and create the ground source of truth for the list of packages in this input. This is the list of PURLs, one per line. This happens outside of ScanCode. We call this the "expected" PURLs
2. For that same given code input, say a Docker image or a package download URL, someone runs their tool and creates an SBOM (in CycloneDX or SPDX format). This happens outside of ScanCode. We call this the "tested" SBOM
3. In ScanCode.io, create a new project using as inputs the SBOM in 2. and the PURLs list in 1. possibly tagged as tested and expected, or using settings, or a hardcoded filename for the PURL list.

• Select the "benchmark_purls" pipeline
• That pipeline would create a simple downloadable output with a difflib.ndiff of the comparison of the sorted PURLs in the tested SBOM and the sorted PURLs in the expected PURLs.

Alternatively, it could be an add-on pipeline that works after a load_inventory pipeline, somehow getting access to the PURLs list.

We can of course refine this in the future for output and more.