Add license information to SPDX SBOMs

[imjasonh]

This relies on a new package, https://github.com/imjasonh/golicenses (name TBD) that weekly scrapes the deps.dev public dataset to produce a gzipped CSV of Go package names -> their license(s), //go:embeds it, and makes it available via golicenses.Get(p).

deps.dev itself periodically scrapes GitHub etc to detect licenses using whatever method it uses to do that, it doesn't matter.

The dependency adds ~4MB to the size of the binary (32MB -> 36MB), and golicenses.Get takes ~200ms the first time it's called, then ~100ns each subsequent call.

Before this change:

$ cosign download sbom $(ko build) | jq -r ".packages[].licenseConcluded" | sort | uniq -c | sort -n -r
...
Found SBOM of media type: spdx+json
 111 NOASSERTION

After this change:

$ cosign download sbom $(go run ./ build) | jq -r ".packages[].licenseConcluded" | sort | uniq -c | sort -n -r
...
Found SBOM of media type: spdx+json
  49 Apache-2.0
  21 MIT
  19 BSD-3-Clause
  13 NOASSERTION
   2 MPL-2.0
   2 LGPL-3.0,MIT
   2 BSD-2-Clause
   2 Apache-2.0,CC-BY-SA-4.0
   1 BSD-3-Clause,MIT
   1 Apache-2.0,BSD-3-Clause,ISC,MIT

[codecov-commenter]

Codecov Report

Merging #766 (f744553) into main (2b470fc) will not change coverage.
The diff coverage is n/a.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@           Coverage Diff           @@
##             main     #766   +/-   ##
=======================================
  Coverage   52.42%   52.42%           
=======================================
  Files          43       43           
  Lines        3325     3325           
=======================================
  Hits         1743     1743           
  Misses       1353     1353           
  Partials      229      229           

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[jonjohnsonjr]

This feels not good

[imjasonh]

I think we should include detected license information, and I have yet to find a better way to do it than this. Reopening to continue the discussion.

[jonjohnsonjr]

How much memory does it consume to use this?

[imjasonh]

How much memory does it consume to use this?

Good question. The whole csv.gz is embeded here -- it's 3.6 MB zipped. The first time it's requested it's unzipped, read and put into a map here. There isn't a lot of unused space in the csv, it's just package[space]license, which ends up directly in the map.

I tried https://pkg.go.dev/github.com/dghubble/trie#RuneTrie to cut down on that, but it actually used more (or I counted wrong). Using https://github.com/imjasonh/golicenses/compare/trie:

with map:

2023/02/13 14:48:45 loading 422016 records took 170.90425ms allocated 58797992

with trie:

2023/02/13 14:51:10 loading 430039 records took 716.396167ms allocated 829266984

[github-actions]

This Pull Request is stale because it has been open for 90 days with
no activity. It will automatically close after 30 more days of
inactivity. Keep fresh with the 'lifecycle/frozen' label.

[evankanderson]

Hey, I see that we decided not to go this way. Is there still appetite for doing this (possibly in a different way)?