Remove or replace licenses.txt

[silverwind]

Feature Description

Given the recent problems with the go-licenses tool, I would propose removing or replacing the licenses.txt file. It serves no clear purpose and is to my knowledge not processed by any tools. So the only "consumers" are humans and I doubt the usefulness.

If our intend with this file is to just list dependencies, I think we would be better off generating a SBOM file with tools like https://github.com/aquasecurity/trivy and from a quick check, this is a excellent and well-maintained tool for such a purpose.

For example to create a SBOM file with trivy, try:

trivy fs --format cyclonedx --output result.json .

The other alternative is to remove the file without any replacement.

[silverwind]

I think it would be quite nice if we run trivy before the webpack build and then just add the generated asset into the webpack build. That way, there will be no files in the repo to worry about.

[silverwind]

Another thing that trivy can do is generate a license report:

$ trivy fs --scanners license --format json --license-full .

generates entries like this:

{
  "Severity": "LOW",
  "Category": "notice",
  "PkgName": "github.com/alecthomas/chroma/v2",
  "FilePath": "go.mod",
  "Name": "MIT",
  "Text": "",
  "Confidence": 1,
  "Link": ""
},

It does not contain actual license text, but I think it should be easy to convert this into a text format to put into licenses.txt.

[wxiaoguang]

Now the assets/go-licenses.json is 1.2MB. If we could only store the short name (eg: MIT) for popular licenses, we could hugely reduce the size.

[TheFox0x7]

Unfortunately I don't think short names only are feasible if the license requires to preserve it/the copyright. Though a link to a license might be a solution.

There's FOSSA that could automate this but API access is required for that part of cli tool.

[silverwind]

A cross-language tool like trivy is definitely the way to go. It simplifies the process a lot. It not including license text or links seems like a bug. Maybe that will improve later.

Maybe there are better tools available, I have not searched too much yet. At minimum, it should include the ability to extract licenses from the go and js dependencies.

[techknowlogick]

At the very least we could try to compress the licenses so that are included but don't take up much space.

[silverwind]

At the very least we could try to compress the licenses so that are included but don't take up much space.

All statically embedded files are already gzip-compressed by vfsgen, this file is no exception.

[techknowlogick]

@silverwind ah, fantastic. Thanks for letting me know:)

[silverwind]

I think we can probably just write the output of trivy fs --quiet --security-checks license --format spdx . into licenses.txt. This is a parseable format that does not include the license text verbatim, but imho that is also not needed. This command is fast enough that we can run it at build time, it currently takes 2.7s on my machine.