x/pkgsite: releasing license detection code as library or API

[imjasonh]

I'd like to be able to determine the license(s) for a Go module, ideally the same one displayed on e.g. https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/s3, and using the same implementation.

It seems the Go team has a method for detecting this license, based on https://pkg.go.dev/github.com/google/licensecheck, which "aims never to give a false positive", which sounds great.

I did some digging and found that the code pkgsite uses to call licensecheck is internal, here: https://cs.opensource.google/go/x/pkgsite/+/master:internal/licenses/licenses.go -- there seems to be some useful code there, e.g., lists of common license file paths, what licenses are redistributable, etc.

If that code was available somewhere as non-internal, I'd happily use it. I understand that comes with a non-zero ongoing support burden, so if you say you want to keep it internal I totally understand. I always have the ability to fork that into my own internal package, pay the support burden to maintain it there, etc.

If releasing the code is not acceptable, another option might be to make this detected license available as an API, perhaps even in proxy.golang.org:

https://proxy.golang.org/github.com/aws/aws-sdk-go-v2/service/s3/@v/v1.26.6.info

If that was updated to include, for example:

{"Version":"v1.26.6","Time":"2022-04-25T18:11:03Z","License":"Apache-2.0"}

(structure/naming TBD obviously; the API could also be hosted separately on pkg.go.dev if that's simpler)

If that existed, then that would solve my problem very nicely, and not incur the support burden of un-internalizing that code.

I'm sure there's no shortage of complexity involved here, even just in getting license information available in the pkgsite UI. I guess my point is, once the hard work of getting that information in the UI is done, making it available via API "somewhere" doesn't seem that much harder. Feel free to educate me and prove me wrong though 😄

cc @jonjohnsonjr @wlynch @jba

[jba]

Ideally this would be part of a pkgsite API, if that were to happen.

Unfortunately making the code available but internal is the best we can do right now. We can't take on the support burden, and we don't want to commit to that package's API. It's fine for pkgsite, but probably not the best general one.

It's also important to understand that the decision about whether a license or variant thereof is redistributable is informed by legal considerations, and pkgsite's decisions are informed by Google's lawyers. We effectively expose those decisions in the UI, by showing you the docs for some packages and not others. But it seems wrong to provide a package to the community that embeds those decisions. Your legal team (if any) is not Google's. (In contrast, the licensecheck package makes no legal decisions; it just matches license text against established templates.)

Although the decision of where to look for license files is less controversial, it's still somewhat arbitrary—we had to draw the line somewhere. I suppose we could parameterize the API by that decision, but that gets back to my point about finding the right API.

So I'm sorry, but it's fork or scrape.

[imjasonh]

Yeah I kinda figured that'd be the answer. In any case, thanks for taking the time to consider it and respond!

I don't plan to make any policy decisions based on the detected license, so licensecheck is 90% of the way there for me. The last 10% was just in reusing pkgsite's license file detection decisions. That's not too bad to fork, if I decide to keep going this way.

Thanks again for your response, and for helping maintain pkgsite in general 👍