go2purl: Go package to Go PURL

[pombredanne]

Create a PURL service that takes a Go package as an input and uses Go heuristics and the Go proxy service to determine the corresponding PURL (and its companion git PURL when relevant and available)

For instance in https://github.com/istio/istio/blob/master/istioctl/pkg/authz/analyzer.go

we have these imports:

• "github.com/envoyproxy/go-control-plane/envoy/config/listener/v3" with PURL: pkg:golang/github.com/envoyproxy/go-control-plane/envoy/config/listener/v3@1.2.3

• "istio.io/istio/istioctl/pkg/util/configdump" with PURL: pkg:golang/istio.io/istio/istioctl/pkg/util/configdump@abcef2323232abcef2323232abcef2323232

Actually, after further review, Go uses multiple styles to reference packages/modules and their versions:

Packages

• A) in Go code, we can import "packages" as in https://github.com/moby/moby/blob/6c10086976d07d4746e03dcfd188972a2f07e1c9/registry/auth.go ... there is no version there and this requires parsing the Go source code. This is not in the scope here

A package is a directory with a bunch of go files, and is further declared in the code of this package itself with "package foo" directives.

We cannot infer a PURL from a package only: we are missing the version and we do not know where the path or name of the modules ends.

Modules

A module is a collection of packages with a go.mod file at the root.

• C) in go.mod we have "module nameminimum version" as in https://github.com/moby/moby/blob/6c10086976d07d4746e03dcfd188972a2f07e1c9/vendor.mod#L125 . This requires parsing the mod file as done elsewhere, like in the ScanCode toolkit.

• D) in go.sum we have with an extra checksum as in https://github.com/moby/moby/blob/6c10086976d07d4746e03dcfd188972a2f07e1c9/vendor.sum . This requires parsing the sum file as done elsewhere, like in the ScanCode toolkit.

• E) when running the go command line we can run go get github.com/foo/foo-package@31c913b to fetch a specific version of a module/package in the workspace. The version after the @ sign is the go module version, and the name before is the module/package path (github.com/foo/foo-package). We cannot infer the module unless we query the go proxy.

Modules can have a PURL and have a version (at least we know either the pinned or minimum version from the mod or sum file).

A) and E) are not in scope here, because we cannot reliably infer a module from a package short of doing extra calls. This is best done elsewhere, for instance in fetchcode.

B) and C) are in scope and the input is that of a go.mod for now. Dealing with checksums is something different that should be handled elsewhere possibly in Scancode like in https://github.com/nexB/scancode-toolkit/blob/66d71661f5ede54cb0f3b36d7663c92a67030299/src/packagedcode/go_mod.py#L206

[TG1999]

Done in package-url/packageurl-python#142

[pombredanne]

This is available as an API function rather than a REST API endpoint as this is designed to be used as a function and not as an API call

[pombredanne]

Based on #246 (comment) we would have a new to_purl/go API endpoint that accepts a Go import

[TG1999]

This is done. We now have:

• an endpoint /api/to_purl/go to get a package-url from a go package string that can be an import or a go package string in go.mod file like this "package version".

These 2 PRs have been merged in packageurl-python

• Add utility to get golang purl from go package package-url/packageurl-python#142
• Add support for go packages in manifest files package-url/packageurl-python#148

This PR has been merged

• Add golang purl service #316

To test this feature:

• Install PurlDB from the main branch
• Make a request to /api/to_purl/go?go_package={package} where package is a valid package import string or go package string in go.mod file

[pombredanne]

For info, I had also these notes:

• We created the core functions to back such a service. It takes a Go package PURL as an input and uses Go heuristics and the Go proxy service to determine the corresponding PURLs. These functions are in FetchCode
  See https://github.com/nexB/fetchcode/blob/d0a3fa9bb56dc3a77f7d3d7bd5b8d0e40c7a8612/src/fetchcode/package_versions.py#L293 and https://github.com/nexB/fetchcode/blob/d0a3fa9bb56dc3a77f7d3d7bd5b8d0e40c7a8612/src/fetchcode/package_versions.py#L326

• We also added support in the purl library to the same effect Add utility to get golang purl from go package package-url/packageurl-python#142