Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: go list -e -m -json all to start displaying hashes of modules #52792

Open
HakanSunay opened this issue May 9, 2022 · 9 comments
Open

Comments

@HakanSunay
Copy link

Proposal source: Gophers Slack

I came to notice that the go tool does not list module hash sums when the list -e -m -json all command is invoked.

$ go mod init hash

$ go get github.com/vmware/govmomi@latest                                                                                                                         ─╯
go: downloading github.com/vmware/govmomi v0.28.0
go: added github.com/vmware/govmomi v0.28.0

$ go list -e -m -json all
{
	"Path": "hash",
	"Main": true,
	"Dir": "/Users/hhalil/test",
	"GoMod": "/Users/hhalil/test/go.mod",
	"GoVersion": "1.18"
}
{
	"Path": "github.com/a8m/tree",
	"Version": "v0.0.0-20210115125333-10a5fd5b637d",
	"Time": "2021-01-15T12:53:33Z",
	"Indirect": true
}
{
	"Path": "github.com/dougm/pretty",
	"Version": "v0.0.0-20171025230240-2ee9d7453c02",
	"Time": "2017-10-25T23:02:40Z",
	"Indirect": true
}
{
	"Path": "github.com/google/uuid",
	"Version": "v1.3.0",
	"Time": "2021-07-12T22:33:52Z",
	"Indirect": true
}
{
	"Path": "github.com/kr/pretty",
	"Version": "v0.3.0",
	"Time": "2020-11-24T22:22:38Z",
	"Indirect": true
}
{
	"Path": "github.com/kr/text",
	"Version": "v0.2.0",
	"Time": "2020-02-14T20:31:06Z",
	"Indirect": true
}
{
	"Path": "github.com/rasky/go-xdr",
	"Version": "v0.0.0-20170217172119-4930550ba2e2",
	"Time": "2017-02-17T17:21:19Z",
	"Indirect": true
}
{
	"Path": "github.com/vmware/govmomi",
	"Version": "v0.28.0",
	"Time": "2022-04-27T15:48:05Z",
	"Indirect": true,
	"Dir": "/Users/hhalil/go/pkg/mod/github.com/vmware/govmomi@v0.28.0",
	"GoMod": "/Users/hhalil/go/pkg/mod/cache/download/github.com/vmware/govmomi/@v/v0.28.0.mod",
	"GoVersion": "1.17"
}
{
	"Path": "github.com/vmware/vmw-guestinfo",
	"Version": "v0.0.0-20170707015358-25eff159a728",
	"Time": "2017-07-07T01:53:58Z",
	"Indirect": true
}

That is not the case with go mod download -json

$ go mod download -json                                                                                                                                         ─╯
{
	"Path": "github.com/vmware/govmomi",
	"Version": "v0.28.0",
	"Info": "/Users/hhalil/go/pkg/mod/cache/download/github.com/vmware/govmomi/@v/v0.28.0.info",
	"GoMod": "/Users/hhalil/go/pkg/mod/cache/download/github.com/vmware/govmomi/@v/v0.28.0.mod",
	"Zip": "/Users/hhalil/go/pkg/mod/cache/download/github.com/vmware/govmomi/@v/v0.28.0.zip",
	"Dir": "/Users/hhalil/go/pkg/mod/github.com/vmware/govmomi@v0.28.0",
	"Sum": "h1:VgeQ/Rvz79U9G8QIKLdgpsN9AndHJL+5iMJLgYIrBGI=",
	"GoModSum": "h1:F7adsVewLNHsW/IIm7ziFURaXDaHEwcc+ym4r3INMdY="
}

The proposal here is to extend the underlying data structures of go list -m to present information such as Sum and GoModSum, just like go mod download -json. By definition, "The -m flag causes list to list modules instead of packages.", therefore I believe it makes sense for the module data to include hash information.

Also, it would be useful if go list -m can also function in workspace mode, meaning that it would present the hashes of the modules after MVS has been applied to all of them, but that is conditional logic on GOWORK.

The target struct to modify might be this:

type ModulePublic struct {
Path string `json:",omitempty"` // module path
Version string `json:",omitempty"` // module version
Versions []string `json:",omitempty"` // available module versions
Replace *ModulePublic `json:",omitempty"` // replaced by this module
Time *time.Time `json:",omitempty"` // time version was created
Update *ModulePublic `json:",omitempty"` // available update (with -u)
Main bool `json:",omitempty"` // is this the main module?
Indirect bool `json:",omitempty"` // module is only indirectly needed by main module
Dir string `json:",omitempty"` // directory holding local copy of files, if any
GoMod string `json:",omitempty"` // path to go.mod file describing module, if any
GoVersion string `json:",omitempty"` // go version used in module
Retracted []string `json:",omitempty"` // retraction information, if any (with -retracted or -u)
Deprecated string `json:",omitempty"` // deprecation message, if any (with -u)
Error *ModuleError `json:",omitempty"` // error loading module
}

... and its implementation of the Stringer interface:

func (m *ModulePublic) String() string {
s := m.Path
versionString := func(mm *ModulePublic) string {
v := mm.Version
if len(mm.Retracted) == 0 {
return v
}
return v + " (retracted)"
}
if m.Version != "" {
s += " " + versionString(m)
if m.Update != nil {
s += " [" + versionString(m.Update) + "]"
}
}
if m.Deprecated != "" {
s += " (deprecated)"
}
if m.Replace != nil {
s += " => " + m.Replace.Path
if m.Replace.Version != "" {
s += " " + versionString(m.Replace)
if m.Replace.Update != nil {
s += " [" + versionString(m.Replace.Update) + "]"
}
}
if m.Replace.Deprecated != "" {
s += " (deprecated)"
}
}
return s
}

@gopherbot gopherbot added this to the Proposal milestone May 9, 2022
@HakanSunay
Copy link
Author

cc: @bcmills

HakanSunay added a commit to HakanSunay/bazel-gazelle that referenced this issue May 9, 2022
With this change gazelle can invoke `update-repos` by processing `go.work` files.
The basis of the change is a plain copy-paste of `update.go` into `work.go`.
However, since `go list` does not show module sums, the whole process is delegated to `go mod download`.
golang/go#52792 was created to track the above.

The standard go tooling in 1.18 can work with `go.work` files so not many changes were actually necessary.
A smoke test is also added to `update_import_test.go` to verify that the latest version of govmomi and rest is always picked.

How to invoke:
$ gazelle update-repos -from_file=go.work -to_macro=deps.bzl%go_deps -prune=True
HakanSunay added a commit to HakanSunay/bazel-gazelle that referenced this issue May 9, 2022
With this change gazelle can invoke `update-repos` by processing `go.work` files.
The basis of the change is a plain copy-paste of `update.go` into `work.go`.
However, since `go list` does not show module sums, the whole process is delegated to `go mod download`.
golang/go#52792 was created to track the above.

The standard go tooling in 1.18 can work with `go.work` files so not many changes were actually necessary.
A smoke test is also added to `update_import_test.go` to verify that the latest version of govmomi and rest is always picked.

How to invoke:
$ gazelle update-repos -from_file=go.work -to_macro=deps.bzl%go_deps -prune=True
HakanSunay added a commit to HakanSunay/bazel-gazelle that referenced this issue May 9, 2022
With this change gazelle can invoke `update-repos` by processing `go.work` files.
The basis of the change is a plain copy-paste of `update.go` into `work.go`.
However, since `go list` does not show module sums, the whole process is delegated to `go mod download`.
golang/go#52792 was created to track the above.

The standard go tooling in 1.18 can work with `go.work` files so not many changes were actually necessary.
A smoke test is also added to `update_import_test.go` to verify that the latest version of govmomi and rest is always picked.

How to invoke:
$ gazelle update-repos -from_file=go.work -to_macro=deps.bzl%go_deps -prune=True
@ianlancetaylor ianlancetaylor added this to Incoming in Proposals (old) May 9, 2022
@rsc
Copy link
Contributor

rsc commented May 11, 2022

GoModSum would at least need to be a []string. When we do a transition to a new algorithm there will be multiple checksums.

@rsc
Copy link
Contributor

rsc commented May 11, 2022

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@rsc rsc moved this from Incoming to Active in Proposals (old) May 11, 2022
@rsc
Copy link
Contributor

rsc commented May 18, 2022

/cc @bcmills @matloob

@rsc
Copy link
Contributor

rsc commented May 31, 2022

Looks like we dropped the ball on go mod download -json about string vs []string. Let's keep GoModSum a single string, and maybe in the future we'll have to add GoModSums listing all of them.

@bcmills
Copy link
Member

bcmills commented May 31, 2022

SGTM. For reference. the go mod download -json struct definition is documented here:
https://pkg.go.dev/cmd/go#hdr-Download_modules_to_local_cache

@bcmills
Copy link
Member

bcmills commented May 31, 2022

So this would be two new fields for the Module definition for go list, paralleling the Module definition for go mod download. The new definition for go list would then be:

type Module struct {
    Path      string       // module path
    Version   string       // module version
    Versions  []string     // available module versions (with -versions)
    Replace   *Module      // replaced by this module
    Time      *time.Time   // time version was created
    Update    *Module      // available update, if any (with -u)
    Main      bool         // is this the main module?
    Indirect  bool         // is this module only an indirect dependency of main module?
    Dir       string       // directory holding files for this module, if any
    GoMod     string       // path to go.mod file used when loading this module, if any
    GoVersion string       // go version used in module
    Retracted string       // retraction information, if any (with -retracted or -u)
    Error     *ModuleError // error loading module
    Sum       string       // checksum for path, version (as in go.sum)
    GoModSum  string       // checksum for go.mod (as in go.sum)
}

(Note that this would leave at least a couple of fields present in go mod download but absent from go list: namely, the Info and Zip fields.)

@rsc
Copy link
Contributor

rsc commented Jun 1, 2022

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

@rsc rsc moved this from Active to Likely Accept in Proposals (old) Jun 1, 2022
@rsc rsc moved this from Likely Accept to Accepted in Proposals (old) Jun 8, 2022
@rsc
Copy link
Contributor

rsc commented Jun 8, 2022

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

@rsc rsc changed the title proposal: cmd/go: go list -e -m -json all to start displaying hashes of modules cmd/go: go list -e -m -json all to start displaying hashes of modules Jun 8, 2022
@rsc rsc modified the milestones: Proposal, Backlog Jun 8, 2022
linzhp pushed a commit to bazelbuild/bazel-gazelle that referenced this issue Jun 12, 2022
With this change gazelle can invoke `update-repos` by processing `go.work` files.
The basis of the change is a plain copy-paste of `update.go` into `work.go`.
However, since `go list` does not show module sums, the whole process is delegated to `go mod download`.
golang/go#52792 was created to track the above.

The standard go tooling in 1.18 can work with `go.work` files so not many changes were actually necessary.
A smoke test is also added to `update_import_test.go` to verify that the latest version of govmomi and rest is always picked.

How to invoke:
$ gazelle update-repos -from_file=go.work -to_macro=deps.bzl%go_deps -prune=True
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Accepted
Development

No branches or pull requests

4 participants