Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: avoid work to compute unused fields in 'go list' #29666

Open
hyangah opened this issue Jan 10, 2019 · 12 comments
Open

cmd/go: avoid work to compute unused fields in 'go list' #29666

hyangah opened this issue Jan 10, 2019 · 12 comments
Assignees
Labels
GoCommand modules NeedsFix ToolSpeed
Milestone

Comments

@hyangah
Copy link
Contributor

@hyangah hyangah commented Jan 10, 2019

We want to get the list of available versions of a module
and the best way is to use go list -m --versions <module>.

Under the hood, the command retrieves the tag list (which we expected)
and also fetches some versions (which is surprising).

Our guess is that the go list does so in order to find information
like the latest version's commit time, etc. For our use case, we don't need
that information and the extra fetch slows down the go list especially
when the repository is large.

Can we have a flag to skip the fetch?

An alternative mentioned by @bcmills is to make go list smarter so it
performs steps necessary to fill in the requested information (e.g. fields
mentioned in -f, or with an additional option to specify fields to prune if
-json is requested)

$ GO111MODULE=on tipgo list -x -m -f '{{.Versions}}' --versions rsc.io/quote

mkdir -p $GOPATH/pkg/mod/cache/vcs # git2 https://github.com/rsc/quote
# lock $GOPATH/pkg/mod/cache/vcs/$HASH.lockmkdir -p $GOPATH/pkg/mod/cache/vcs/$HASH # git2 https://github.com/rsc/quote
cd $GOPATH/pkg/mod/cache/vcs/$HASH; git init --bare
0.006s # cd $GOPATH/pkg/mod/cache/vcs/$HASH; git init --bare
cd $GOPATH/pkg/mod/cache/vcs/$HASH; git remote add origin https://github.com/rsc/quote 
0.004s # cd $GOPATH/pkg/mod/cache/vcs/$HASH; git remote add origin https://github.com/rsc/quote
cd $GOPATH/pkg/mod/cache/vcs/$HASH; git ls-remote -q origin
0.298s # cd $GOPATH/pkg/mod/cache/vcs/$HASH; git ls-remote -q origin
cd $GOPATH/pkg/mod/cache/vcs/$HASH; git cat-file --batch
0.004s # cd $GOPATH/pkg/mod/cache/vcs/$HASH; git cat-file --batch

cd $GOPATH/pkg/mod/cache/vcs/$HASH; git fetch -f origin refs/tags/v2.0.0:refs/tags/v2.0.0 refs/tags/v2.0.1:refs/tags/v2.0.1 refs/tags/v3.0.0:refs/tags/v3.0.0
0.365s # cd $GOPATH/pkg/mod/cache/vcs/$HASH; git fetch -f origin refs/tags/v2.0.0:refs/tags/v2.0.0 refs/tags/v2.0.1:refs/tags/v2.0.1 refs/tags/v3.0.0:refs/tags/v3.0.0

cd $GOPATH/pkg/mod/cache/vcs/$HASH; git cat-file --batch
0.004s # cd $GOPATH/pkg/mod/cache/vcs/$HASH; git cat-file --batch
go: finding rsc.io/quote v1.5.2
cd $GOPATH/pkg/mod/cache/vcs/$HASH; git tag -l
0.003s # cd $GOPATH/pkg/mod/cache/vcs/$HASH; git tag -l
cd $GOPATH/pkg/mod/cache/vcs/$HASH; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' refs/tags/v1.5.2
0.004s # cd $GOPATH/pkg/mod/cache/vcs/$HASH; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' refs/tags/v1.5.2

[v1.0.0 v1.1.0 v1.2.0 v1.2.1 v1.3.0 v1.4.0 v1.5.0 v1.5.1 v1.5.2 v1.5.3-pre1]

@bcmills @heschik @katiehockman @jayconrod

@heschi
Copy link
Contributor

@heschi heschi commented Jan 10, 2019

From what I can piece together, Hana's example above is particularly poor, but probably unfixable. It's looking for a v1 module, but it needs to download every non-v1 version too to make sure that they all have a go.mod. (If one didn't, then it'd be +incompatible and a valid answer for the query.)

Previously we had been looking at the way it behaves when fetching rsc.io/quote/v2, or a v1 module from a repo that doesn't have v2 yet. That can probably be improved.

@jayconrod jayconrod added NeedsDecision GoCommand modules labels Jan 10, 2019
@jayconrod jayconrod added this to the Go1.13 milestone Jan 10, 2019
@bcmills
Copy link
Member

@bcmills bcmills commented Jan 11, 2019

I would rather do the field-based pruning than add a separate flag. Field-based pruning gives us the opportunity to eliminate a lot of different kinds of redundant work, whereas a “skip timestamp fetching” flag would have a very narrow benefit.

@bcmills
Copy link
Member

@bcmills bcmills commented Jan 11, 2019

One nice option for the JSON field-pruning might be to allow the -json flag to accept a comma-separated list of fields with an explicit = connector:

go list -m -json=Path,Versions,Replace.Path --versions rsc.io/quote

@bcmills
Copy link
Member

@bcmills bcmills commented Jan 11, 2019

It's looking for a v1 module, but it needs to download every non-v1 version too to make sure that they all have a go.mod.

It occurs to me that this could interact with caching in an interesting way: if you're only looking for new versions and you know when you last checked the repo, then it's probably safe to assume that any tag that was +incompatible still is, and any tag that was a module still is. So perhaps some sort of flag to restrict the result to “tags added since timestamp T” would avoid the need for that fetch too.

@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Jan 11, 2019

I really like the idea of -json accepting a list of fields that are actually needed. I think we should do that on the regular go list (not -m) side as well.

@bcmills bcmills changed the title cmd/go: allow 'go list -m --versions' to skip vcs-fetching cmd/go: avoid work to compute unused fields in 'go list' Jan 17, 2019
@bcmills bcmills added ToolSpeed NeedsFix and removed NeedsDecision labels Jan 17, 2019
@bcmills
Copy link
Member

@bcmills bcmills commented Jan 17, 2019

Retitled to address the general problem: we won't be adding a specific flag to avoid VCS fetches, but we certainly can optimize the work that we do based on the requested output.

@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Nov 18, 2020

In CL 270858, we noticed that go list -m all loads .info files for each module in the build list in order to find the version's timestamp. That's printed with -json and maybe -f. If neither of those flags are specified, then .info files aren't used: go build and go get don't need them. We should skip loading those files if we don't need their content.

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 10, 2021

Change https://golang.org/cl/309190 mentions this issue: cmd/go/internal/modload: avoid loading the module graph to list only the name of the main module

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 10, 2021

Change https://golang.org/cl/309191 mentions this issue: cmd/go/internal/modload: avoid loading the full module graph when listing specific modules

gopherbot pushed a commit that referenced this issue Apr 30, 2021
…the name of the main module

For #36460
For #29666

Change-Id: I9e46f7054d52c053be80c483757cdd34b22822d9
Reviewed-on: https://go-review.googlesource.com/c/go/+/309190
Trust: Bryan C. Mills <bcmills@google.com>
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
Reviewed-by: Michael Matloob <matloob@golang.org>
gopherbot pushed a commit that referenced this issue Apr 30, 2021
…ting specific modules

For #36460
For #41297
Updates #29666

Change-Id: I5f324c0ef9a164f8043d2188101d141bb5fa7454
Reviewed-on: https://go-review.googlesource.com/c/go/+/309191
Trust: Bryan C. Mills <bcmills@google.com>
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
Reviewed-by: Michael Matloob <matloob@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Mar 2, 2022

Change https://go.dev/cl/381035 mentions this issue: cmd/go: allow users to specify required fields in JSON output

gopherbot pushed a commit that referenced this issue Mar 2, 2022
For #29666

Change-Id: Ibae3d75bb2c19571c8d473cb47d6c4b3a880bba8
Reviewed-on: https://go-review.googlesource.com/c/go/+/381035
Trust: Michael Matloob <matloob@golang.org>
Run-TryBot: Michael Matloob <matloob@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Mar 14, 2022

Change https://go.dev/cl/392495 mentions this issue: cmd/go: don't compute Deps fields if it's not needed

gopherbot pushed a commit that referenced this issue Apr 20, 2022
If the user provides the -json flag to explicitly specify fields, but
doesn't specify the Deps or DepsErrors fields, skip computing the deps
fields.

For #29666

Change-Id: I15596c374aba1af13bdf5808d11d54abdc838667
Reviewed-on: https://go-review.googlesource.com/c/go/+/392495
Reviewed-by: Bryan Mills <bcmills@google.com>
Run-TryBot: Michael Matloob <matloob@golang.org>
Auto-Submit: Michael Matloob <matloob@golang.org>
Reviewed-by: Michael Matloob <matloob@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Apr 27, 2022

Change https://go.dev/cl/402736 mentions this issue: cmd/go: skip computing BuildInfo in go list unless it's needed

gopherbot pushed a commit that referenced this issue May 4, 2022
The only fields of the go list output that require BuildInfo to be
computed are the Stale and StaleReason fields. If a user explicitly
requests JSON fields and does not ask for Stale or StaleReason, skip
the computation of BuildInfo.

For #29666

Change-Id: Ie77581c44babedcb5cb7f3dc7d6ed1078b56eee4
Reviewed-on: https://go-review.googlesource.com/c/go/+/402736
Run-TryBot: Michael Matloob <matloob@golang.org>
Reviewed-by: Michael Matloob <matloob@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Michael Matloob <matloob@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
GoCommand modules NeedsFix ToolSpeed
Projects
None yet
Development

No branches or pull requests

8 participants