Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: reconsider the large number of third-party dependencies #29935

Closed
mvdan opened this issue Jan 25, 2019 · 33 comments
Closed

x/build: reconsider the large number of third-party dependencies #29935

mvdan opened this issue Jan 25, 2019 · 33 comments

Comments

@mvdan
Copy link
Member

@mvdan mvdan commented Jan 25, 2019

Today, a coworker and myself noticed how doing a go get -u downloaded a large number of modules which were completely uninteresting to us, mostly from @dmitshur:

go: finding github.com/shurcooL/go-goon latest
go: finding github.com/shurcooL/github_flavored_markdown v0.0.0-20181002035957-2122de532470
go: finding github.com/shurcooL/component latest
go: finding github.com/shurcooL/htmlg v0.0.0-20170918183704-d01228ac9e50
go: finding honnef.co/go/tools latest
go: finding github.com/shurcooL/events latest
go: finding github.com/shurcooL/highlight_diff v0.0.0-20170515013008-09bb4053de1b
go: finding github.com/shurcooL/issues v0.0.0-20181008053335-6292fdc1e191
go: finding github.com/gregjones/httpcache latest
go: finding sourcegraph.com/sqs/pbtypes v0.0.0-20180604144634-d3ebe8f20ae4
go: finding github.com/neelance/sourcemap latest
go: finding github.com/shurcooL/httperror v0.0.0-20170206035902-86b7830d14cc
go: finding github.com/jellevandenhooff/dkim latest
go: finding github.com/shurcooL/htmlg latest
go: finding github.com/shurcooL/highlight_diff latest
go: finding github.com/shurcooL/github_flavored_markdown latest
go: finding golang.org/x/exp v0.0.0-20190121172915-509febef88a4
go: finding github.com/shurcooL/go latest
go: finding github.com/russross/blackfriday v1.5.2
go: finding github.com/shurcooL/issues latest
go: finding sourcegraph.com/sqs/pbtypes v1.0.0
go: finding github.com/shurcooL/httperror latest
go: finding github.com/prometheus/procfs latest

You can reproduce this by running go get -u on https://github.com/xo/usql. What's more worrying, commands like go mod why github.com/shurcooL/httpgzip report that the main module doesn't need all those packages.

After some manual digging, we ended up tracing this to the cloud.google.com/go module: https://github.com/googleapis/google-cloud-go/blob/52299f000fab02ff521d180fd9f101ceb44d0f38/go.mod

And in particular, its requirement on the golang.org/x/build module: https://github.com/golang/build/blob/ad59fb13d31575de611cdf1252fa3fbfd8f07794/go.mod

It's in that last go.mod file where one can see over fifty third-party module requirements, counting // indirect lines.

Now, I would understand that the module which sets up golang.org's CI infrastructure needs to have lots of dependencies. After all, it needs to do lots of work. And that would be fine, as long as extremely few packages required the x/build module.

However, cloud.google.com/go, and by extension many other popular modules like google.golang.org/grpc and google.golang.org/api, require x/build. So any developer wishing to use Google Cloud with Go is forced to download dozens of third-party Go modules for no apparent reason. This can add up quickly even on a warm cache, as go get -u needs to do much more work.

I do understand that, once module mirrors are widespread, we'll only have to download go.mod files for these unnecessary modules, and not clone their entire git repositories. Still, I think it's bad design that cloud.google.com/go is pulling so many module requirements, some of which are exclusively for frontend/JS coding.

I can imagine multiple solutions, besides leaving the situation as it is now:

  • Actively discourage "library" modules from requiring x/build (though cloud.google.com/go imports non-trivial chunks of x/build)
  • Split up the library-useful bits of x/build into a separate module
  • Gradually remove the third party dependencies of x/build, bringing it more in line with other golang.org/x/... repos

I realise I could have raised this issue against the google-cloud-go repository, but I think it's best to open it here first. Developers also tend to trust golang.org/x repos, so I personally think it's reasonable of them to default to trusting x/build.

/cc @bradfitz @dmitshur @andybons @bcmills

@gopherbot gopherbot added this to the Unreleased milestone Jan 25, 2019
@gopherbot gopherbot added the Builders label Jan 25, 2019
@dmitshur
Copy link
Member

@dmitshur dmitshur commented Jan 25, 2019

What's more worrying, commands like go mod why github.com/shurcooL/httpgzip report that the main module doesn't need all those packages.

Every single one of those packages coming from me are needed only by a single command in x/build:

https://godoc.org/golang.org/x/build/maintner/cmd/maintserve

See:

https://godoc.org/golang.org/x/build/maintner/cmd/maintserve?imports

It’s a small program that reuses a lot of my external code to visualize maintner data offline. It’s very dependency heavy.

Before modules, it wasn’t a problem to have it in x/build because you would only fetch its dependencies if you ran go get -u golang.org/x/build/maintner/cmd/maintserve. They wouldn’t be downloaded if you were installing any other libraries or commands in x/build or things that imported a library from x/build.

If it’s causing problems for people in module world, we can consider removing that command from x/build, or making it be a part of a separate module. /cc @bcmills

I’m happy to discuss this.

Some additional observations:

  • I’m surprised that google cloud libraries would be relying on x/build module
  • I don’t understand why dependencies of a single command in x/build module need to be downloaded when building something that only imports libraries from x/build
    Update: I understand this by now
  • it’s unfortunate to hear that go mod why wasn’t more helpful for you to figure out where the httpgzip package is coming from
@bcmills
Copy link
Member

@bcmills bcmills commented Jan 25, 2019

(Responses in no particular order.)

commands like go mod why github.com/shurcooL/httpgzip report that the main module doesn't need all those packages.

go mod why gives you a path through the package-import graph, not through the module graph. (See #27900.)

@bcmills
Copy link
Member

@bcmills bcmills commented Jan 25, 2019

After some manual digging, we ended up tracing this to the cloud.google.com/go module

I note:

~/src/cloud.google.com/go$ go mod why -m golang.org/x/build
# golang.org/x/build
cloud.google.com/go/profiler/proftest
golang.org/x/build/kubernetes

Arguably cloud.google.com/go should use the official Kubernetes client (k8s.io/client-go) instead of the one from golang.org/x/build (which is specifically tailored for use within x/build, and to my knowledge carries no compatibility guarantees).

If the official kubernetes client is too heavy-weight for cloud.google.com, that's probably something they should take up with Kubernetes folks directly.

(CC @jadekler @enocom @jba @zombiezen for cloud.google.com; CC @liggitt for k8s.io/client-go)

@bcmills
Copy link
Member

@bcmills bcmills commented Jan 25, 2019

Note that x/build is almost entirely project-internal infrastructure.

x/tools is the more user-facing “utility junk drawer”, and it does depend on x/build (via golang.org/x/tools/cmd/tip), but we should arguably solve that by carving cmd/tip out of x/tools rather than worrying too much about the dependencies of x/build.

@bcmills
Copy link
Member

@bcmills bcmills commented Jan 25, 2019

So I guess that's all to say, the option I prefer is the first one that @mvdan mentioned:

  • Actively discourage "library" modules from requiring x/build
@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Jan 25, 2019

I'm also fine splitting out useful stuff elsewhere.

@dmitshur
Copy link
Member

@dmitshur dmitshur commented Jan 25, 2019

About x/tools/cmd/tip, we plan to move that out of x/tools (into x/website or x/build) as part of #29206. /cc @cnoellekb

@enocom
Copy link

@enocom enocom commented Jan 25, 2019

/cc @aalexand whose team owns the cloud.google.com/go/profiler package.

@bcmills bcmills added the modules label Jan 25, 2019
@mvdan
Copy link
Member Author

@mvdan mvdan commented Jan 25, 2019

Thanks all for the replies :)

not through the module graph. (See #27900.)

Thanks - that will help dig through these issues in the future.

Note that x/build is almost entirely project-internal infrastructure.

I'm of course fine with this solution, as long as it's enforced properly. In that case, I'm happy for this issue to be forked into x/tools and cloud-go issues, if that would be better.

I don’t understand why dependencies of a single command in x/build module need to be downloaded when building something that only imports libraries from x/build

That's how modules and MVS work; #29832 is somewhat related. In short, go get -u needs to get all the go.mod files in the entire module graph. Perhaps @bcmills can clarify if I said anything wrong.

@mvdan
Copy link
Member Author

@mvdan mvdan commented Jan 25, 2019

when building something that only imports libraries from x/build

A clarification - note that my issue is about running go get -u, not go build or go test. Having to fetch the latest version of all these unnecessary modules is what's causing slowness and pain.

@aalexand
Copy link
Contributor

@aalexand aalexand commented Jan 25, 2019

It should be totally reasonable to get rid of golang.org/x/build/kubernetes dependency in cloud.google.com/go/profiler/proftest, probably replacing it with k8s.io/client-go. In fact, for now that code can just be removed as the profiler GKE tests are broken and that code needs to be updated anyway. So the immediate action is I think to remove the dependency and the code from proftest.go, then figure out how to do it proper.

Another thing to discuss here (and this becomes really cloud-go specific, so not sure how much we want to discuss it here) is that the proftest package is a package that is intended to be used only by tests in profiling agents. Currently it's just use by the Go profiling agent integration test plus by Node.js profiling agent integration test that we own at https://github.com/googleapis/cloud-profiler-nodejs. This package is intended to be useful to us and "invisible" / transparent to the rest of cloud-go users. If it's going to hurt users of cloud-go even with k8s.io/client-go dependency instead of golang.org/x/build/kubernetes, we should re-think the existence of this package.

@nolanmar511 @jqll @kalyanac FYI

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Jan 25, 2019

Can you hide that package behind a build tag?

@aalexand
Copy link
Contributor

@aalexand aalexand commented Jan 25, 2019

@bradfitz If I add a build tag in proftest.go, will a test with a dependency on that package like https://github.com/googleapis/cloud-profiler-nodejs/blob/6190f20fa3cb24456d8d9ac37b0891b39ebb122a/testing/integration_test.go#L30 still build / run as is?

@thepudds
Copy link

@thepudds thepudds commented Jan 25, 2019

go mod tidy looks across all build tags, so build tag alone might be insufficient to address the initial concern raised by @mvdan.

Edit: and sorry, not attempting to comment on all that’s been said here. Just wanted to inject one comment on current behavior.

@bcmills
Copy link
Member

@bcmills bcmills commented Jan 25, 2019

Parameterizing mod tidy on build tags is #25873.

@bcmills
Copy link
Member

@bcmills bcmills commented Jan 25, 2019

This package is intended to be useful to us and "invisible" / transparent to the rest of cloud-go users.

In that case, you could consider splitting it out into its own module.

@jadekler
Copy link
Member

@jadekler jadekler commented Jan 25, 2019

That will happen soon (tm) but we're not ready with tooling for multiple modules in that repository yet, regrettably. By 1.13 we should hopefully be there.

@gopherbot
Copy link

@gopherbot gopherbot commented Feb 13, 2019

Change https://golang.org/cl/162401 mentions this issue: godoc/dl, godoc/proxy, godoc/short, internal/memcache: delete

gopherbot pushed a commit to golang/tools that referenced this issue Feb 20, 2019
These packages existed only to power cmd/godoc for the purpose of
serving the golang.org website. That functionality has moved into
x/website as part of golang/go#29206. x/website has become the
canonical source of golang.org in CL 162157, the golang.org-serving
code was removed from cmd/godoc in CL 162400, and these packages can
be deleted too now.

This removes the last dependency on the cloud.google.com/go module,
which results in a significant reduction of the number of indirect
dependencies in x/tools (this is due to issue golang/go#29935, which
affects the current version of the cloud.google.com/go module).

Run go mod tidy (using Go 1.12 RC 1).

Updates golang/go#29206
Updates golang/go#29981

Change-Id: If07e3ccae8538b3ebd51af64b6af5be5463f4906
Reviewed-on: https://go-review.googlesource.com/c/162401
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Channing Kimble-Brown <channing@golang.org>
Reviewed-by: Michael Matloob <matloob@golang.org>
@dmitshur
Copy link
Member

@dmitshur dmitshur commented Mar 1, 2019

@aalexand @jadekler If I understand correctly, the current plan on your side is to make the cloud.google.com/go module no longer require the golang.org/x/build module, is that right?

If so, it'd be helpful to open an issue in https://github.com/googleapis/google-cloud-go/issues to track that task, so people can subscribe to it for further updates on this topic. I can open the issue if that'd be helpful.

@jadekler
Copy link
Member

@jadekler jadekler commented Mar 5, 2019

@dmitshur Absolutely. Please track in googleapis/google-cloud-go#1344

@gopherbot
Copy link

@gopherbot gopherbot commented Mar 13, 2019

Change https://golang.org/cl/167461 mentions this issue: maintner/cmd/maintserve: carve out into a separate module

gopherbot pushed a commit to golang/build that referenced this issue Mar 13, 2019
This change creates a new module with the module path
golang.org/x/build/maintner/cmd/maintserve, and moves the maintserve
command into it. This is done by adding a go.mod file in the
maintner/cmd/maintserve directory. As a result, the maintserve command
and its dependencies are removed from this and later pseudo-versions of
the golang.org/x/build module.

This is similar to how the h2demo command was carved out into a separate
module in CL 162822.

The motivation for doing this is that the maintserve command requires a
large number of dependencies for its own use, that no other package in
x/build needs. Even though authors of library modules should not be
requiring x/build, it still happens (see golang/go#29935). So, improve
the situation by moving maintserve into a module separate from x/build,
which in turn significantly reduces the dependencies in x/build.

Updates golang/go#29935

Change-Id: Ic64da32be142fedf6475a11fea787cdfe245e0ce
Reviewed-on: https://go-review.googlesource.com/c/build/+/167461
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Mar 13, 2019

Change https://golang.org/cl/167463 mentions this issue: maintner/cmd/maintserve: require golang.org/x/build@latest

gopherbot pushed a commit to golang/build that referenced this issue Mar 14, 2019
Now that a pseudo-version of golang.org/x/build module with maintserve
carved out exists, the module golang.org/x/build/maintner/cmd/maintserve
can depend on it. Do so now.

This is analogous to CL 166857.

Updates golang/go#29935

Change-Id: I661f32336ede26fde5581527959d31d164a00b63
Reviewed-on: https://go-review.googlesource.com/c/build/+/167463
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Mar 28, 2019

Change https://golang.org/cl/170001 mentions this issue: [release-branch.go1.12] godoc/dl, godoc/proxy, godoc/short, internal/memcache: delete

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 16, 2019

Change https://golang.org/cl/170863 mentions this issue: cmd/gopherbot: CC triaged issues to owners

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Apr 16, 2019

On a fresh machine with Go master (cb66462), I tried to go get a single binary out of x/build.

I don't have GOPROXY set. It took a long time downloading things and was a much worse experience than GOPATH mode:

ci20@ci20:~$ ./go/bin/go get -v golang.org/x/build/cmd/buildlet
Fetching https://golang.org/x/build/cmd/buildlet?go-get=1
Parsing meta tags from https://golang.org/x/build/cmd/buildlet?go-get=1 (status code 200)
get "golang.org/x/build/cmd/buildlet": found meta tag get.metaImport{Prefix:"golang.org/x/build", VCS:"git", RepoRoot:"https://go.googlesource.com/build"} at https://golang.org/x/build/cmd/buildlet?go-get=1
get "golang.org/x/build/cmd/buildlet": verifying non-authoritative meta tag
Fetching https://golang.org/x/build?go-get=1
Parsing meta tags from https://golang.org/x/build?go-get=1 (status code 200)
go: finding golang.org/x/build/cmd/buildlet latest
Fetching https://golang.org/x/build/cmd?go-get=1
Parsing meta tags from https://golang.org/x/build/cmd?go-get=1 (status code 200)
get "golang.org/x/build/cmd": found meta tag get.metaImport{Prefix:"golang.org/x/build", VCS:"git", RepoRoot:"https://go.googlesource.com/build"} at https://golang.org/x/build/cmd?go-get=1
get "golang.org/x/build/cmd": verifying non-authoritative meta tag
go: finding golang.org/x/build/cmd latest
Fetching https://golang.org/x/build?go-get=1
Parsing meta tags from https://golang.org/x/build?go-get=1 (status code 200)
get "golang.org/x/build": found meta tag get.metaImport{Prefix:"golang.org/x/build", VCS:"git", RepoRoot:"https://go.googlesource.com/build"} at https://golang.org/x/build?go-get=1
go: finding golang.org/x/build latest
go: downloading golang.org/x/build v0.0.0-20190416153556-d37bb9a5eaf1
go: extracting golang.org/x/build v0.0.0-20190416153556-d37bb9a5eaf1
Fetching https://go4.org?go-get=1
Fetching https://google.golang.org/appengine?go-get=1
Fetching https://golang.org/x/net?go-get=1
Fetching https://cloud.google.com/go?go-get=1
Fetching https://golang.org/x/sys?go-get=1
Fetching https://golang.org/x/time?go-get=1
Fetching https://grpc.go4.org?go-get=1
Parsing meta tags from https://golang.org/x/net?go-get=1 (status code 200)
get "golang.org/x/net": found meta tag get.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net?go-get=1
Parsing meta tags from https://golang.org/x/time?go-get=1 (status code 200)
get "golang.org/x/time": found meta tag get.metaImport{Prefix:"golang.org/x/time", VCS:"git", RepoRoot:"https://go.googlesource.com/time"} at https://golang.org/x/time?go-get=1
Parsing meta tags from https://golang.org/x/sys?go-get=1 (status code 200)
get "golang.org/x/sys": found meta tag get.metaImport{Prefix:"golang.org/x/sys", VCS:"git", RepoRoot:"https://go.googlesource.com/sys"} at https://golang.org/x/sys?go-get=1
go: finding github.com/davecgh/go-spew v1.1.1
go: finding github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568
go: finding github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7
go: finding golang.org/x/net v0.0.0-20181029044818-c44066c5c816
go: finding golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2
go: finding golang.org/x/sys v0.0.0-20181029174526-d69651ed3497
Parsing meta tags from https://google.golang.org/appengine?go-get=1 (status code 200)
get "google.golang.org/appengine": found meta tag get.metaImport{Prefix:"google.golang.org/appengine", VCS:"git", RepoRoot:"https://github.com/golang/appengine"} at https://google.golang.org/appengine?go-get=1
go: finding google.golang.org/appengine v1.2.0
Parsing meta tags from https://cloud.google.com/go?go-get=1 (status code 200)
get "cloud.google.com/go": found meta tag get.metaImport{Prefix:"cloud.google.com/go", VCS:"git", RepoRoot:"https://code.googlesource.com/gocloud"} at https://cloud.google.com/go?go-get=1
Parsing meta tags from https://go4.org?go-get=1 (status code 200)
get "go4.org": found meta tag get.metaImport{Prefix:"go4.org", VCS:"git", RepoRoot:"https://github.com/go4org/go4"} at https://go4.org?go-get=1
go: finding cloud.google.com/go v0.31.0
Parsing meta tags from https://grpc.go4.org?go-get=1 (status code 200)
get "grpc.go4.org": found meta tag get.metaImport{Prefix:"grpc.go4.org", VCS:"git", RepoRoot:"https://github.com/go4org/grpc"} at https://grpc.go4.org?go-get=1
go: finding go4.org v0.0.0-20180809161055-417644f6feb5
go: finding grpc.go4.org v0.0.0-20170609214715-11d0a25b4919
go: finding github.com/google/martian v2.1.0+incompatible
go: finding github.com/googleapis/gax-go v2.0.0+incompatible
go: finding github.com/google/go-github v17.0.0+incompatible
Fetching https://golang.org/x/oauth2?go-get=1
Parsing meta tags from https://golang.org/x/oauth2?go-get=1 (status code 200)
get "golang.org/x/oauth2": found meta tag get.metaImport{Prefix:"golang.org/x/oauth2", VCS:"git", RepoRoot:"https://go.googlesource.com/oauth2"} at https://golang.org/x/oauth2?go-get=1
go: finding golang.org/x/oauth2 v0.0.0-20181017192945-9dcd33a902f4
Fetching https://go.opencensus.io?go-get=1
go: finding github.com/google/go-cmp v0.2.0
Parsing meta tags from https://go.opencensus.io?go-get=1 (status code 200)
get "go.opencensus.io": found meta tag get.metaImport{Prefix:"go.opencensus.io", VCS:"git", RepoRoot:"https://github.com/census-instrumentation/opencensus-go"} at https://go.opencensus.io?go-get=1
go: finding go.opencensus.io v0.18.0
go: finding github.com/golang/protobuf v1.2.0
Fetching https://golang.org/x/perf?go-get=1
Parsing meta tags from https://golang.org/x/perf?go-get=1 (status code 200)
get "golang.org/x/perf": found meta tag get.metaImport{Prefix:"golang.org/x/perf", VCS:"git", RepoRoot:"https://go.googlesource.com/perf"} at https://golang.org/x/perf?go-get=1
go: finding golang.org/x/perf v0.0.0-20180704124530-6e6d33e29852
go: finding golang.org/x/net v0.0.0-20180724234803-3673e40ba225
Fetching https://gopkg.in/inf.v0?go-get=1
Parsing meta tags from https://gopkg.in/inf.v0?go-get=1 (status code 200)
get "gopkg.in/inf.v0": found meta tag get.metaImport{Prefix:"gopkg.in/inf.v0", VCS:"git", RepoRoot:"https://gopkg.in/inf.v0"} at https://gopkg.in/inf.v0?go-get=1
go: finding gopkg.in/inf.v0 v0.9.1
Fetching https://golang.org/x/text?go-get=1
Parsing meta tags from https://golang.org/x/text?go-get=1 (status code 200)
get "golang.org/x/text": found meta tag get.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text?go-get=1
go: finding golang.org/x/text v0.3.0
Fetching https://golang.org/x/crypto?go-get=1
Parsing meta tags from https://golang.org/x/crypto?go-get=1 (status code 200)
get "golang.org/x/crypto": found meta tag get.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto?go-get=1
go: finding golang.org/x/crypto v0.0.0-20181030102418-4d3f4d9ffa16
Fetching https://google.golang.org/genproto?go-get=1
Parsing meta tags from https://google.golang.org/genproto?go-get=1 (status code 200)
get "google.golang.org/genproto": found meta tag get.metaImport{Prefix:"google.golang.org/genproto", VCS:"git", RepoRoot:"https://github.com/google/go-genproto"} at https://google.golang.org/genproto?go-get=1
go: finding google.golang.org/genproto v0.0.0-20180831171423-11092d34479b
Fetching https://google.golang.org/api?go-get=1
Parsing meta tags from https://google.golang.org/api?go-get=1 (status code 200)
get "google.golang.org/api": found meta tag get.metaImport{Prefix:"google.golang.org/api", VCS:"git", RepoRoot:"https://code.googlesource.com/google-api-go-client"} at https://google.golang.org/api?go-get=1
go: finding google.golang.org/api v0.0.0-20181030000543-1d582fd0359e
go: finding google.golang.org/api v0.0.0-20180910000450-7ca32eb868bf
Fetching https://gopkg.in/yaml.v2?go-get=1
Parsing meta tags from https://gopkg.in/yaml.v2?go-get=1 (status code 200)
get "gopkg.in/yaml.v2": found meta tag get.metaImport{Prefix:"gopkg.in/yaml.v2", VCS:"git", RepoRoot:"https://gopkg.in/yaml.v2"} at https://gopkg.in/yaml.v2?go-get=1
go: finding gopkg.in/yaml.v2 v2.2.1
go: finding github.com/google/go-querystring v1.0.0
go: finding github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910
Fetching https://gopkg.in/check.v1?go-get=1
Parsing meta tags from https://gopkg.in/check.v1?go-get=1 (status code 200)
get "gopkg.in/check.v1": found meta tag get.metaImport{Prefix:"gopkg.in/check.v1", VCS:"git", RepoRoot:"https://gopkg.in/check.v1"} at https://gopkg.in/check.v1?go-get=1
go: finding google.golang.org/genproto v0.0.0-20181029155118-b69ba1387ce2
go: finding golang.org/x/net v0.0.0-20180906233101-161cd47e91fd
go: finding gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405
go: finding github.com/grpc-ecosystem/grpc-gateway v1.5.0
Fetching https://golang.org/x/sync?go-get=1
Parsing meta tags from https://golang.org/x/sync?go-get=1 (status code 200)
get "golang.org/x/sync": found meta tag get.metaImport{Prefix:"golang.org/x/sync", VCS:"git", RepoRoot:"https://go.googlesource.com/sync"} at https://golang.org/x/sync?go-get=1
go: finding github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b
go: finding golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f
go: finding github.com/openzipkin/zipkin-go v0.1.1
go: finding golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e
go: finding github.com/prometheus/procfs v0.0.0-20180725123919-05ee40e3a273
go: finding github.com/ghodss/yaml v1.0.0
go: finding github.com/bradfitz/go-smtpd v0.0.0-20170404230938-deb6d6237625
go: finding github.com/prometheus/client_golang v0.8.0
go: finding github.com/prometheus/common v0.0.0-20180801064454-c7de2306084e
go: finding github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239
Fetching https://google.golang.org/grpc?go-get=1
Parsing meta tags from https://google.golang.org/grpc?go-get=1 (status code 200)
get "google.golang.org/grpc": found meta tag get.metaImport{Prefix:"google.golang.org/grpc", VCS:"git", RepoRoot:"https://github.com/grpc/grpc-go"} at https://google.golang.org/grpc?go-get=1
go: finding git.apache.org/thrift.git v0.0.0-20180902110319-2566ecd5d999
go: finding google.golang.org/grpc v1.14.0
go: finding github.com/jellevandenhooff/dkim v0.0.0-20150330215556-f50fe3d243e1
go: finding golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2
go: finding github.com/gliderlabs/ssh v0.1.1
go: finding github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973
go: finding github.com/matttproud/golang_protobuf_extensions v1.0.1
go: finding github.com/kr/pty v1.1.3
go: finding github.com/tarm/serial v0.0.0-20180830185346-98f6abe2eb07
go: finding github.com/coreos/go-systemd v0.0.0-20181012123002-c6f51f82210d
go: downloading cloud.google.com/go v0.31.0
go: downloading google.golang.org/api v0.0.0-20181030000543-1d582fd0359e
go: downloading golang.org/x/oauth2 v0.0.0-20181017192945-9dcd33a902f4
go: downloading golang.org/x/net v0.0.0-20181029044818-c44066c5c816
go: downloading gopkg.in/inf.v0 v0.9.1
go: extracting gopkg.in/inf.v0 v0.9.1
go: extracting golang.org/x/oauth2 v0.0.0-20181017192945-9dcd33a902f4
go: extracting golang.org/x/net v0.0.0-20181029044818-c44066c5c816
go: extracting cloud.google.com/go v0.31.0
go: extracting google.golang.org/api v0.0.0-20181030000543-1d582fd0359e
golang.org/x/net/context
golang.org/x/build
golang.org/x/net/context/ctxhttp
golang.org/x/oauth2/jws
cloud.google.com/go/compute/metadata
golang.org/x/oauth2/internal
golang.org/x/oauth2
google.golang.org/api/googleapi/internal/uritemplates
google.golang.org/api/googleapi
golang.org/x/oauth2/jwt
golang.org/x/oauth2/google
google.golang.org/api/gensupport
golang.org/x/build/maintner/maintnerd/maintapi/version
golang.org/x/build/types
gopkg.in/inf.v0
google.golang.org/api/compute/v1
google.golang.org/api/oauth2/v2
golang.org/x/build/kubernetes/api
golang.org/x/build/kubernetes
golang.org/x/build/internal/httpdl
golang.org/x/build/pargzip
golang.org/x/build/revdial
golang.org/x/build/buildenv
golang.org/x/build/dashboard
golang.org/x/build/buildlet
golang.org/x/build/cmd/buildlet
ci20@ci20:~$ 

The actual deps are:

	"Deps": [
		"archive/tar",
		"bufio",
		"bytes",
		"cloud.google.com/go/compute/metadata",
		"compress/flate",
		"compress/gzip",
		"container/list",
		"context",
		"crypto",
		"crypto/aes",
		"crypto/cipher",
		"crypto/des",
		"crypto/dsa",
		"crypto/ecdsa",
		"crypto/elliptic",
		"crypto/hmac",
		"crypto/internal/randutil",
		"crypto/internal/subtle",
		"crypto/md5",
		"crypto/rand",
		"crypto/rc4",
		"crypto/rsa",
		"crypto/sha1",
		"crypto/sha256",
		"crypto/sha512",
		"crypto/subtle",
		"crypto/tls",
		"crypto/x509",
		"crypto/x509/pkix",
		"encoding",
		"encoding/asn1",
		"encoding/base64",
		"encoding/binary",
		"encoding/hex",
		"encoding/json",
		"encoding/pem",
		"errors",
		"flag",
		"fmt",
		"golang.org/x/build",
		"golang.org/x/build/buildenv",
		"golang.org/x/build/buildlet",
		"golang.org/x/build/dashboard",
		"golang.org/x/build/internal/httpdl",
		"golang.org/x/build/kubernetes",
		"golang.org/x/build/kubernetes/api",
		"golang.org/x/build/maintner/maintnerd/maintapi/version",
		"golang.org/x/build/pargzip",
		"golang.org/x/build/revdial",
		"golang.org/x/build/types",
		"golang.org/x/net/context",
		"golang.org/x/net/context/ctxhttp",
		"golang.org/x/oauth2",
		"golang.org/x/oauth2/google",
		"golang.org/x/oauth2/internal",
		"golang.org/x/oauth2/jws",
		"golang.org/x/oauth2/jwt",
		"google.golang.org/api/compute/v1",
		"google.golang.org/api/gensupport",
		"google.golang.org/api/googleapi",
		"google.golang.org/api/googleapi/internal/uritemplates",
		"google.golang.org/api/oauth2/v2",
		"gopkg.in/inf.v0",
		"hash",
		"hash/crc32",
		"internal/bytealg",
		"internal/cpu",
		"internal/errinternal",
		"internal/fmtsort",
		"internal/nettrace",
		"internal/oserror",
		"internal/poll",
		"internal/race",
		"internal/reflectlite",
		"internal/singleflight",
		"internal/syscall/unix",
		"internal/testlog",
		"io",
		"io/ioutil",
		"log",
		"math",
		"math/big",
		"math/bits",
		"math/rand",
		"mime",
		"mime/multipart",
		"mime/quotedprintable",
		"net",
		"net/http",
		"net/http/httptrace",
		"net/http/httputil",
		"net/http/internal",
		"net/textproto",
		"net/url",
		"os",
		"os/exec",
		"os/signal",
		"os/user",
		"path",
		"path/filepath",
		"reflect",
		"regexp",
		"regexp/syntax",
		"runtime",
		"runtime/cgo",
		"runtime/internal/atomic",
		"runtime/internal/math",
		"runtime/internal/sys",
		"sort",
		"strconv",
		"strings",
		"sync",
		"sync/atomic",
		"syscall",
		"time",
		"unicode",
		"unicode/utf16",
		"unicode/utf8",
		"unsafe",
		"vendor/golang.org/x/crypto/chacha20poly1305",
		"vendor/golang.org/x/crypto/cryptobyte",
		"vendor/golang.org/x/crypto/cryptobyte/asn1",
		"vendor/golang.org/x/crypto/curve25519",
		"vendor/golang.org/x/crypto/hkdf",
		"vendor/golang.org/x/crypto/internal/chacha20",
		"vendor/golang.org/x/crypto/internal/subtle",
		"vendor/golang.org/x/crypto/poly1305",
		"vendor/golang.org/x/net/dns/dnsmessage",
		"vendor/golang.org/x/net/http/httpguts",
		"vendor/golang.org/x/net/http/httpproxy",
		"vendor/golang.org/x/net/http2/hpack",
		"vendor/golang.org/x/net/idna",
		"vendor/golang.org/x/text/secure/bidirule",
		"vendor/golang.org/x/text/transform",
		"vendor/golang.org/x/text/unicode/bidi",
		"vendor/golang.org/x/text/unicode/norm"
	],

But note above that it was downloading go-smtpd, yaml, gopkg.in/check.v1`, and all sorts of stuff not required by this binary.

So I'm in favor of more sub-modules in x/build, as cmd/go apparently can't do any better.

@thepudds
Copy link

@thepudds thepudds commented Apr 16, 2019

Most often, setting GOPROXY can help significantly, including because it allows downloading a go.mod file without needing to fetch a repo, and my understanding is source code is not downloaded with a GOPROXY if it is not needed for the package-level import graph. In addition, the HTTPS requests to a GOPROXY are generally more efficient than using git as the transport in the absence of a GOPROXY.

It could be interesting to check if you see better behavior if you do something like export GOPROXY='https://gocenter.io' or perhaps point towards any Athens instance you might be running.

A couple sample datapoints:

https://github.com/sagikazarmark/go-module-dependency-test

johanbrandhorst/certify#67 (comment)

@thepudds
Copy link

@thepudds thepudds commented Apr 17, 2019

@bradfitz Just to give some rough numbers, I ran a quick test of go get golang.org/x/build/cmd/buildlet with and without setting GOPROXY='https://gocenter.io':

  • With GOPROXY='https://gocenter.io':
    • 14 sec., 136 MB in GOPATH/pkg/mod
  • Without GOPROXY set:
    • 58 sec., 381 MB in GOPATH/pkg/mod

YMMV. This was with a clean GOPATH each time (and hence no module cache), but not a clean machine.

You had cited gopkg.in/check.v1 as an example concern. If you inspect the module cache afterwards, you can see that when GOPROXY was not set, the code for gopkg.in/check.v1 ends up in the module cache (because the go command needed to fetch all the code via git to find its go.mod). On the other hand, when GOPROXY was set, the module cache has the go.mod but not the code for gopkg.in/check.v1.

Sample test steps:

export GOPROXY='https://gocenter.io'
export GOPATH=$(mktemp -d)
cd $(mktemp -d)
time gotip get golang.org/x/build/cmd/buildlet
fkorotkov added a commit to cirruslabs/cirrus-ci-docs that referenced this issue Apr 22, 2019
Should speed up downloads. See golang/go#29935 (comment) for details
fkorotkov added a commit to cirruslabs/cirrus-ci-docs that referenced this issue Apr 22, 2019
Should speed up downloads. See golang/go#29935 (comment) for details
@dmitshur
Copy link
Member

@dmitshur dmitshur commented Jun 4, 2019

Is there anything else we want to do in x/build specifically? I'm not seeing more to be done, and I think the original issue is resolved by now.

The overall resolution seems to be:

  • many maintserve-only dependencies were removed from x/build module and moved into a nested module in CL 167461
  • the remaining module requirements in x/build seem reasonable, nothing jumps out as unexpected
  • library authors should not depending on x/build module because it's large and exists primarily for the needs of the Go build infrastructure
  • if there are some small and targeted Go packages in x/build that are desirable for external library consumption, they can be factored out if we're willing to support them; individual issues can be filed for that
@bcmills
Copy link
Member

@bcmills bcmills commented Jun 5, 2019

nothing jumps out as unexpected

I dunno, it at least seems weird that we have direct dependencies on two different go-shlex modules.

@dmitshur
Copy link
Member

@dmitshur dmitshur commented Jun 5, 2019

Both go-shlex module requirements I see are indirect:

github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239 // indirect
github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568 // indirect

What makes you say they're direct dependencies?

We are directly importing and requiring github.com/gliderlabs/ssh v0.1.1, which in turn directly imports and uses github.com/anmitsu/go-shlex. anmitsu/go-shlex has an example that imports flynn/go-shlex:

https://github.com/anmitsu/go-shlex/blob/648efa622239a2f6ff949fed78ee37b48d499ba4/example_test.go#L8

I'm not seeing a problem here that we need to fix.

@thepudds
Copy link

@thepudds thepudds commented Sep 13, 2019

Wanted to briefly bounce this. Is there anything left to do here?

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Sep 13, 2019

What do you mean by "bounce this"? Close it? Work on it? "Ping"?

@dmitshur
Copy link
Member

@dmitshur dmitshur commented Sep 13, 2019

My interpretation of "bounce" is the same meaning as "ping".

Is there anything left to do here?

In #29935 (comment), I concluded that the original issue has been resolved and that I don't see anything actionable left to do. Bryan brought up one concern, and I addressed it. No one has brought anything up since then, and it's been 3 months.

I think this is done, so I'll close. Otherwise feel free to speak up or open a new issue. Thanks to everyone who helped with this.

@dmitshur dmitshur closed this Sep 13, 2019
Bogdan-D pushed a commit to Bogdan-D/godocx that referenced this issue Nov 13, 2019
These packages existed only to power cmd/godoc for the purpose of
serving the golang.org website. That functionality has moved into
x/website as part of golang/go#29206. x/website has become the
canonical source of golang.org in CL 162157, the golang.org-serving
code was removed from cmd/godoc in CL 162400, and these packages can
be deleted too now.

This removes the last dependency on the cloud.google.com/go module,
which results in a significant reduction of the number of indirect
dependencies in x/tools (this is due to issue golang/go#29935, which
affects the current version of the cloud.google.com/go module).

Run go mod tidy (using Go 1.12 RC 1).

Updates golang/go#29206
Updates golang/go#29981

Change-Id: If07e3ccae8538b3ebd51af64b6af5be5463f4906
Reviewed-on: https://go-review.googlesource.com/c/162401
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Channing Kimble-Brown <channing@golang.org>
Reviewed-by: Michael Matloob <matloob@golang.org>
@golang golang locked and limited conversation to collaborators Sep 13, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
9 participants
You can’t perform that action at this time.