New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: add go mod download to download modules #26610

Closed
sgnn7 opened this Issue Jul 25, 2018 · 17 comments

Comments

Projects
None yet
9 participants
@sgnn7

sgnn7 commented Jul 25, 2018

What version of Go are you using (go version)?

go1.11beta2

Does this issue reproduce with the latest release?

Yes (1.11beta2)

What operating system and processor architecture are you using (go env)?

amd64 - linux

What did you do?

Copying just go.mod into an empty directory should allow go mod -vendor (or maybe even go mod -sync) to download the dependencies regardless of having any other files within the directory. Without this, you cannot use something like Docker to cache the dependencies independent of the codebase itself.

$ docker run -it --rm golang:1.11beta2-alpine /bin/sh
/go # apk add -u git
/go # mkdir /foo
/go # cd /foo

/foo # echo "module github.com/sgnn7/test
> 
> require (
>         cloud.google.com/go v0.25.0 // indirect
> )
> " > go.mod

/foo # cat go.mod
module github.com/sgnn7/test

require (
        cloud.google.com/go v0.25.0 // indirect
)

/foo # go mod -vendor
go: finding cloud.google.com/go v0.25.0
go: no dependencies to vendor

/foo # go mod -sync
/foo # # Nothing

What did you expect to see?

go mod should have downloaded go.mod dependencies based on go.mod content regardless if there's code in the repo or not.

What did you see instead?

Neither go mod -vendor nor go mod -sync downloaded anything.

@sgnn7 sgnn7 changed the title from cmd/go: `go mod` vendor requires more than just go.mod to cmd/go: `go mod -vendor` requires more than just `go.mod` content Jul 25, 2018

@bcmills bcmills added this to the Go1.11 milestone Jul 25, 2018

@bcmills

This comment has been minimized.

Member

bcmills commented Jul 25, 2018

Without this, you cannot use something like Docker to cache the dependencies independent of the codebase itself.

Can you include some more detail about your use-case? (Why is that a thing you want to do?)

@sgnn7

This comment has been minimized.

sgnn7 commented Jul 25, 2018

@bcmills Sure.

Let's say that you have your code build a Docker image on every git change (most pipelines do this and is probably the biggest use case).

  • With dep you could create a Docker layer of cached dependencies by just copying Gopkg.* and fetching the dependencies after which you then copy the full repo and build the project (see here for the original issue for dep on this). This meant that unless you changed the Gopkg files, no dependency re-download was ever needed, making builds with non-dependency changes quick and efficient.
  • With current go mod, you cannot just copy go.mod and go.sum and then run go mod -vendor to get this feature parity since it just doesn't download anything without Go files in the repo, forcing the Docker layer to be cache-busted on the smallest of changes in the repo (e.g. README), causing re-download of all dependencies every time which is both slow and extremely inefficient.
@rsc

This comment has been minimized.

Contributor

rsc commented Jul 26, 2018

Vendor is tailored for the dependencies of a specific module,
not warming a cache.

Go modules have an actual cache, separate from the vendor directory.
If you want to warm that cache in a base docker image, that would be:

go list -e $(go list -m all)

That will trigger a download of every module from go.mod
(and their dependencies) into the GOPATH/src/mod cache,
which should make those downloads available to higher
docker layers, all without using the vendor directory.

@rsc rsc closed this Jul 26, 2018

@sgnn7

This comment has been minimized.

sgnn7 commented Jul 26, 2018

@rsc Hmm.. I'll give that a try. Is there a reason why this isn't a base go mod command?

@sgnn7

This comment has been minimized.

sgnn7 commented Jul 26, 2018

@rsc: Your cache warming suggestion doesn't work out of the box. The fixed version is below:

go list $(go list -m all 2>/dev/null)

Edit: Nope that doesn't work either fully. This should be working though (may need escaping for Dockerfile):

 go list -e $(go list -m all 2>/dev/null | awk '{print $1}')

^ this is really not something that should be found in Dockerfiles

Edit 2 Third iteration that is the least messy so far:

go list -e $(go list -f '{{.Path}}' -m all 2>/dev/null)

IMO, If this seems like something that should be used often, it probably needs to get into the main CLI. As a sidenote to this hack, go list has no distinction in docs as to why it is the only one that can download modules vs -sync and -vendor.

@rsc

This comment has been minimized.

Contributor

rsc commented Jul 29, 2018

All the commands downloads modules as needed.
sync and vendor are defined to chase down only the package-granularity dependencies of the packages in the main module.
If there are no packages in the main module, they chase down nothing.
List on the other hand can be asked directly about dependency packages,
and answering requires downloading them to look at the code.
It's certainly not a direct answer, but it should work.

That said, it seems to me that go get -d -m asks for exactly this
operation and fails to deliver it (the -d is ignored with -m).
We should fix that.

@rsc rsc changed the title from cmd/go: `go mod -vendor` requires more than just `go.mod` content to cmd/go: get -d -m should download modules Jul 29, 2018

@rsc rsc reopened this Jul 29, 2018

@moretea

This comment has been minimized.

moretea commented Jul 30, 2018

I got pretty close with #26610 (comment). To make this work for now, I am using the following Dockerfile:

FROM golang as builder
RUN go get -u golang.org/x/vgo
WORKDIR /go/src/github.com/$ORG/$REPO

# Populate the module cache based on the go.{mod,sum} files.
COPY go.mod .
COPY go.sum .
RUN vgo list -e $(vgo list -f '{{.Path}}' -m all)

# Build all binaries, with -getmode=local, this will _not_ fetch packages over the network.
COPY . .
RUN vgo install -getmode=local ./...

# Runtime image
FROM alpine AS base
COPY --from=builder /go/bin/$PROG /bin/$PROG
ENTRYPOINT ["/bin/$PROG"]
@shanna

This comment has been minimized.

shanna commented Jul 31, 2018

@moretea This takes care of caching downloads but if I add -v to the go install/build the dependencies are being built each pass still so build/install times can still be very slow if you have a lot of dependencies (firebase, grpc, kubernetes etc).

@sgnn7

This comment has been minimized.

sgnn7 commented Jul 31, 2018

Small update for those that are using the go list to fetch the packages: #26631 PR by @marwan-at-work should remove the need to pipe stderr to /dev/null once it gets into the releases.

@marwan-at-work

This comment has been minimized.

Contributor

marwan-at-work commented Jul 31, 2018

@sgnn7 the CL was closed by @myitcv, not sure if it needs to be re-opened. But my own use case for having that CL is resolved by just using stdout.
Thanks

@sgnn7

This comment has been minimized.

sgnn7 commented Jul 31, 2018

@marwan-at-work I don't know why that issue would be abandoned 😢. Maybe it's because the v1.11 GA release date is tomorrow?

As for using stdout, that's what the interim fix uses but it would be nice not to need 2>/dev/null in it to separate stdout from stderr.

Edit: Your PR looked fine to be tbh - I don't really understand the current stdout/stderr separation if stderr shows informational messages.

@myitcv

This comment has been minimized.

Member

myitcv commented Jul 31, 2018

@marwan-at-work I closed the CL based on our discussion, yes. Did I prematurely close it? Was there something I missed?

@sgnn7 as things stand, stdout is defined to contain the JSON output; stderr informational messages, with the exit code ultimately used to then determine success or otherwise (with the caveat that -e can alter the handling of erroneous packages, per go help list) of the list operation.

Does that help to clarify things?

@marwan-at-work

This comment has been minimized.

Contributor

marwan-at-work commented Jul 31, 2018

@myitcv CL being closed is good on my side :) Just wanted to ping you about @sgnn7's use case.
Thanks

@sgnn7

This comment has been minimized.

sgnn7 commented Jul 31, 2018

@myitcv Makes sense though I wouldn't say that it increases simplicity/usability cases (compared to most other *nix tooling). Either way, this issue is not for that discussion and thank you for the clarification. 👍

@rsc

This comment has been minimized.

Contributor

rsc commented Aug 7, 2018

I started looking at go get -d -m but it's a weird combination to redefine.
Instead I am going to add a 'go mod download' that does what you want.
So you could run

go mod download

to warm a Docker image cache.

@rsc rsc changed the title from cmd/go: get -d -m should download modules to cmd/go: add go mod download to download modules Aug 7, 2018

@marwan-at-work

This comment has been minimized.

Contributor

marwan-at-work commented Aug 7, 2018

@rsc will this new subcommand take care of this issue? #26577
Most notably, go mod download would not download dependencies except just the targeted import path. Or at least has the option to not download dependencies.

Would love to help in any way.

Thanks!

Edit: it does :)

@gopherbot

This comment has been minimized.

gopherbot commented Aug 7, 2018

Change https://golang.org/cl/128355 mentions this issue: cmd/go: add go mod download

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment