Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: add go mod download to download modules #26610

Closed
sgnn7 opened this issue Jul 25, 2018 · 25 comments

Comments

Projects
None yet
@sgnn7
Copy link

commented Jul 25, 2018

What version of Go are you using (go version)?

go1.11beta2

Does this issue reproduce with the latest release?

Yes (1.11beta2)

What operating system and processor architecture are you using (go env)?

amd64 - linux

What did you do?

Copying just go.mod into an empty directory should allow go mod -vendor (or maybe even go mod -sync) to download the dependencies regardless of having any other files within the directory. Without this, you cannot use something like Docker to cache the dependencies independent of the codebase itself.

$ docker run -it --rm golang:1.11beta2-alpine /bin/sh
/go # apk add -u git
/go # mkdir /foo
/go # cd /foo

/foo # echo "module github.com/sgnn7/test
> 
> require (
>         cloud.google.com/go v0.25.0 // indirect
> )
> " > go.mod

/foo # cat go.mod
module github.com/sgnn7/test

require (
        cloud.google.com/go v0.25.0 // indirect
)

/foo # go mod -vendor
go: finding cloud.google.com/go v0.25.0
go: no dependencies to vendor

/foo # go mod -sync
/foo # # Nothing

What did you expect to see?

go mod should have downloaded go.mod dependencies based on go.mod content regardless if there's code in the repo or not.

What did you see instead?

Neither go mod -vendor nor go mod -sync downloaded anything.

@sgnn7 sgnn7 changed the title cmd/go: `go mod` vendor requires more than just go.mod cmd/go: `go mod -vendor` requires more than just `go.mod` content Jul 25, 2018

@bcmills bcmills added this to the Go1.11 milestone Jul 25, 2018

@bcmills

This comment has been minimized.

Copy link
Member

commented Jul 25, 2018

Without this, you cannot use something like Docker to cache the dependencies independent of the codebase itself.

Can you include some more detail about your use-case? (Why is that a thing you want to do?)

@sgnn7

This comment has been minimized.

Copy link
Author

commented Jul 25, 2018

@bcmills Sure.

Let's say that you have your code build a Docker image on every git change (most pipelines do this and is probably the biggest use case).

  • With dep you could create a Docker layer of cached dependencies by just copying Gopkg.* and fetching the dependencies after which you then copy the full repo and build the project (see here for the original issue for dep on this). This meant that unless you changed the Gopkg files, no dependency re-download was ever needed, making builds with non-dependency changes quick and efficient.
  • With current go mod, you cannot just copy go.mod and go.sum and then run go mod -vendor to get this feature parity since it just doesn't download anything without Go files in the repo, forcing the Docker layer to be cache-busted on the smallest of changes in the repo (e.g. README), causing re-download of all dependencies every time which is both slow and extremely inefficient.
@rsc

This comment has been minimized.

Copy link
Contributor

commented Jul 26, 2018

Vendor is tailored for the dependencies of a specific module,
not warming a cache.

Go modules have an actual cache, separate from the vendor directory.
If you want to warm that cache in a base docker image, that would be:

go list -e $(go list -m all)

That will trigger a download of every module from go.mod
(and their dependencies) into the GOPATH/src/mod cache,
which should make those downloads available to higher
docker layers, all without using the vendor directory.

@rsc rsc closed this Jul 26, 2018

@sgnn7

This comment has been minimized.

Copy link
Author

commented Jul 26, 2018

@rsc Hmm.. I'll give that a try. Is there a reason why this isn't a base go mod command?

@sgnn7

This comment has been minimized.

Copy link
Author

commented Jul 26, 2018

@rsc: Your cache warming suggestion doesn't work out of the box. The fixed version is below:

go list $(go list -m all 2>/dev/null)

Edit: Nope that doesn't work either fully. This should be working though (may need escaping for Dockerfile):

 go list -e $(go list -m all 2>/dev/null | awk '{print $1}')

^ this is really not something that should be found in Dockerfiles

Edit 2 Third iteration that is the least messy so far:

go list -e $(go list -f '{{.Path}}' -m all 2>/dev/null)

IMO, If this seems like something that should be used often, it probably needs to get into the main CLI. As a sidenote to this hack, go list has no distinction in docs as to why it is the only one that can download modules vs -sync and -vendor.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Jul 29, 2018

All the commands downloads modules as needed.
sync and vendor are defined to chase down only the package-granularity dependencies of the packages in the main module.
If there are no packages in the main module, they chase down nothing.
List on the other hand can be asked directly about dependency packages,
and answering requires downloading them to look at the code.
It's certainly not a direct answer, but it should work.

That said, it seems to me that go get -d -m asks for exactly this
operation and fails to deliver it (the -d is ignored with -m).
We should fix that.

@rsc rsc changed the title cmd/go: `go mod -vendor` requires more than just `go.mod` content cmd/go: get -d -m should download modules Jul 29, 2018

@rsc rsc reopened this Jul 29, 2018

@moretea

This comment has been minimized.

Copy link

commented Jul 30, 2018

I got pretty close with #26610 (comment). To make this work for now, I am using the following Dockerfile:

FROM golang as builder
RUN go get -u golang.org/x/vgo
WORKDIR /go/src/github.com/$ORG/$REPO

# Populate the module cache based on the go.{mod,sum} files.
COPY go.mod .
COPY go.sum .
RUN vgo list -e $(vgo list -f '{{.Path}}' -m all)

# Build all binaries, with -getmode=local, this will _not_ fetch packages over the network.
COPY . .
RUN vgo install -getmode=local ./...

# Runtime image
FROM alpine AS base
COPY --from=builder /go/bin/$PROG /bin/$PROG
ENTRYPOINT ["/bin/$PROG"]
@shanna

This comment has been minimized.

Copy link

commented Jul 31, 2018

@moretea This takes care of caching downloads but if I add -v to the go install/build the dependencies are being built each pass still so build/install times can still be very slow if you have a lot of dependencies (firebase, grpc, kubernetes etc).

@sgnn7

This comment has been minimized.

Copy link
Author

commented Jul 31, 2018

Small update for those that are using the go list to fetch the packages: #26631 PR by @marwan-at-work should remove the need to pipe stderr to /dev/null once it gets into the releases.

@marwan-at-work

This comment has been minimized.

Copy link
Contributor

commented Jul 31, 2018

@sgnn7 the CL was closed by @myitcv, not sure if it needs to be re-opened. But my own use case for having that CL is resolved by just using stdout.
Thanks

@sgnn7

This comment has been minimized.

Copy link
Author

commented Jul 31, 2018

@marwan-at-work I don't know why that issue would be abandoned 😢. Maybe it's because the v1.11 GA release date is tomorrow?

As for using stdout, that's what the interim fix uses but it would be nice not to need 2>/dev/null in it to separate stdout from stderr.

Edit: Your PR looked fine to be tbh - I don't really understand the current stdout/stderr separation if stderr shows informational messages.

@myitcv

This comment has been minimized.

Copy link
Member

commented Jul 31, 2018

@marwan-at-work I closed the CL based on our discussion, yes. Did I prematurely close it? Was there something I missed?

@sgnn7 as things stand, stdout is defined to contain the JSON output; stderr informational messages, with the exit code ultimately used to then determine success or otherwise (with the caveat that -e can alter the handling of erroneous packages, per go help list) of the list operation.

Does that help to clarify things?

@marwan-at-work

This comment has been minimized.

Copy link
Contributor

commented Jul 31, 2018

@myitcv CL being closed is good on my side :) Just wanted to ping you about @sgnn7's use case.
Thanks

@sgnn7

This comment has been minimized.

Copy link
Author

commented Jul 31, 2018

@myitcv Makes sense though I wouldn't say that it increases simplicity/usability cases (compared to most other *nix tooling). Either way, this issue is not for that discussion and thank you for the clarification. 👍

@rsc

This comment has been minimized.

Copy link
Contributor

commented Aug 7, 2018

I started looking at go get -d -m but it's a weird combination to redefine.
Instead I am going to add a 'go mod download' that does what you want.
So you could run

go mod download

to warm a Docker image cache.

@rsc rsc changed the title cmd/go: get -d -m should download modules cmd/go: add go mod download to download modules Aug 7, 2018

@marwan-at-work

This comment has been minimized.

Copy link
Contributor

commented Aug 7, 2018

@rsc will this new subcommand take care of this issue? #26577
Most notably, go mod download would not download dependencies except just the targeted import path. Or at least has the option to not download dependencies.

Would love to help in any way.

Thanks!

Edit: it does :)

@gopherbot

This comment has been minimized.

Copy link

commented Aug 7, 2018

Change https://golang.org/cl/128355 mentions this issue: cmd/go: add go mod download

@mattwilliamson

This comment has been minimized.

Copy link

commented Dec 28, 2018

There is a caveat here, you must have a main package in the directory with go.mod for go mod download.

@marwan-at-work

This comment has been minimized.

Copy link
Contributor

commented Dec 28, 2018

@zangbuild no longer the case with go1.12, there's already a beta out if you want to give it a shot.

@mattwilliamson

This comment has been minimized.

Copy link

commented Dec 28, 2018

Nice. I will try when I can.

@mbana

This comment has been minimized.

Copy link

commented Feb 25, 2019

Does go mod download also build (and install) the packages?

Edit: Will this do the what I want? go list -m -f '{{.Path}}' all | xargs -I{} sh -c 'GO111MODULE=on go get -x -a {}'

@marwan-at-work

This comment has been minimized.

Copy link
Contributor

commented Feb 25, 2019

@mbana go mod download just downloads content for a module, it does not build and install the package nor does it fetch its dependencies.

@mcluseau

This comment has been minimized.

Copy link

commented Mar 4, 2019

Hi, related question: is there a go mod download-from-vendor for those already having everything in their vendor directory? Thanks!

@bcmills

This comment has been minimized.

Copy link
Member

commented Mar 5, 2019

@mcluseau, the vendor directory stores individual packages, not complete modules. It is not a suitable input to the module cache.

@mcluseau

This comment has been minimized.

Copy link

commented Mar 18, 2019

So just to follow up, the best way I've found considering that is to use a Go proxy (athens) and the following Dockerfile:

from golang:1.12.1-alpine3.9 as build
run apk add --update git

env CGO_ENABLED=0
arg GOPROXY

workdir /src
add go.mod go.sum ./
run go mod download

add . ./
run go test ./...
run go install . ./cmd/...

from alpine:3.9
#entrypoint ...
copy --from=build /go/bin/ /bin/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.