mockgen stopped working after upgrading to Golang 1.13

[veqryn]

We run gomock/mockgen inside of a docker container. Previously that docker container was based on golang 1.12, but we upgrade it to 1.13 today.
After upgrading to golang 1.13, mockgen started failing out:

mockgen --package=mock_seeds --destination=/go/src/github.com/xxx/yyy/mock_seeds/mock_seeds.go github.com/xxx/yyy/seeds ExternalAPIClient,ExternalAPIServer                                        
go: github.com/xxx/zzz@v0.0.0-20190917161345-2ad29436c755: invalid version: git fetch -f https://github.com/xxx/zzz refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /go/pkg/mod/cache/vcs/595ef178a 519bba79a7580920dc80e819bacb31e39f9842a1de283c683f92cd1: exit status 128:
     fatal: could not read Username for 'https://github.com': terminal prompts disabled

Our project has several golang modules it depends on that are private, with the obfuscated https://github.com/xxx/zzz being one of them. We use go mod vendor to commit them to the repo.

Why is mockgen running go get or git fetch or anything like that?

mockgen shouldn't be getting my source files, when they are all right there already.

What is going on, and how to solve it?

[codyoss]

I will look into this, thanks for the report

[henry-jackson]

@veqryn , did your team ever find a solution to this, or did you just stay on Go 1.12? I am experiencing the same issue you are describing

This commit may be related: 577071b

[veqryn]

Stayed on golang 1.12 for now

[minicuts]

@veqryn @henry-jackson I think this is caused by the fact that 1.13 by default enables module aware behavior Under the covers, mockgen runs go build. And since you are invoking the mockgen in a directory with go.mod the executed go build command starts resolving the module dependencies. When I try go build with 1.12, it does take vendor directory into an account by default.

As a workaround, can you add to your mockgen command flag -build_flags="-mod=vendor" ?

In general, I think there should be at least better error message provided and more output so that the user user understand what mockgen does under the covers.

May I also ask a bit more details about your use case? When do you run go mod vendor ? Would it make any sense for you to run mockgen right after you run go mod vendor and "git push" the mocks into your private repo?

[veqryn]

I believe i tried using that flag and had some problems getting it to work. I can try again this weekend and report the results.

We run go mod vendor manually on our laptops whenever we update our dependencies, which is not often.

We run mockgen whenever we update any of our many interfaces, including the ones in generated files (from protobuf definitions). This happens all the time, which is why we created a docker container image that runs both protobuf generate, go generate, and mockgen (and we do commit the results from all of that, and push them up manually later as part of normal dev workflow). It would Not make sense for us to run mockgen separately, outside of the container.

[minicuts]

@veqryn Ok makes sense, thanks for explaining.

The workaround also assumes that you run the mockgen from the directory where the vendor folder and go.mod are located.

[veqryn]

That might be why it didn't work. Our go.mod and vendor directory are both in the root folder of the project, while mockgen is being run by golang files with go generate directives in them, spread deep across the project. If I remember correctly, commands run via go generate treat the directory containing the golang file as the current directory.

[minicuts]

If you run the mockgen in the context of a module (in any subdirectory of the root directory) it should work.

[minicuts]

@veqryn any update?

[veqryn]

Sorry, holidays got in the way. Will try soon

[codyoss]

@veqryn still having issues?

[veqryn]

@nguyenfilip @codyoss
Adding --build_flags="--mod=vendor" works when run manually from my terminal. But it doesn't work with go generate for some reason.

Examples:

Here is the go generate line in my file:

// Generate mocks
//go:generate mockgen --build_flags="--mod=vendor" --package=mock_seeds --destination=${GOPATH}/src/github.com/ReturnPath/seeds/go/proto/seeds/mock_seeds/mock_seeds.go github.com/ReturnPath/seeds/go/proto/seeds ExternalAPIClient,ExternalAPIServer,InternalAPIClient,InternalAPIServer,InternalAPI_MailserviceAccountsGetClient,InternalAPI_MailserviceAccountsGetServer

When run:

root@be0862e1b1c3:/go/src/github.com/ReturnPath/seeds/proto# go generate -x --mod=vendor .
mockgen --build_flags="--mod=vendor" --package=mock_seeds --destination=/go/src/github.com/ReturnPath/seeds/go/proto/seeds/mock_seeds/mock_seeds.go github.com/ReturnPath/seeds/go/proto/seeds ExternalAPIClient,ExternalAPIServer,InternalAPIClient,InternalAPIServer,Internal
API_MailserviceAccountsGetClient,InternalAPI_MailserviceAccountsGetServer
can't load package: named files must be .go files: "--mod=vendor"
can't load package: named files must be .go files: "--mod=vendor"
2020/01/24 12:49:36 Loading input failed: exit status 1
gogenerate.go:18: running "mockgen": exit status 1

If I run go generate without the mod build flag, it doesn't even get to mockgen before it fails:

root@be0862e1b1c3:/go/src/github.com/ReturnPath/seeds/proto# go generate -x .
<whole bunch of crap>
go: github.com/ReturnPath/ecm@v0.0.0-20191227100444-512b5f597fe0: invalid version: git fetch -f https://github.com/ReturnPath/ecm refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /go/pkg/mod/cache/vcs/595ef178a519bba79a7580920dc80e819bacb31e39f9842a1de283c683f92cd1:
exit status 128:
        fatal: could not read Username for 'https://github.com': terminal prompts disabled

If I remove the build flags from the mockgen line, here is the file:

// Generate mocks
//go:generate mockgen --package=mock_seeds --destination=${GOPATH}/src/github.com/ReturnPath/seeds/go/proto/seeds/mock_seeds/mock_seeds.go github.com/ReturnPath/seeds/go/proto/seeds ExternalAPIClient,ExternalAPIServer,InternalAPIClient,InternalAPIServer,InternalAPI_MailserviceAccountsGetClient,InternalAPI_MailserviceAccountsGetServer

If I run go generate on that, with the build flag, I get:

root@be0862e1b1c3:/go/src/github.com/ReturnPath/seeds/proto# go generate -x --mod=vendor .
mockgen --package=mock_seeds --destination=/go/src/github.com/ReturnPath/seeds/go/proto/seeds/mock_seeds/mock_seeds.go github.com/ReturnPath/seeds/go/proto/seeds ExternalAPIClient,ExternalAPIServer,InternalAPIClient,InternalAPIServer,InternalAPI_MailserviceAccountsGetCli
ent,InternalAPI_MailserviceAccountsGetServer
go: github.com/ReturnPath/ecm@v0.0.0-20191227100444-512b5f597fe0: invalid version: git fetch -f https://github.com/ReturnPath/ecm refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /go/pkg/mod/cache/vcs/595ef178a519bba79a7580920dc80e819bacb31e39f9842a1de283c683f92cd1:
exit status 128:
        fatal: could not read Username for 'https://github.com': terminal prompts disabled
build command-line-arguments: cannot load github.com/ReturnPath/seeds/go/proto/seeds: git ls-remote -q https://github.com/ReturnPath/seeds in /go/pkg/mod/cache/vcs/85f75baab50262f48948c10018a83a105529db42d8b3d91d442e9efd9ecf8e59: exit status 128:
        fatal: could not read Username for 'https://github.com': terminal prompts disabled
Confirm the import path was entered correctly.
If this is a private repository, see https://golang.org/doc/faq#git_https for additional information.
2020/01/24 12:52:21 Loading input failed: exit status 1
gogenerate.go:18: running "mockgen": exit status 1

Here is my go env for reference, when running inside of a golang:1.13.6-buster container image:

root@be0862e1b1c3:/go/src/github.com/ReturnPath/seeds/proto# go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/go/src/github.com/ReturnPath/seeds/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build174822346=/tmp/go-build -gno-record-gcc-switches"

And my go.mod file, located at /go/src/github.com/ReturnPath/seeds/go.mod

module github.com/ReturnPath/seeds

go 1.13

require (
	github.com/Masterminds/squirrel v1.1.0
	github.com/ReturnPath/ecm v0.0.0-20191227100444-512b5f597fe0
)

[veqryn]

I also tried with single quotes, such as: --build_flags='--mod=vendor'
This does not work, and gives the same error when run with go generate: can't load package: named files must be .go files: '--mod=vendor'

[minicuts]

@veqryn I think that the correct combination of flags for you is to have --mod=vendor in BOTH:

• your .go files as directives. This will make sure that the mockgen uses the vendor directory when generating mocks
• your go generate command. This will make sure that go generate will use vendor for dependencies

I think that the root cause of your problem is the same as for issue #354. In a nutshell the problem is that in Docker (because of effective root privileges) the mockgen doesn't get executed in "current directory" but it is executed first in module cache. I hope to fix that bug with #390

I will try to reproduce the combination of features (vendoring + go generate + docker) locally. I assume it is not possible that you provide a live reproducer?

[veqryn]

Since it requires a private repo, that is vendored, it is probably not possible for me to provide a live reproducer.

What do you mean by having --mod=vendor in your .go files as directives ?
Do you mean like my example above, with it in the go:generate line?

[minicuts]

@veqryn I have prepared a sample that seems to work (see below). I think that the problem in your case is that your go:generate directive has quotes around the --mod=vendor and also maybe you are using old mockgen. Try the following with mockgen 1.4.1:

// Generate mocks
//go:generate mockgen --build_flags=--mod=vendor --package=mock_seeds --destination=${GOPATH}/src/github.com/ReturnPath/seeds/go/proto/seeds/mock_seeds/mock_seeds.go github.com/ReturnPath/seeds/go/proto/seeds ExternalAPIClient,ExternalAPIServer,InternalAPIClient,InternalAPIServer,InternalAPI_MailserviceAccountsGetClient,InternalAPI_MailserviceAccountsGetServer

The sample
With the latest gomock (1.4.1) I think I have been able to accomplish what you want, see https://github.com/nguyenfilip/mockgen-docker. In that github repo you can see that:

• module secret.com/secretlib gets vendored. It contains interface SecretInterface.
• in file main.go in the module company.com/main there is a go:generate directive that uses the vendored module

To run the sample, just clone the repo and run the commands that are mentioned in the readme file.

[veqryn]

It looks like removing the quotes fixed the problem.

I'll do some more testing, and will re-open the issue if any problems come up.

Thank you!

[veqryn]

A few questions:

1. Why would quotes break go generate + mockgen?
2. What if I had multiple build flags (and therefore need quotes)?
3. Should any documentation get updated based on this?

[henry-jackson]

I resolved this issue for myself a couple weeks ago after realizing I somehow had 2 different versions of mockgen installed. I ran which mockgen and then removed that file. Running which mockgen again showed that another executable file was in my PATH, installed to a different directory. I removed that one as well. Finally reinstalled mockgen according to the README instructions and the tool started working for me as intended on go 1.13

Might be worth trying for anyone else that is still running into similar issues

[minicuts]

@henry-jackson the mockgen now has a -version flag that you can use to verify running versin

[minicuts]

@veqryn good questions

1. its the behavior of go generate. go:generate directive splits the arguments by space or quotes but doesn't behave like shell (see design doc for go generate). The problem with that is, that when you pass in --build_flags="-mod=vendor" then mockgen flag build_flags value is "-mod=vendor" (including the quotes!) This doesn't happen when you invoke mockgen from shell, because quotes are expanded in shell and mockgen sees only -mod=vendor.
2. You can put whole --build_flags arg to quotes. That will cause go generate to pass it as one flag to mockgen. For example:

//go:generate mockgen "--build_flags=-mod=vendor arg2 arg3" --destination mock.go secret.com/secretlib SecretInterface

3. Hm. I am not sure where would be the best place. My instinct says we could emit a warning to stderr in case a build_flags flag value starts and ends with a quote. WDYT? That should save more peoples time than burying it somewhere in wiki.

[veqryn]

I think an error/warning message would be great.
thank you