dep ensure -add command fails due to constraints in terratest

[gregwhorley]

I am trying to grab dependencies for a test I wrote that uses the terratest library by using dep. The project I'm working in has no other Go code, so I ran dep init from the root of the project to get things started. After dep created Gopkg* files and vendor directory, I checked out the branch that has my test and then tried to run dep ensure for all of the modules in my import list minus the golang built-ins:

import (
	"fmt"
	"path/filepath"
	"strings"
	"testing"
	"github.com/stretchr/testify/require"
	"github.com/gruntwork-io/terratest/modules/helm"
	"github.com/gruntwork-io/terratest/modules/k8s"
	"github.com/gruntwork-io/terratest/modules/random"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

Any attempt at retrieving dependencies via dep ensure -add on any of those modules results in various constraint errors, such as:

$ dep ensure -add github.com/stretchr/testify/require
Fetching sources...

Solving failure: No versions of golang.org/x/crypto met constraints:
	master: unable to update checked out version: fatal: reference is not a tree: 38d8ce5564a5b71b2e3a00553993f1b9a7ae852f
: command failed: [git checkout 38d8ce5564a5b71b2e3a00553993f1b9a7ae852f]: exit status 128
	release-branch.go1.11: Could not introduce golang.org/x/crypto@release-branch.go1.11, as it is not allowed by constraint master from project github.com/gruntwork-io/terratest.
	release-branch.go1.12: Could not introduce golang.org/x/crypto@release-branch.go1.12, as it is not allowed by constraint master from project github.com/gruntwork-io/terratest.

If I try to run dep ensure -add on one of the terratest modules, I get a ton of error output about mismatched constraints. I'll provide a gist of the full error output if needed.

$ go version
go version go1.12 linux/amd64

$ dep version
dep:
 version     : v0.5.1
 build date  : 2019-03-11
 git hash    : faa6189
 go version  : go1.12
 go compiler : gc
 platform    : linux/amd64
 features    : ImportDuringSolve=false

What am I missing?

[yorinasub17]

Hi @gwhorleyGH,

Hmm, I can't seem to repro the issue. Can you share your Gopkg.toml file?

That said, I have been running into a few issues starting a project from scratch so this might be related.

For reference, here are the steps I took to try to repro the issue:

• Create a new dir in GOPATH and cd to it
• Create a new file foo.go that contains those imports + test code from https://github.com/gruntwork-io/terratest-helm-testing-example so that the compiler doesn't complain.
• run dep init
• run dep ensure -add github.com/stretchr/testify/require

My versions are:
go version go1.12.1 darwin/amd64

dep:
 version     : v0.5.1
 build date  : 2019-03-11
 git hash    : faa6189
 go version  : go1.12
 go compiler : gc
 platform    : darwin/amd64
 features    : ImportDuringSolve=false

[yorinasub17]

By the way, you shouldn't need to run ensure -add if you ran dep init.

[gregwhorley]

My Gopkg.toml file hasn't been edited from its default state, maybe that is the problem? Also, the Gopkg files and vendor directory are located at the base directory of the project while my test is located in k8s/tests. Does that matter?

# Gopkg.toml example
#
# Refer to https://golang.github.io/dep/docs/Gopkg.toml.html
# for detailed Gopkg.toml documentation.
#
# required = ["github.com/user/thing/cmd/thing"]
# ignored = ["github.com/user/project/pkgX", "bitbucket.org/user/project/pkgA/pkgY"]
#
# [[constraint]]
#   name = "github.com/user/project"
#   version = "1.0.0"
#
# [[constraint]]
#   name = "github.com/user/project2"
#   branch = "dev"
#   source = "github.com/myfork/project2"
#
# [[override]]
#   name = "github.com/x/y"
#   version = "2.4.0"
#
# [prune]
#   non-go = false
#   go-tests = true
#   unused-packages = true


[prune]
  go-tests = true
  unused-packages = true

[yorinasub17]

AFAIK I think it still works even if the Gopkg files and vendor dirs are at the root.

That Gopkg.toml file is slightly different from what I expected. In my experiment, my Gopkg.toml was (removed the comments for brevity):

[[constraint]]
  name = "github.com/gruntwork-io/terratest"
  version = "0.14.4"

[[constraint]]
  branch = "release-1.12"
  name = "k8s.io/api"

[prune]
  go-tests = true
  unused-packages = true

Can you try the following? I want to rule out environment differences:

1. clone https://github.com/gruntwork-io/terratest-helm-testing-example
2. Remove the Gopkg files in the test folder
3. At the root, run dep init
4. Run go test -v ./test
5. You can Ctrl+C out of it if it starts running the tests

If that works, then something must be preventing dep from parsing the dependencies correctly. Can you see if you have any syntax errors by running go fmt on your code?

[gregwhorley]

@yorinasub17 Thanks for helping me troubleshoot. I went through the steps you requested until an error occurred at step 3:

$ pwd
/home/greg/go/src/github.com/gruntwork-io/terratest-helm-testing-example
$ ls test/Gopkg*
ls: cannot access test/Gopkg*: No such file or directory
$ dep init
Importing configuration from godep. These are only initial constraints, and are further refined during the solve process.
Importing configuration from glide. These are only initial constraints, and are further refined during the solve process.
init failed: unable to solve the dependency graph: Solving failure: No versions of golang.org/x/net met constraints:
	master: unable to update checked out version: fatal: reference is not a tree: b630fd6fe46bcfc98f989005d8b8ec1400e60a6e
: command failed: [git checkout b630fd6fe46bcfc98f989005d8b8ec1400e60a6e]: exit status 128
	release-branch.go1.10: Could not introduce golang.org/x/net@release-branch.go1.10, as it is not allowed by constraint master from project github.com/gruntwork-io/terratest.
	release-branch.go1.11: Could not introduce golang.org/x/net@release-branch.go1.11, as it is not allowed by constraint master from project github.com/gruntwork-io/terratest.
	release-branch.go1.12: Could not introduce golang.org/x/net@release-branch.go1.12, as it is not allowed by constraint master from project github.com/gruntwork-io/terratest.
	release-branch.go1.6: Could not introduce golang.org/x/net@release-branch.go1.6, as it is not allowed by constraint master from project github.com/gruntwork-io/terratest.
	release-branch.go1.7: Could not introduce golang.org/x/net@release-branch.go1.7, as it is not allowed by constraint master from project github.com/gruntwork-io/terratest.
	release-branch.go1.8: Could not introduce golang.org/x/net@release-branch.go1.8, as it is not allowed by constraint master from project github.com/gruntwork-io/terratest.
	release-branch.go1.9: Could not introduce golang.org/x/net@release-branch.go1.9, as it is not allowed by constraint master from project github.com/gruntwork-io/terratest.

Running go fmt in my tests directory produced no output.

EDIT I should also mention that I gave up trying dep earlier and ran go get github.com/gruntwork-io/terratest/modules/helm to at least get the test to run.

[yorinasub17]

EDIT I should also mention that I gave up trying dep earlier and ran go get github.com/gruntwork-io/terratest/modules/helm to at least get the test to run.

Makes sense.

Huh. I am really stumped in the difference in behavior here. What happens if you just use my Gopkg.toml above and run dep ensure?

[gregwhorley]

@yorinasub17 If I run dep ensure with a Gopkg.toml that has your changes (I replaced k8s.io/api with k8s.io/apimachinery to stop a warning about direct dependencies from being output) I get

$ dep ensure
Solving failure: No versions of golang.org/x/crypto met constraints:
	master: unable to update checked out version: fatal: reference is not a tree: 38d8ce5564a5b71b2e3a00553993f1b9a7ae852f
: command failed: [git checkout 38d8ce5564a5b71b2e3a00553993f1b9a7ae852f]: exit status 128
	release-branch.go1.11: Could not introduce golang.org/x/crypto@release-branch.go1.11, as it is not allowed by constraint master from project github.com/gruntwork-io/terratest.
	release-branch.go1.12: Could not introduce golang.org/x/crypto@release-branch.go1.12, as it is not allowed by constraint master from project github.com/gruntwork-io/terratest.

[yorinasub17]

Gah so it's the same issue...

Looking around, I found this: golang/dep#484 which looks similar to your issue. In there, they suggest clearing the dep cache. Does $GOPATH/pkg/dep/https---golang.org-x-crypto exist? If so, try deleting that and running dep again.

[gregwhorley]

I see several things inside $GOPATH/pkg/dep/sources but nothing for https---golang.org-x-crypto

$ pwd
/home/greg/go/pkg/dep
$ ls
sources
$ cd sources/ ; ls | grep golang
https---github.com-golang-appengine
https---github.com-golang-glog
https---github.com-golang-protobuf

So I just cleared out the contents of $GOPATH/pkg/dep/sources then went back to my project and ran dep ensure. New error this time:

$ dep ensure
Solving failure: No versions of k8s.io/client-go met constraints:
	v11.0.0: Could not introduce k8s.io/client-go@v11.0.0, as it is not allowed by constraint release-9.0 from project github.com/gruntwork-io/terratest.
*walks down several other versions until we get to release-9.0*
        release-9.0: Could not introduce k8s.io/client-go@release-9.0, as it requires package k8s.io/api/admissionregistration/v1alpha1 from k8s.io/api, but in version master that package is missing.

[gregwhorley]

It has to be something in my environment causing the issue. I just ran dep ensure from within a golang docker container with my project source mounted in the container's GOPATH and it populated Gopkg.lock and the vendor directory without throwing errors. Thanks to @yorinasub17 for taking the time to help troubleshoot! I'm closing this since I have a workaround.

[yorinasub17]

Thanks for the update! I am sorry we couldn't figure it out in the end, but glad to know you got a workaround working with docker.

[brikis98]

One thing I noticed in one of the comments:

Importing configuration from glide. These are only initial constraints, and are further refined during the solve process.

Do you have a glide.yaml or something in there that might be messing things up?

[yorinasub17]

@brikis98 That's actually coming from the kubernetes packages. I had that too (as well as similar info for godep), but it didn't affect my config.

[brikis98]

Ah, OK then. Back when we used glide, I had issues with golang.org-x-crypto too... It was one of the things that motivated the move to dep. Figured I'd mention it, but sounds like it's unrelated in this case.