cmd/go: 'go mod init' should validate module paths for v2+ dependencies

[thepudds]

What version of Go are you using (go version)?

go 1.12.4

Does this issue reproduce with the latest release?

Yes, including tip.

What did you do?

Observed problems in modules that were ultimately traced back to losing information in the go mod init process.

What did you expect to see?

go mod init setting proper module paths for v2+ dependencies when converting from a prior dependency manager.

What did you see instead?

go mod init never seems to set the /vN as part of the module path for a require statement

Example from scratch

This can happen in different ways in the wild, but here is an example from scratch.

$ export GO111MODULE=off
$ go get github.com/gobuffalo/packr/v2
$ mkdir -p $GOPATH/src/example.com/hello
$ cd $GOPATH/src/example.com/hello
$ cat <<EOF > hello.go
package hello
import _ "github.com/gobuffalo/packr/v2"
EOF

$ dep init
  ...
  Using ^2.2.0 as constraint for direct dep github.com/gobuffalo/packr
  ... 

$ export GO111MODULE=on
$ gotip mod init
go: creating new go.mod: module example.com/scratchpad/hello
go: copying requirements from Gopkg.lock

$ grep packr go.mod
        github.com/gobuffalo/packr v0.0.0-20190416161152-cda4ac255773

cda4ac255773 corresponds to the v2.2.0 tag, but the require in the created go.mod is missing the required /v2 after packr.

In other words, go mod init created an incorrect go.mod.

Gopkg.lock and Gopkg.toml created by this example

$ cat Gopkg.lock
# This file is autogenerated, do not edit; changes may be undone by the next 'dep ensure'.


[[projects]]
  digest = "1:d29ee5ef14a7e0253facd0bcebe6a69a7a4e02a67eb24d2aacd8ccb4a7cea6fc"
  name = "github.com/gobuffalo/envy"
  packages = ["."]
  pruneopts = "UT"
  revision = "043cb4b8af871b49563291e32c66bb84378a60ac"
  version = "v1.7.0"

[[projects]]
  branch = "master"
  digest = "1:5cc0801073158de62986111e35292e498e5d762a45da812bac744986fbe11daa"
  name = "github.com/gobuffalo/genny"
  packages = ["."]
  pruneopts = "UT"
  revision = "3ca520ef0d9ea4981534aae98bf9a003d9b18678"

[[projects]]
  branch = "master"
  digest = "1:7779ec172ea5898029020425bf58586c68c341fe4c3c23c2508605767de3ce47"
  name = "github.com/gobuffalo/gogen"
  packages = [
    ".",
    "goimports",
    "gomods",
  ]
  pruneopts = "UT"
  revision = "8f38393713f59beb43221f903c07f58f0377a063"

[[projects]]
  branch = "master"
  digest = "1:f391538f4166f7c0aa50943372af196304488b039fe663c134d42f064e1ed92c"
  name = "github.com/gobuffalo/logger"
  packages = ["."]
  pruneopts = "UT"
  revision = "86e12af44bc273e0063fad5b6ad588890b8cfe6b"

[[projects]]
  digest = "1:a3259d2a6c245b62e40232800e3ebb4d7d5840940638de0e4754cb13cdd37790"
  name = "github.com/gobuffalo/mapi"
  packages = ["."]
  pruneopts = "UT"
  revision = "0bb5e840be332d4280e40f2e6c50777c615bbac5"
  version = "v1.0.2"

[[projects]]
  branch = "master"
  digest = "1:8561bb77169b6b26a39bcf2945f582a10ccbe3533d644ada72094bd5fc86b4d5"
  name = "github.com/gobuffalo/packd"
  packages = ["."]
  pruneopts = "UT"
  revision = "a385830c7fc0495b810318788a043c7655da337d"

[[projects]]
  digest = "1:0fac441be66684180b031233313b8315644c0734fd73944c3e156e3cf5fbe203"
  name = "github.com/gobuffalo/packr"
  packages = [
    "v2",
    "v2/file",
    "v2/file/resolver",
    "v2/file/resolver/encoding/hex",
    "v2/jam/parser",
    "v2/plog",
  ]
  pruneopts = "UT"
  revision = "cda4ac25577350dac682c8715dc6659327232596"
  version = "v2.2.0"

[[projects]]
  branch = "master"
  digest = "1:b90ac64448d67ef218124185a8524f3c665fb7f51f096d5dd51ce950a1775d89"
  name = "github.com/gobuffalo/syncx"
  packages = ["."]
  pruneopts = "UT"
  revision = "33c29581e754bd354236e977dfe426e55331c45d"

[[projects]]
  digest = "1:ecd9aa82687cf31d1585d4ac61d0ba180e42e8a6182b85bd785fcca8dfeefc1b"
  name = "github.com/joho/godotenv"
  packages = ["."]
  pruneopts = "UT"
  revision = "23d116af351c84513e1946b527c88823e476be13"
  version = "v1.3.0"

[[projects]]
  digest = "1:38d2e838817528fd7d1ccdf7fa6700ee0f1727b3c484d13cab54bfbdaab12b0e"
  name = "github.com/karrick/godirwalk"
  packages = ["."]
  pruneopts = "UT"
  revision = "6d1c7760ec857d2984abe0a23fa263877f50d3f0"
  version = "v1.8.0"

[[projects]]
  digest = "1:31e761d97c76151dde79e9d28964a812c46efc5baee4085b86f68f0c654450de"
  name = "github.com/konsorten/go-windows-terminal-sequences"
  packages = ["."]
  pruneopts = "UT"
  revision = "f55edac94c9bbba5d6182a4be46d86a2c9b5b50e"
  version = "v1.0.2"

[[projects]]
  branch = "master"
  digest = "1:6e2ed1bdbf1d14b4d0be58bcd3f1c3000c1e226964354457b8e6ca69e83a1cbb"
  name = "github.com/markbates/oncer"
  packages = ["."]
  pruneopts = "UT"
  revision = "bf2de49a0be218916e69a11d22866e6cd0a560f2"

[[projects]]
  digest = "1:28687e854cec240942c103259668b132a8450b05a6cc677eea2282b26ae29310"
  name = "github.com/markbates/safe"
  packages = ["."]
  pruneopts = "UT"
  revision = "6fea05a5732486546a4836b7a1d596c5ec687b98"
  version = "v1.0.1"

[[projects]]
  digest = "1:cf31692c14422fa27c83a05292eb5cbe0fb2775972e8f1f8446a71549bd8980b"
  name = "github.com/pkg/errors"
  packages = ["."]
  pruneopts = "UT"
  revision = "ba968bfe8b2f7e042a574c888954fccecfa385b4"
  version = "v0.8.1"

[[projects]]
  digest = "1:e09ada96a5a41deda4748b1659cc8953961799e798aea557257b56baee4ecaf3"
  name = "github.com/rogpeppe/go-internal"
  packages = [
    "modfile",
    "module",
    "semver",
  ]
  pruneopts = "UT"
  revision = "438578804ca6f31be148c27683afc419ce47c06e"
  version = "v1.3.0"

[[projects]]
  digest = "1:fd61cf4ae1953d55df708acb6b91492d538f49c305b364a014049914495db426"
  name = "github.com/sirupsen/logrus"
  packages = ["."]
  pruneopts = "UT"
  revision = "8bdbc7bcc01dcbb8ec23dc8a28e332258d25251f"
  version = "v1.4.1"

[[projects]]
  branch = "master"
  digest = "1:bbe51412d9915d64ffaa96b51d409e070665efc5194fcf145c4a27d4133107a4"
  name = "golang.org/x/crypto"
  packages = ["ssh/terminal"]
  pruneopts = "UT"
  revision = "b43e412143f90fca62516c457cae5a8dc1595586"

[[projects]]
  branch = "master"
  digest = "1:e18a5437c48ea1bd6dcd960efdd634d05ae5c2dd033d2d17f73cc20be92895ac"
  name = "golang.org/x/sys"
  packages = [
    "unix",
    "windows",
  ]
  pruneopts = "UT"
  revision = "f0ce4c0180bef7e9c51babed693a6e47fdd8962f"

[[projects]]
  branch = "master"
  digest = "1:be1ab6d2b333b1d487c01f1328aef9dc76cee4ff4f780775a552d2a1653f0207"
  name = "golang.org/x/tools"
  packages = [
    "go/ast/astutil",
    "go/gcexportdata",
    "go/internal/gcimporter",
    "go/internal/packagesdriver",
    "go/packages",
    "go/types/typeutil",
    "imports",
    "internal/fastwalk",
    "internal/gopathwalk",
    "internal/module",
    "internal/semver",
  ]
  pruneopts = "UT"
  revision = "6732636ccdfd99c4301d1d1ac2307f091331f767"

[solve-meta]
  analyzer-name = "dep"
  analyzer-version = 1
  input-imports = ["github.com/gobuffalo/packr/v2"]
  solver-name = "gps-cdcl"
  solver-version = 1

$ cat Gopkg.toml

# Gopkg.toml example
#
# Refer to https://golang.github.io/dep/docs/Gopkg.toml.html
# for detailed Gopkg.toml documentation.
#
# required = ["github.com/user/thing/cmd/thing"]
# ignored = ["github.com/user/project/pkgX", "bitbucket.org/user/project/pkgA/pkgY"]
#
# [[constraint]]
#   name = "github.com/user/project"
#   version = "1.0.0"
#
# [[constraint]]
#   name = "github.com/user/project2"
#   branch = "dev"
#   source = "github.com/myfork/project2"
#
# [[override]]
#   name = "github.com/x/y"
#   version = "2.4.0"
#
# [prune]
#   non-go = false
#   go-tests = true
#   unused-packages = true


[[constraint]]
  name = "github.com/gobuffalo/packr"
  version = "2.2.0"

[prune]
  go-tests = true
  unused-packages = true

Additional comments

This issue is not about how to resolve this manually, which I know how to do. Rather, it would be better to not to have to resolve manually, but more importantly would better if others did not encounter follow-on issues due to this behavior. Also, in some cases go mod tidy or similar can let someone be "lucky" and not notice this, but in some cases the version information converted by go mod init is important to avoid follow-on issues.

This was a less significant problem early in the history of modules because not many v2+ packages had adopted modules yet.

However, more v2+ dependencies are adopting go.mod files themselves as modules move towards being on by default in 1.13.

I have seen this behavior be the root cause of hard-to-diagnose problems in the wild, and the rate of those problems likely would increase with more v2+ modules in the ecosystem. (And then later, this problem rate of course would decrease, given it is transitional in nature).

This is related to #30161, but that suggested an alternative solution of dropping the dependency during the conversion process, and that issue was ultimately closed by the reporter after the current behavior was explained and the reporter understood how to manually work around. However, ideally go mod init would record the right information here without the need for someone to read documentation or manually work around.

[bcmills]

Somewhat related to #29814.

[dmitshur]

I know there have been some improvements in go mod init in this area, so this may be in scope of that.

Additionally, this may be related to (or already implemented in) the experimental gorelease tool. /cc @jayconrod

/cc @jayconrod @matloob @bcmills per owners.

[thepudds]

Hi @bcmills, as far as I could tell from a quick look, go mod init in go 1.15.3 seems to be falsely complaining more than it should about finding /v2, /v5, and /v22 in import paths in vendor.conf. A possible example of this is in moby/moby#41504 (comment).

This is probably the wrong place to make this comment, but here we are 😅