cmd/go: go modules ignores go.mod in semver repos not using semantic import versioning #27009
Comments
|
@gopherbot, please add label modules |
|
Hi @markbates, I'm not 100% sure this is the root cause, but given 'pop' is >= v2, I believe you need to have the major version at the end of the module path in your Right now, it shows:
https://github.com/gobuffalo/pop/blob/d38e78212cc54cd9317a67522137701a3f40af82/go.mod This is covered a bit on the "Modules" wiki here: https://github.com/golang/go/wiki/Modules#semantic-import-versioning Could you try appending the major version to the end of the module path in the |
|
Yes, but as noted in this blog post, https://blog.gobuffalo.io/the-road-to-1-0-go-modules-c55b42af70de, doing so would cause breakage for lots of people. I would rather not create a release that would potentially harm users to something like that. According to @spf13 the Semantic Import Versioning rule has been relaxed, so it would make sense that this would work. The logic doesn't make sense. For packages that are already about v2 can't use Go Modules as it currently stands, or else Go Modules will ignore them. That is definitely not the logical workflow. I can use |
|
Also, at the risk of getting ahead of ourselves, if that was the issue that was biting you, I would suggest re-purposing this issue into how it might have been easier to diagnose the issue (e.g., with a warning, or with -v output from 'go get', or ___). edit: Sorry, got ahead of things with this comment. Please ignore this particular comment for now. |
|
Hi @markbates, I'm not sure specifically what @spf13 was referencing, but perhaps he was describing the work where Go versions 1.9.7+ and 1.10.3+ have been updated to know how to properly interpret a /v2 or higher that appears in an import path, for example: https://go-review.googlesource.com/c/go/+/109340 As far as I am aware, that would still requiring having a /v4 at the end of the If you are looking to still support Go 1.8.x with a v4.x.x module, then the only alternatives I am aware of are:
All that said, I'm just sharing my current personal understanding... I'm happy to learn more myself, and I certainly don't want to mis-direct you, so please take with a grain of salt. |
|
@thepudds All of that has been relaxed because it presents a chicken and egg problem for a lot of >2 repos out there. The logic here is either coded incorrectly, which is what I think it is (a left over from earlier stricter releases), or the logic is fundamentally flawed, which I don't think it is. Not pulling perfectly good releases when searching for "latest" because they contain a /cc @rsc |
|
Hi @markbates, and just to confirm, part of the current goal is to continue to support people using Go 1.8.x? I see on the Buffalo site:
I'm guessing the intent is to continue to support 1.8.x right now, but I thought worth asking. |
|
We’re moving to 1.9 in the next release. |
|
And just to be clear, this has zero to do with Buffalo. I’m just using that repo as an example. Please don’t get hung up on that. This is about the logic of finding the “latest” version of a v2x repo. |
|
Is it acceptable to require version 1.9.7+ for buffalo? As far as I am aware, that it the oldest release that picks up the minimal module awareness that makes older releases able to most easily consume a v2+ module. |
|
Please stop making this about Buffalo. This is about the logic of the resolver. |
|
Hi @markbates, sorry, we posted at close to the same time. |
|
Hi @markbates, I'll summarize what I was trying to say, and then leave it to others that are more knowledgeable. My (possibly incorrect) understanding:
All that said, my understanding might be stale, or perhaps was just wrong to begin with. ;-) |
bradfitz
changed the title
|
Hi Mark, Sorry for all the confusion. Lots going on here. It's going to take a while for everyone to get up to speed and for a good variety of intro docs to be written. First, it appears you think you can opt out of semantic import versioning. You cannot. If you're using modules, you must use semantic import versioning.
Normally, if the go command sees a tag v4.0.0 in the gobuffalo/pop repo, it expects that tag to be the github.com/gobuffalo/pop/v4 module. As an exception, to help with repos tagged before semantic import versioning, if the go command sees a tag >= v2.0.0 but the tagged tree has no go.mod, then the go command puts that version into the v0/v1 series - that is, it lets it be a version for github.com/gobuffalo/pop - but with a This exception is what made it possible for You can see all the versions by using: But when you create and tag a version with a go.mod, that's interpreted as a signal that the repo is aware of modules and therefore aware of semantic import versioning, so if v4.5.10 has a go.mod, then it should be saying As @thepudds points out, pop can move to the /v4/ paths and older go commands (1.9.7+ and 1.10.3+) will do the right thing. But not 1.8 (which is officially unsupported at this point, since 1.10 came out). |
|
I’m sorry Russ, but the logic definitely doesn’t make sense. So the solution is to not use modules? Again, this is just one repo out of a lot of repos controlled by a lot of people. This is really confusing behavior I spent a month tracking down. Either don’t pull 4x and only pull 1x or allow all 4x releases. This is a major inconsistency that does not follow the principle or least surprise.
…--
Mark Bates
On Aug 17, 2018, 11:49 PM -0400, Russ Cox ***@***.***>, wrote:
Hi Mark,
Sorry for all the confusion. Lots going on here. It's going to take a while for everyone to get up to speed and for a good variety of intro docs to be written.
First, it appears you think you can opt out of semantic import versioning. You cannot. If you're using modules, you must use semantic import versioning.
• github.com/gobuffalo/pop is the module path and import path for the v0/v1 series of versions in the gobuffalo/pop repo.
• github.com/gobuffalo/pop/v2 is the module path and import path for the v2 series.
• github.com/gobuffalo/pop/v3 is the module path and import path for the v3 series.
• github.com/gobuffalo/pop/v4 is the module path and import path for the v4 series.
Normally, if the go command sees a tag v4.0.0 in the gobuffalo/pop repo, it expects that tag to be the github.com/gobuffalo/pop/v4 module.
As an exception, to help with repos tagged before semantic import versioning, if the go command sees a tag >= v2.0.0 but the tagged tree has no go.mod, then the go command puts that version into the v0/v1 series - that is, it lets it be a version for github.com/gobuffalo/pop - but with a +incompatible suffix. go help modules explains:
Code written before the semantic import versioning convention
was introduced may use major versions v2 and later to describe
the same set of unversioned import paths as used in v0 and v1.
To accommodate such code, if a source code repository has a
v2.0.0 or later tag for a file tree with no go.mod, the version is
considered to be part of the v1 module's available versions
and is given an +incompatible suffix when converted to a module
version, as in v2.0.0+incompatible. The +incompatible tag is also
applied to pseudo-versions derived from such versions, as in
v2.0.1-0.yyyymmddhhmmss-abcdefabcdef+incompatible.
This exception is what made it possible for go get github.com/gobuffalo/pop to consider the v4.5.9 tag part of the v0/v1 series. It called the version v4.5.9+incompatible instead of v4.5.9.
You can see all the versions by using:
$ go list -m -versions -f '{{join .Versions "\n"}}' ***@***.***
v3.3.0+incompatible
v3.3.1+incompatible
...
v4.5.8+incompatible
v4.5.9+incompatible
v4.5.11+incompatible
$
But when you create and tag a version with a go.mod, that's interpreted as a signal that the repo is aware of modules and therefore aware of semantic import versioning, so if v4.5.10 has a go.mod, then it should be saying module github.com/gobuffalo/pop/v4 and the code should be importing directories in that repo using paths with pop/v4 in them. That's why all those later tags are not seen - now that there's a go.mod in the repo, the expectation is that those tags are tags for the module github.com/gobuffalo/pop/v4, not plain pop (v1).
As @thepudds points out, pop can move to the /v4/ paths and older go commands (1.9.7+ and 1.10.3+) will do the right thing. But not 1.8 (which is officially unsupported at this point, since 1.10 came out).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
edit: The context for this comment here is that semantic import versioning means code that opts into modules must include the major version in the import path for any imported v2+ modules. However, a natural question is then how does that interact with other code in the same build if that other code has not yet opted in to modules. The short version is the go command "does the right thing" when using Go 1.9.7+, 1.10.3+ or 1.11, including old code that has not opted into modules does not need to be updated to include the major version in import paths when consuming v2+ modules. The remainder of this comment attempts to describe that in more detail, including via a runnable example that includes (in a single build) a shared v2+ module being imported both by code that has opted in to modules as well as by code that has not yet opted in to modules. The Buffalo blog post mentioned above includes the following snippet as part of the problem statement:
It sounds like that problem is not the only modules-related issue confronting Buffalo, but I wanted to share a quick runnable example that tries to show an illustration of that particular issue (although using only three trivial packages). I think this example ends up showing:
Perhaps this example below helps makes an aspect of the conversation here more concrete, including for any others reading this issue. (Or perhaps this example is not on target, in which case I will likely learn something myself). In any event, the more concrete example is:
So at first glance it seems we could be in trouble, because in a single build:
However, my understanding is that potential issue is automatically resolved if using 1.9.7+, 1.10.3+, or 1.11, because the go tooling has been taught how to automatically handle that and let a build straddle a pre-module world and post-module world (and it does so without requiring oldhello.goHere is Package aHere is package
Also,
Running example with Go 1.9.7 (works)Here is running Running example with Go 1.8 (fails, as expected)In contrast, here is building SummaryFor the piece of the problem statement in the blog snippet at the top of this comment from the Buffalo blog:
All that said, this is just a simple example intended to help the conversation. Sorry for the length. |
|
@rsc & @bcmills I had a chat with @markbates about this and it's clear to me from this that there is an issue here that needs to be addressed. It's not that there is a problem with Go modules, per say, rather it's a flaw in our education. There are people currently publishing versions incorrectly and it's becoming an issue. Another example is https://github.com/gofrs/uuid which added a go.mod file in the tag 3.1.1 with module "github.com/gofrs/uuid/v3". The reason this is an issue is that there are now 4 possibilities when importing this package and each of them results in a different set of source being imported: You import "github.com/gofrs/uuid" and you are using modules → 3.1.0 |
bcmills
added
the
NeedsInvestigation
|
@spf13 I did some investigation (see gofrs/uuid#61 (comment)), and found that:
|
|
Building on Bryan's comment -- one other aspect that might have been undercommunicated is "minimal module awareness", and the exact behavior you get in different scenarios with the backported behavior in Go versions 1.9.7+ and 1.10.3+, and also how it all interacts with Go 1.11 (including with the different possible settings of GO111MODULE environment variable) for code that has opted in to modules, or code that has not opted in to modules, or mixed scenarios. On the surface, it can seem like the "minimal module awareness" is simple, but I think I've seen very savvy people (including perhaps some even within the core Go team?) go through a mental process regarding how "minimal module awareness" works (and how to use it) that might be something along the lines of:
At least, I know I went through some confusion on the topic. When you look at the technical details discussed in this issue here as well as in gofrs/uuid#61, I think the exact behavior of "minimal module awareness" plays into at least parts of it. It might be that "minimal module awareness" was one of the biggest improvements to On the one hand "minimal module awareness" is just a transitional measure, but on the other hand, we are very much in a transitional phase right now and it seems it is playing an important role in practice right now. In any event, just sharing an observation on where some early adopters might have hit some snags... |
|
Change https://golang.org/cl/162699 mentions this issue: |
In the general case, we do not know the correct module path for a new module unless we have checked its VCS tags for a major version. If we do not know the correct path, then we should not synthesize a go.mod file automatically from it. On the other hand, we don't want to run VCS commands in the working directory without an explicit request by the user to do so: 'go mod init' can reasonably invoke a VCS command, but 'go build' should not. Therefore, we should only create a go.mod file during 'go mod init'. This change removes the previous behavior of synthesizing a file automatically, and instead suggests a command that the user can opt to run explicitly. Updates #29433 Updates #27009 Updates #30228 Change-Id: I8c4554969db17156e97428df220b129a4d361040 Reviewed-on: https://go-review.googlesource.com/c/162699 Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Jay Conrod <jayconrod@google.com>
In the general case, we do not know the correct module path for a new module unless we have checked its VCS tags for a major version. If we do not know the correct path, then we should not synthesize a go.mod file automatically from it. On the other hand, we don't want to run VCS commands in the working directory without an explicit request by the user to do so: 'go mod init' can reasonably invoke a VCS command, but 'go build' should not. Therefore, we should only create a go.mod file during 'go mod init'. This change removes the previous behavior of synthesizing a file automatically, and instead suggests a command that the user can opt to run explicitly. Updates golang#29433 Updates golang#27009 Updates golang#30228 Change-Id: I8c4554969db17156e97428df220b129a4d361040 Reviewed-on: https://go-review.googlesource.com/c/162699 Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Jay Conrod <jayconrod@google.com>
In the general case, we do not know the correct module path for a new module unless we have checked its VCS tags for a major version. If we do not know the correct path, then we should not synthesize a go.mod file automatically from it. On the other hand, we don't want to run VCS commands in the working directory without an explicit request by the user to do so: 'go mod init' can reasonably invoke a VCS command, but 'go build' should not. Therefore, we should only create a go.mod file during 'go mod init'. This change removes the previous behavior of synthesizing a file automatically, and instead suggests a command that the user can opt to run explicitly. Updates golang#29433 Updates golang#27009 Updates golang#30228 Change-Id: I8c4554969db17156e97428df220b129a4d361040 Reviewed-on: https://go-review.googlesource.com/c/162699 Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Jay Conrod <jayconrod@google.com>
When trying to import a package, in this case https://github.com/gobuffalo/pop, that has a semver tag
>=2.0.0, in Pop's case it isv4.6.4Go modules skips over versions that havego.modfiles.In this example Go Modules will always return
v4.5.9, which is the highest version that does not have ago.mod. Because all versions above this havego.modfiles, Go Modules refuses to pick them, resulting in strange results.Setting a version explicitly will work, but letting Go Modules find it, always fails.
It would appear that this is the line throwing away the good releases https://github.com/golang/go/blob/master/src/cmd/go/internal/modfetch/coderepo.go#L137
A repo that shows the problem can be found here: https://github.com/gobuffalo/pop-vgo
What version of Go are you using (
go version)?go version go1.11rc1 darwin/amd64Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (
go env)?What did you do?
What did you expect to see?
What did you see instead?
The text was updated successfully, but these errors were encountered: