Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.Sign up
GitHub is where the world builds software
Millions of developers and companies build, ship, and maintain their software on GitHub — the largest and most advanced development platform in the world.
cmd/go: do not allow the main module to replace (to or from) itself #34417
What version of Go are you using (
I took a quick look at this, need to investigate further though.
As I said, need to investigate further, but this seems like a strange edge case of
Thanks for taking a look. If I understand you correctly, a workaround might be to specify the versions of sample1 and sample2 in the replace statements. That's certainly something I can try. I'm worried about the implications, though, since what I'm modeling is two modules under active development locally. Would I need to increment the versions each time the source changes to deal with cache issues?
I’m not quite following this example, but why are the modules replacing themselves?
Again, I am not quite following the example, but I think you shouldn’t need to increment versions if you are doing local filesystem-based replaces of modules that are being locally developed.
This FAQ has an example:
Sorry for the quick comment, and sorry if it is not helpful.
@rselph-tibco, it should be very rare for a module to need to replace itself, since the main module itself is always considered higher-precedence than any explicit version of its own path.
The only reason I can think of for a module to replace itself would be if some other interdependent module depends on an invalid version of it, in which case it should suffice to
In the rare case in which you are developing two entirely new interdependent modules from scratch, #33370 may be relevant.
Hi, thanks for all the comments. I arrived at the modules replacing themselves only because without it, I get this when building one module or the other (in this case, sample2):
Yeah, that's not surprising: it makes
So it removes those requirements from the main module, but because of the
The core of the problem is that the module system assumes that versions are immutable, but
Probably that implies that we should also not allow the target of the
The two diagnostic fixes we can make for this are:
If we no longer allow the main module to replace itself, how would the following scenario work: gomodtest.tar.gz
Here you have modules having circular dependency on one another, but not on the constituent packages so they build successfully. Without the replace directive, Go is trying to download a second copy of the main module to satisfy the dependency of the dependent module which seems like it will cause having multiple copies of the main module as part of the build which would lead to all sorts of unexpected consequences.
@segevfiner In your example, you have two go.mod files:
It's fine for these modules to depend on each other. But it's not okay for the main module to depend on itself through
Instead, you can replace specific versions of the main module like this:
I think you might want to do this if you wanted to tag a pair of modules that depend on each other at the newly tagged versions. This would be one way to test them. It's not an easy thing to do, but it can be done without replacing the main module and without the main module's directory serving as a replacement.
That doesn't look like something that can serve as a valid work flow...
It feels like in addition to not allowing the main module to replace itself, perhaps it should also be made to compile correctly without it needing to replace itself in the case of circular dependencies? Not sure, I don't yet fully understand why this doesn't work without it...
This should not be a common workflow. It would only be needed when you need to simultaneously release multiple versions of modules that depend on each other. This workflow lets you transitively require a module version that doesn't exist yet. The
In most cases, modules versions should be released independently. If a group of modules are routinely released together, it may be better to combine them into a single module.
This doesn't work because when the main module serves as a replacement for any module, it appears that its requirements are redundant with another module (actually itself), so they are removed. You end up with an empty requirement list after any command that can modify
Sadly it doesn't even work cleanly during development of such interdependent modules without a workaround like this. Of course, having such modules in the first place is not a good idea, sadly you are sometimes stuck with lazily designed code that has such dependencies, especially coming from the vanilla
Was pointed to this issue and advised to add my data point here, in case it's related.
I'm concerned that it is practically impossible to develop Go modules in some cases without Internet access; or, in the best case with Internet access, without kind of a hacky commit to a remote server.
In developing Caddy 2 extensions, there arose a situation where two repos created a Go module cycle: https://gist.github.com/mholt/e0a14049026840e480f57c1f601f9d86 -- but not a package import cycle, so the program still compiled. Note that both modules use
There are two modules in play here: caddy/v2 and tls.dns. Both modules depended on each other, but tls.dns was still being developed only locally -- was not published yet. (No GOPATH; using only module mode.) The caddy/v2 module contained the
However, all the
Basically, it became impossible to be productive on my new Go module without another, already-published module to import and build it. There was no way for me to continue working on tls.dns in isolation.
@heschik advised me to push a commit to the tls.dns remote -- any commit, even one that just added a README. Which I did. That was enough to give me a commit hash which I could then do
A few things strike me as odd/concerning/buggy/unintuitive in this situation:
IMO, no third-party go.mod should ever have the ability to DoS the go command in your own module. It should also not be required to go online to be able to run basic go commands like
It would be nice if these problems could be addressed soon. This is a huge barrier to developing new Go modules.
To be clear, a dummy commit is not strictly necessary.
The root of the complexity here is that the main module must always be the selected version of itself. It cannot be replaced with anything else, and since it has no explicit version, its effective version must be interpreted to be higher than any other version of the same module path.
Since the main module cannot be replaced with anything else, the
One solution, as you found, is to push an empty commit and name that version.
However, another is to decide on the version that names the initial
#33370 is very relevant to this use-case, but as noted in #33370 (comment), we probably shouldn't be writing out an explicit
I think there is still a need for this ability. It may be that #33370 also would meet the need, I'm not sure. But here's the situation I'm facing, which was the basis for the reproducer I created for #39570 :
Module "foo" exists, and has a dependency on module "bar". Those modules are being simultaneously developed, and in fact the only real reason they're not the same module is that "bar" is closed-source code, requiring it to be in a separate repository.
Because they're effectively the "same module", of course they have circular dependencies. And because they're both in development, the only correct way to build them is to have them both checked out into separate directories and use "replace" directive to point at each other.
Now, when building an executable in "foo", it finds "bar" correctly. However, "bar" imports other packages in "foo". Since only "foo"'s go.mod's "replace" directives are used in the build, this means foo/go.mod must have a "replace" directive for itself so that bar's code can compile.
This actually works today, except for the issue mentioned in #39570 that "go mod tidy" gets confused by the situation and erases some necessary entries in go.mod.
As a side note: "foo" is not a module in the sense that it is ever expected any other code in the world will import foo. It's a module because it wants to use go.mod to define its own dependencies. So there will likely never be any "releases" of foo, nor should Go require us to figure out how to "release" foo such that "bar" can depend on an explicit version.
As I said, I can't tell whether #33370 would fix this or not. If it allows us to build "foo" in such a way that "bar" resolves "foo" to the same directory containing the local "foo" code (and does so in such a way that "go mod tidy" doesn't break "go build -mod=readonly"), then that would be just fine.
I want to clarify a couple points of discussion.
This issue happens because it appears to the
This breaks some invariants. It appears to the
This tends to come up when people are developing two or more modules that depend on each other, and it's not practical to release versions of those modules. There are several use cases for this: releasing new versions concurrently; developing offline; projects with separate open
The problem is triggered with a line like this (where
Instead, use this (where
A full example (you can extract with txtar).
You can run
When the main module's directory serves as a replacement, the
This still doesn't give us a very intuitive workflow. Some improvements are planned already:
Some other ideas (brainstorming):
Or perhaps, we could just do that when all versions of the main module are replaced with the main module's directory.
I will try out the "empty" workaround, thank you for the suggestion.
I still don't understand why the (annoying but seemingly intuitive) solution of "replace example.com/a => ." must be disallowed. It means "use the content of . for all imports of all versions of example.com/a", which is the same semantics as any other replace statement. The "empty" workaround is much less obvious, since it means "use the content of ../empty for all imports of all versions example.com/a (except don't really do that because you already know example.com/a exists in the current directory)".
I'm sure the difficulty in implementing this comes down to needing to special-case that situation, since go's logic for resolving modules and transitive dependencies and saving them in go.mod is a little hairy. But from a usability perspective, from the outside looking in, it feels much more direct.