Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.Sign up
cmd/go: support simultaneous edits of interdependent modules #27542
The main workaround at the moment is to add
One option (based on #26377) might be to set up a sort of pseudo-
Assuming that all of those modules include metadata from some supported VCS tool, the
One of the downsides of that approach is that it reintroduces the need to indicate the root of the “local modules” tree.
We already have (had?) this: it's the vendor directory. If you run
a) would it need to rely on VCS metadata (vs. maybe it could rely on relative on-disk location), and
The file tree with a
One piece of the puzzle might allowing a user to opt-in in some way to having the relative on-disk location between the "little GOPATHs" always be the same in their development/environment/test/build/CI environment (or more precisely, not "always" the same, but have the relative on-disk locations between the "little GOPATHs" be constant until someone starts further modifying things such as by introducing
Perhaps a rule could be a "parent"
In other words, if you have a "parent"
Then perhaps the rule could be that
Or maybe that might break compatibility with Go 1.11 behavior? In which case, maybe there is some signal in the "parent"
(or maybe rather than
I've been noodling on something like this for a bit given the repeated questions around the current need for
Reusing the vendor directory is an interesting idea, although I see a few rough edges:
Any idea how we could resolve those? (Or do you suppose that they'll turn out not to be a big deal in practice?)
I think so, yes: in particular, version tags (and commit hashes, for pseudoversions) are not otherwise present in the source tree.
That's an interesting idea, but at the moment when we find a
Agreed that would be a change in behavior. Part of what I was trying to outline towards the end of my comment above #27542 (comment) was that it could be a conscious choice to opt-in to these semantics (e.g., perhaps the "parent"
Side note is that the number of directories that would need to be walked upward from a given
In any event, perhaps another approach is better, but setting aside the particulars of what I had sketched out above, for the purposes of this issue it could be worth thinking more broadly about how to exploit information that might naturally already exist in terms of inter-related modules (which is part of what I was aiming for here in terms of using the location of
Hope this issue been solved.
I'm using local module for some reason ( not ready yet to publish to the pubic VCS, so no domain names for the module, and our internal git servicec not gettable for now, because of https ( no insecure setting) ).
Currently, using local module is a bad experience ( need manual replace directive for every package(module) in a project ).
Some experience of add module support
Recording some other suggestions/comments from this thread:
These two comments were follow-ups to conversation in that thread around
Based on a conversation in the Go Slack I was asked:
My favorite part about the
While this usage has become less common with tools like
Lastly, I've seen people use it to be able to zip up the entire tree and ship that to another developer to help them troubleshoot a weird issue. They easily had the full working tree of the project.
So in summary, I'd like
General comment: There could potentially be a broader issue opened up with a title something like:
The comment from @theckman above could fall into such an issue.
Even though modules are in many ways "a little GOPATH", my personal opinion is that the biggest way modules today do not provide an overall "GOPATH-like experience" is the increase in complexity that arises once you have multiple modules, and hence the more use-case-based comments above from @theckman about what he values about GOPATH also make sense here in this issue about dealing with multiple modules.
Above, @theckman is placing a very high value on the consistency of GOPATH, including across developers. I have seen others express similar sentiments. At least for me, I don't know if modules will ever provide 100% of a GOPATH-like experience, mainly because modules enable more choice. For example, even if an individual or a team chooses to place 100% of their modules together in some consistent location (and assuming things like this issue here #27542 is resolved in some nice way), their approach might turn into very much of a GOPATH-like experience for that individual or team (including consistency), but the fact that a different team might make a different choice seems to imply that retaining all the consistency that was delivered by GOPATH might be at odds with the flexibility that modules offer (including the flexibility modules offer to the people who state "I just want to clone a repo wherever I want on my disk").
But even if 100% of a GOPATH-like experience might not be possible, it might still be an interesting question as to how many of GOPATH's benefits can be preserved in a modules world...
A few more specific reactions:
Right now, it's probably fair to say that use case is fairly awkward with the core go tooling, but reasonably nice for at least one-off fixes with @rogpeppe's gohack (and which again gets to the subject of this issue #27542 in terms of dealing with multiple modules).
@theckman also wrote:
Those two comments are things that could be preserved with modules, but in part depends on how modules evolve. Perhaps those could be used as at least part of the criteria for evaluating the solution to this multi-module issue #27542.
Following up from comments made to my post at #28868. My issue is not so much a build issue as described above, but how an IDE is supposed to support all of this, and I want to keep the IDE issue on the radar here. IDE's have bigger needs than the build. They do code completion, syntax checking, etc. The current module implementation which allows a particular module to override other modules depending on what directory you call the build tools out of makes the IDE's job difficult. The example situation I describe in #28868 is the most basic. Imagining a very active project with many modules in different stages of work, and with multiple executables, can make things quite difficult to keep straight.
Both of these answers are making some assumptions about an IDE that are not necessarily true. What is this "working directory" in which the IDE was opened in a multi-project situation? Whether you have multiple modules in one workspace, or multiple workspaces, one for each module, the IDE still needs to know what go.mod file the developer intends to use in a particular build in case there are replace statements there that point to alternate sources.
I agree with the goals. Its just that the replace statement implementation has recreated that problem. With modules, the build system now relies on the active go.mod file, which changes depending on the current working directory from which the build tools are called. Because of this, IDE's cannot track this in a many-module situation. One or two modules, no problem, so 90% of the time, its probably fine.
I was noodling on this, and not sure it solves all the problems listed, but here is a suggestion. However, as a preface, I think one issue is that go.mod is simply overloaded. I believe it was originally thought of as a better vendoring tool, and then it also became a replacement for GOPATH, and with the
Therefore, I think it makes sense to break it apart, creating an additional file, similar to the idea of the go.mod.local idea mentioned, but with the idea it would just have replace and exclude statements. However, instead of this being picked up by name convention like the go.mod file is, I think the go tool should be modified so that you can specify on the command line what the overriding go.mod file (or whatever it gets called), should be. That way the replace statements are not based on the CWD.
Also, if someone wants to still use a vendor directory, maybe something in there could specify this? Perhaps a replace directive that points to a top-level domain would do the trick?
In addition, since this is essentially a build configuration file at this point, it should have a mechanism to respond to GO's current build configuration mechanism, which are build tags. So, something like this:
Something like that.
Anyways, its a start of an idea we can poke at.
Re-activating this issue with some thoughts after some related chat in #26344.
From the existing comments it appears that either there are a few different issues that are being conflated or, more likely, the actual use-case that we want to facilitate is not defined clearly enough. What mostly points me in this direction is that we are discussing to what measure the "GOPATH feeling" can be preserved with modules. That is a good, but also vague, term that covers many different behaviours. Something that is further highlighted by the different opinions already expressed above.
If I try to break down the different use-cases from my own perspective I get the following:
Modifying inter-dependent modules in different repositories
Preserving the "canonical" directory structure of GOPATH.
Or put differently, the "zip up my dependency tree and send it to a friend for debugging" compatibility.
This is still possible in the current state without modifications to your workflow. Either your debugging is limited to a single repo and you simply send over your branch / commit. The
Preparing upstream patches to inter-dependent modules and ensuring that cross-module version references are accurate.
The issue here is actually less tied to modules, or even Go, than to VCS in general. I'll use the example of modifying related modules A and B. If the dependency is only from A to B then after preparing your patches locally you are forced (even in GOPATH-mode and / or with the use of
Modifying inter-dependent modules in the same repository
This is a more curious situation and it is part of a wider problem that is actually acutely felt by multiple large-scale community projects (k8s, Prometheus, ...). I've done a much more elaborated discussion of this in the introduction of a design document for a tool to help deal with this specific situation.
Although it is possible to keep modules in the same repository relatively independent, we'll assume here that there are two and that they are quite strongly related. Them being in the same repository mainly opens up the possibility for developers to commit changes to both the modules in a single PR. A direct consequences of this is a likely need for some form of strong consistency in the versions that they are referencing (either one-way or in a circular dependency), ideally the same commit. This however would run into the same issue, even if we are in the same VCS, as what I described above for cross-repo inter-dependent modules.
Just wanted to add a hypothetical example that is very similar to a situation I'm facing right now. We are on go 1.13 and are trying to plan a migration from
Assume that we have fully qualified package names - I'm just shortening them for this example.
Suppose we have a project called "govlc", a video player with some fancy UI. I want to separate out the video encoding/decoding engine into a library, "golibvlc" that others can use. I also want to separate out individual decoders such as "gox264" and "gorawvideo". These modules do not necessarily live on a centralized online repository (Git was built to be able to work in a decentralized way, after all).
For development, I set up a local project workspace directory with the following subdirectories (each an independent local Git project and a module):
When working on "govlc", I'll also be working on "golibvlc" at the same time. I need my local changes available without needing to push "golibvlc", "gorawvideo", or "gox264" work in progress commits.
Let's say we hired an intern, and don't want to grant access to the main repository. We want him to work on a fork of "govlc" called "govlcfree". This fork only depends on "gorawvideo", and he'll be working on "gorawvideo" and "govlcfree" at the same time. He occasionally pulls from my personal copy of the repository, and will push acceptable changes directly to me for "gorawvideo".
From what I can tell, the current Go modules implementation makes these kinds of workflows difficult. It forces all modules to go into centralized, online revision control before it can be used by another project. As a result, a developer can't work on multiple modules concurrently.
We could use the "replace" keyword to make local references, but we must use relative paths, which are hacky (personal opinion) because we need to make assumptions of the directory hierarchy.
If we combine the components of the entire project into a single module, then we'd have to export the entire stack including all the proprietary parts.
We'd really like a way to use modules locally, so they can refer to each other without revision control or assumptions about directory hierarchy. I've seen a bunch of ideas - such as a
That way, we could even emulate the old
That block marks those three modules as related projects that may exist on the local filesystem. The Go tools could use them instead of the managed version from