New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: support simultaneous edits of interdependent modules #27542

Open
bcmills opened this Issue Sep 6, 2018 · 14 comments

Comments

Projects
None yet
6 participants
@bcmills
Member

bcmills commented Sep 6, 2018

Many module issues and questions seem to center on editing, testing, and deploying multiple (possibly mutually-interdependent, possibly cyclic) modules (examples: #27514, #27056, #26640, #26377).

The main workaround at the moment is to add replace directives among the modules to be edited, but maintaining those directives is tedious and error-prone. @rogpeppe's gohack tool automates away some of the tedium, but doesn't seem to remove the risk of accidentally checking in a go.mod with what were intended to be local, temporary replacements.

The go command should support multi-module edits in some form. It's not yet clear to me what form that should take, but I figured I'd go ahead and file an issue to collect ideas.

(CC: @rsc @myitcv @thepudds @marwan-at-work @oiooj @hyangah)

@bcmills bcmills added this to the Go1.12 milestone Sep 6, 2018

@bcmills

This comment has been minimized.

Member

bcmills commented Sep 6, 2018

One option (based on #26377) might be to set up a sort of pseudo-GOPATH tree containing the modules to be edited, with the local modules located at their module paths within that tree.

Assuming that all of those modules include metadata from some supported VCS tool, the go command would update the require and replace directives in each local module to accurately reflect the pseudo-versions (or locally-tagged versions!) of its dependencies in use, and go get operations would update the local checkouts to reflect the newly-selected versions.

One of the downsides of that approach is that it reintroduces the need to indicate the root of the “local modules” tree.

@rasky

This comment has been minimized.

Member

rasky commented Sep 6, 2018

One option (based on #26377) might be to set up a sort of pseudo-GOPATH tree containing the modules to be edited, with the local modules located at their module paths within that tree.

We already have (had?) this: it's the vendor directory. If you run go mod vendor, you get a very clean representation of your dependencies, without any binary zip file, laid out on disk to be edited, diffed, committed.

@thepudds

This comment has been minimized.

thepudds commented Sep 6, 2018

@bcmills regarding your comment in #27542 (comment):

a) would it need to rely on VCS metadata (vs. maybe it could rely on relative on-disk location), and
b) would it need to actually insert the replace directives (vs. maybe it could avoid updating the on-disk go.mod files)?

The file tree with a go.mod has been described as something like "a little GOPATH", but as has been observed, it can be awkward to wire together multiple "little GOPATHs", especially if there are many, and especially if it is a routine situation (and not something like a one-off quick debugging investigation into a dependency).

One piece of the puzzle might allowing a user to opt-in in some way to having the relative on-disk location between the "little GOPATHs" always be the same in their development/environment/test/build/CI environment (or more precisely, not "always" the same, but have the relative on-disk locations between the "little GOPATHs" be constant until someone starts further modifying things such as by introducing replace directives if they wanted to re-arrange things such as to try a local fork).

Perhaps a rule could be a "parent" go.mod serves as the anchor for how child/descendant modules could find each other (and without needing any replace directives in any of the child modules).

In other words, if you have a "parent" go.mod located in something like /my/project, the entirety of that go.mod might be:

module example.com/me/something

And /my/project/foo/go.mod might have:

module example.com/me/something/foo

require example.com/me/something/bar v1.2.3

And /my/project/bar/go.mod might have:

module example.com/me/something/bar

Then perhaps the rule could be that foo can find bar without needing any explicit replace directives in any go.mod (given foo is able to relate its own module path to its parent's module path, and map that relationship directly to the on-disk relative directory location)

Or maybe that might break compatibility with Go 1.11 behavior? In which case, maybe there is some signal in the "parent" go.mod that the user is opting in to the behavior. Maybe the "parent" go.mod could read:

module example.com/me/something

replace relative

(or maybe rather than replace relative, the "parent" go.mod could have replace auto, or replace children, or have a something added to the module directive, or an alternative filename extension, or whatever other signpost is deemed more aesthetically pleasing).

I've been noodling on something like this for a bit given the repeated questions around the current need for replace directives, but I'll confess it is not fully thought out.

@bcmills

This comment has been minimized.

Member

bcmills commented Sep 7, 2018

@rasky

We already have (had?) this: it's the vendor directory. If you run go mod vendor, you get a very clean representation of your dependencies, without any binary zip file, laid out on disk to be edited, diffed, committed.

Reusing the vendor directory is an interesting idea, although I see a few rough edges:

  1. The vendor directory itself needs to be rooted somewhere, presumably in some other module. How do we decide which module goes at the top level? (Presumably it's the one you want to commit last, but what if the task you're working on is splitting or merging two modules that are/were logically peers?)
  2. If you're making upstream edits, you likely want submodules from the same repository to remain in the same copy of their repo. (Otherwise, working with commits and branches for those submodules could get confusing pretty quickly.) But if you've copied those submodules into vendor/, now you might have two copies of the top-level repo (or a confusing symlink structure in the vendor/ directory).
  3. The existing semantics of the vendor folder are “use exactly these dependencies” — otherwise ignoring go.mod files and module dependencies. In particular, with -mod=vendor we'll use those versions of the dependencies even if the go.mod files say otherwise. In contrast, when we're making changes to a set of modules we presumably want to ensure that the go.mod requirements exactly match the source tree. That difference might be resolvable, but it's a pretty big semantic change.

Any idea how we could resolve those? (Or do you suppose that they'll turn out not to be a big deal in practice?)

@bcmills

This comment has been minimized.

Member

bcmills commented Sep 7, 2018

a) would it need to rely on VCS metadata (vs. maybe it could rely on relative on-disk location)

I think so, yes: in particular, version tags (and commit hashes, for pseudoversions) are not otherwise present in the source tree.

b) would it need to actually insert the replace directives (vs. maybe it could avoid updating the on-disk go.mod files)?

The replace directives are maybe not that big a deal, but it would definitely need to update the version requirements to reflect new tags. For example, if you're splitting one module into two mutually-dependent modules, you want to ensure that each requires the correct (updated) version of the other.

Perhaps a rule could be a "parent" go.mod serves as the anchor for how child/descendant modules could find each other (and without needing any replace directives in any of the child modules).

That's an interesting idea, but at the moment when we find a go.mod we stop looking upward in the directory tree, and I think that's probably a valuable property to preserve. For example, if we have multiple submodules in the same repo, I don't think we want to force edits to those submodules to always occur in lock-step.

@thepudds

This comment has been minimized.

thepudds commented Sep 7, 2018

@bcmills

That's an interesting idea, but at the moment when we find a go.mod we stop looking upward in the directory tree, and I think that's probably a valuable property to preserve. For example, if we have multiple submodules in the same repo, I don't think we want to force edits to those submodules to always occur in lock-step.

Agreed that would be a change in behavior. Part of what I was trying to outline towards the end of my comment above #27542 (comment) was that it could be a conscious choice to opt-in to these semantics (e.g., perhaps the "parent" go.mod only implies these semantics if it contains a replace relative directive, or some other signal). That would provide for backwards compatibility with 1.11 (including given that would not be a valid 1.11 go.mod), as well as make these semantics only kick in when someone does indeed want these different properties.

Side note is that the number of directories that would need to be walked upward from a given go.mod hunting for a "parent" go.mod would be limited to roughly the number of elements in the module path. (In other words, if you were in an extreme case where you are 1,000 levels deep in your directory structure and your go.mod reads module example.com/some/project/some/child, you don't need to walk up 1,000 directories to check for a possible "parent" go.mod, because you run out of pieces of your module path to be meaningful for this behavior (that is, for the behavior sketched out above of optionally using a "parent" go.mod as a sort of "super root go.mod" that effectively defines the relative directory location of the encompassed go.mod files in order to understand on-disk relationships between the inter-related modules without requiring N actual require directives to spell out the various relative paths between the related modules).

In any event, perhaps another approach is better, but setting aside the particulars of what I had sketched out above, for the purposes of this issue it could be worth thinking more broadly about how to exploit information that might naturally already exist in terms of inter-related modules (which is part of what I was aiming for here in terms of using the location of go.mod files relative to a "parent" go.mod), or perhaps thinking about how picking some convention in terms of how someone sets up inter-related modules might generate information that could then be exploited to automatically understand the relationships...

@chinglinwen

This comment has been minimized.

chinglinwen commented Sep 19, 2018

Hope this issue been solved.

I'm using local module for some reason ( not ready yet to publish to the pubic VCS, so no domain names for the module, and our internal git servicec not gettable for now, because of https ( no insecure setting) ).

Currently, using local module is a bad experience ( need manual replace directive for every package(module) in a project ).

Some experience of add module support

  1. go mod init ( need execute this for every package, in the main, and in the sub directory)
  2. change go.mod ( add require directive line and replace directive line for every local modules )
  3. the go module path is unclear ( currently using relative path ../other-project/pkg/foo, adds an restrict for other person to clone the projects )
    // while the old go get just simply works ( why can't go mod figure out local package path(which relate to gopath) )
@thepudds

This comment has been minimized.

thepudds commented Sep 25, 2018

Recording some other suggestions/comments from this thread:
https://groups.google.com/d/msg/golang-nuts/0KQ4ZuSpzy8/tAsI8_vVBAAJ

These two comments were follow-ups to conversation in that thread around go.mod.local and the risk of accidentally checking in a go.mod with a dev only replace.

Another idea is to have a sort of "publish local" semantics, where the go tool has support for something like -devel tags, which override the defined in go.mod. So then you would "publish" the next version to your local mod cache (just creating/updating the module of the special version), and the go tool would then make use of that.

and

Publishing to the local cache is how Maven, a build tool for Java, does it. There is the concept of snapshot versions. For Golang maybe stating master (or any other branch) as version would be fitting. Then your CI could use a master checkout as well.

@theckman

This comment has been minimized.

Contributor

theckman commented Oct 25, 2018

Based on a conversation in the Go Slack I was asked:

What you personally like about GOPATH, and/or what you hope a future modules-based experience will preserve about what you like about GOPATH

My favorite part about the GOPATH is the consistency it guarantees about where source will live on my workstation, and on the workstations of fellow developers. I know that my code will always be at $GOPATH/src/github.com/theckman/. Go became my favorite language because of the consistency it aimed for, and it's been an amazing experience having a language where I can finally write, or to provide instructions, that will work on any Go developer workstation with very little difference.

While this usage has become less common with tools like glide and dep, I also liked how easy it was to build with my own fork of a dependency if I have an outstanding upstream PR... or if the project is dead and has bugs.

Lastly, I've seen people use it to be able to zip up the entire tree and ship that to another developer to help them troubleshoot a weird issue. They easily had the full working tree of the project.

So in summary, I'd like modules to retain the on-disk consistency I've been able to rely on thus-far.

@thepudds

This comment has been minimized.

thepudds commented Oct 25, 2018

General comment: There could potentially be a broader issue opened up with a title something like:
"Try to preserve ~90% of a GOPATH-like experience with modules"

The comment from @theckman above could fall into such an issue.

However, there are already several related issues (e.g., see the initial @bcmills comment here of #27542 (comment)), so not sure if a broader new issue is useful or not.

Even though modules are in many ways "a little GOPATH", my personal opinion is that the biggest way modules today do not provide an overall "GOPATH-like experience" is the increase in complexity that arises once you have multiple modules, and hence the more use-case-based comments above from @theckman about what he values about GOPATH also make sense here in this issue about dealing with multiple modules (if the topic here of multiple modules is indeed one of the major barriers for a GOPATH-like experience).

Above, @theckman is placing a very high value on the consistency of GOPATH, including across developers. I have seen others express similar sentiments. At least for me, I don't know if modules will ever provide 100% of a GOPATH-like experience, mainly because modules enable more choice. For example, even if an individual or a team chooses to place 100% of their modules together in some consistent location (and assuming things like this issue here #27542 is resolved in some nice way), their approach might turn into very much of a GOPATH-like experience for that individual or team (including consistency), but the fact that a different team might make a different choice seems to imply that retaining all the consistency that was delivered by GOPATH might be at odds with the flexibility that modules offer (including the flexibility modules offer to the people who state "I just want to clone a repo wherever I want on my disk").

But even if 100% of a GOPATH-like experience might not be possible, it might still be an interesting question as to how many of GOPATH's benefits can be preserved in a modules world...

@thepudds

This comment has been minimized.

thepudds commented Oct 25, 2018

A few more specific reactions:

@theckman wrote:

I also liked how easy it was to build with my own fork of a dependency if I have an outstanding upstream PR... or if the project is dead and has bugs.

Right now, it's probably fair to say that use case is fairly awkward with the core go tooling, but reasonably nice for at least one-off fixes with @rogpeppe's gohack (and which again gets to the subject of this issue #27542 in terms of dealing with multiple modules).

@theckman also wrote:

Lastly, I've seen people use it to be able to zip up the entire tree and ship that to another developer to help them troubleshoot a weird issue. They easily had the full working tree of the project.

and:

it's been an amazing experience having a language where I can finally write, or to provide instructions, that will work on any Go developer workstation with very little difference.

Those two comments are things that could be preserved with modules, but in part depends on how modules evolve. Perhaps those could be used as at least part of the criteria for evaluating the solution to this multi-module issue #27542.

@bcmills

This comment has been minimized.

Member

bcmills commented Nov 15, 2018

We don't have a plan for this, and there are more urgent modules issues for 1.13. Leaving open to collect ideas, but moving to Unplanned.

@spekary

This comment has been minimized.

spekary commented Nov 19, 2018

Following up from comments made to my post at #28868. My issue is not so much a build issue as described above, but how an IDE is supposed to support all of this, and I want to keep the IDE issue on the radar here. IDE's have bigger needs than the build. They do code completion, syntax checking, etc. The current module implementation which allows a particular module to override other modules depending on what directory you call the build tools out of makes the IDE's job difficult. The example situation I describe in #28868 is the most basic. Imagining a very active project with many modules in different stages of work, and with multiple executables, can make things quite difficult to keep straight.

The use case above seems to be covered by #27824 (comment), specifically the link to #26640

Somehow a developer will need to tell an IDE what go.mod file the developer intends to use for a build, so that the correct replace directives can be used to find the correct versions of all of the sources so that the IDE can then do its syntax checking and code completion magic.

Yes, that's true. (@myitcv calls that the “workspace”, if I recall correctly.) In general that should be the working directory in which the IDE or project was opened.

Both of these answers are making some assumptions about an IDE that are not necessarily true. What is this "working directory" in which the IDE was opened in a multi-project situation? Whether you have multiple modules in one workspace, or multiple workspaces, one for each module, the IDE still needs to know what go.mod file the developer intends to use in a particular build in case there are replace statements there that point to alternate sources.

Yes, and if we used environment substitutions, changing the environment would also change the effective module definitions. Fundamentally you're talking about making the build system rely on the environment (rather than the code) to specify which versions to use. That's what we had with GOPATH, and part of what we're trying to avoid with modules.

I agree with the goals. Its just that the replace statement implementation has recreated that problem. With modules, the build system now relies on the active go.mod file, which changes depending on the current working directory from which the build tools are called. Because of this, IDE's cannot track this in a many-module situation. One or two modules, no problem, so 90% of the time, its probably fine.

@spekary

This comment has been minimized.

spekary commented Nov 24, 2018

Leaving open to collect ideas

I was noodling on this, and not sure it solves all the problems listed, but here is a suggestion. However, as a preface, I think one issue is that go.mod is simply overloaded. I believe it was originally thought of as a better vendoring tool, and then it also became a replacement for GOPATH, and with the replace and exclude statements that only works for the top-level go.mod file, now its also a kind of build configuration tool. Its trying to do too many things at once.

Therefore, I think it makes sense to break it apart, creating an additional file, similar to the idea of the go.mod.local idea mentioned, but with the idea it would just have replace and exclude statements. However, instead of this being picked up by name convention like the go.mod file is, I think the go tool should be modified so that you can specify on the command line what the overriding go.mod file (or whatever it gets called), should be. That way the replace statements are not based on the CWD.

Also, if someone wants to still use a vendor directory, maybe something in there could specify this? Perhaps a replace directive that points to a top-level domain would do the trick?

In addition, since this is essentially a build configuration file at this point, it should have a mechanism to respond to GO's current build configuration mechanism, which are build tags. So, something like this:

// vendoring
replace github.com => vendor/github.com

// +build debug
replace github.com/A/B=> ../B-src

// +build !debug releasetest
replace passwords=> ../release-passwords

Something like that.

Advantages:

  • The current go.mod file will work the same
  • You can check in the file. Since it responds to build tags and only works if you specify it with the command line, you can architect it so that the command-line controls the build and you don't have to worry about developers accidentally checking in a bad go.mod file.
  • IDEs can ask for the location of this file, and then they don't have to worry about the changing view of the world based on the location of the file being edited.
  • vendoring tools can still work the same as they do today. Its just that to turn on vendoring, the developer will need to spell that out in this build-config file.

Anyways, its a start of an idea we can poke at.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment