Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: cmd/go: introduce 'module GOPATH' to allow transition from GOPATH to modules #44649

Open
bcmills opened this issue Feb 26, 2021 · 29 comments
Labels
Projects
Milestone

Comments

@bcmills
Copy link
Member

@bcmills bcmills commented Feb 26, 2021

In versions of Go before modules, users were able to set up a single GOPATH containing a group of interdependent packages. With the advent of modules, a single module may contain multiple interdependent packages. However, all of those packages must share the same import-path prefix, which is the module path declared in the main module's go.mod file.

For some users — especially hobbyists who did not publish their code for external use — the process of migrating from this “global” namespace to the “local” namespaces of modules can be burdensome. (See, for example, the various discussions in #37755 and #44347.)

When we started using modules to manage the vendored dependencies of the Go standard library itself (#30241), we created a special std module which, unlike all other modules so far, does not add an import-path prefix to the packages it contains.

It recently occurred to me that we could use this same approach to allow “tinkering” users to more easily convert their GOPATH workspace to work in module mode.

For such a module:

  • The go.mod file would explicitly indicate that the user wants a module with no import-path prefix, perhaps by declaring a distinguished module path (module GOPATH or module src?) or using a special directive (noprefix?).

  • Within an unprefixed module, the usual module-mode commands, including go mod tidy and go get, would be supported.

    • go get would add module dependencies as in module mode, not clone repos into the local module as in GOPATH mode.
  • No other module may itself require an unprefixed module. (This is to preserve the invariant that a user can identify which modules may contain a given package by comparing the package path to the module path.)

  • As in an ordinary module, collisions between packages defined in the unprefixed module and packages defined in the module dependencies of that module would not be allowed.

    • However, because we know that no other module may require an unprefixed module, we could relax this restriction in the future. Packages definitions in the main module would always take precedence over package definitions in module dependencies.

CC @jayconrod @matloob

@bcmills bcmills added this to the Proposal milestone Feb 26, 2021
@robpike
Copy link
Contributor

@robpike robpike commented Feb 26, 2021

This is clever and would have helped me a number of times.

@robpike
Copy link
Contributor

@robpike robpike commented Feb 27, 2021

See #44660

@thepudds
Copy link

@thepudds thepudds commented Feb 27, 2021

Hi Bryan, I like this idea, and I very much like the general concept of a gentler on-ramp to modules and import paths and so on. One thing to consider is a graduation mechamism. Ideally, cmd/go would rewrite the import paths and module path in a consistent manner. In other words, something like start via go mod intro, and later go mod intro -graduate <module path> (though not trying to suggest a specific spelling or trigger bikeshedding).

It would be nice if cmd/go could cross the import path editing rubicon (which might also justify a cmd/go version of #32014), but if not, some gopher from the broader community would almost certainly create some flavor of a go-mod-graduate or similar.

If some version of what you outlined was supported, I would likely recommend it as a starting point to some of my colleagues who have written plenty of Go, but who still fumble around when starting a module from scratch. An automated graduation path in turn would help them transition if needed to a working, consistent full-blown module, which is then much easier to expand upon.

@bcmills
Copy link
Member Author

@bcmills bcmills commented Mar 1, 2021

@thepudds, I think we already have a graduation mechanism: namely, carving out “nested” modules, where in this case a “nested” module is any module with a non-empty import prefix.

That is: at any point, you can carve out any subpackage into a module by... making it its own module, and adding a require directive to point the prefixless module to that nested module. (The only constraint on that refactoring is that the new (prefixed) module itself may only require other modules that have prefixes, so the process of converting individual paths to modules has to begin with modules containing the “leaf” packages.)

@mvdan
Copy link
Member

@mvdan mvdan commented Mar 1, 2021

I used to do the same kind of GOPATH tinkering and I've got used to module test pretty easily (#37641). Paths like test/foobar are not too long compared to foobar, and they're still clearly not for external use or publishable. Do we need to add a special case to avoid a test/ prefix?

@icholy
Copy link

@icholy icholy commented Mar 1, 2021

This almost seems like a re-introduction of the previous vendor behaviour.

@bcmills
Copy link
Member Author

@bcmills bcmills commented Mar 1, 2021

@icholy, the crucial difference between this and GOPATH-mode vendor is that under this proposal each package still has only one definition (used by the whole module), rather than potentially one per importer.

(In module mode, there is only one vendor directory for the whole module, not one per package subtree as in GOPATH mode.)

@icholy
Copy link

@icholy icholy commented Mar 1, 2021

The go.mod file would explicitly indicate that the user wants a module with no import-path prefix, perhaps by declaring a distinguished module path (module GOPATH or module src?) or using a special directive (noprefix?).

Why not just omit the module name?

module

go 1.16
@bcmills
Copy link
Member Author

@bcmills bcmills commented Mar 1, 2021

Maybe! But is that too difficult to distinguish from an accidental typo? (I'm honestly not sure.)

@icholy
Copy link

@icholy icholy commented Mar 1, 2021

Actually, that won't work. Older versions of Go will choke before getting to the go 1.N directive.

go: errors parsing go.mod:
/home/icholy/src/go.mod:1: usage: module module/path

edit: but I guess that wouldn't matter because it can't be required.
edit: re: typo, I can see it being a footgun.

@fzipp
Copy link
Contributor

@fzipp fzipp commented Mar 1, 2021

Maybe! But is that too difficult to distinguish from an accidental typo? (I'm honestly not sure.)

module _

go 1.16

maybe?

@bcmills
Copy link
Member Author

@bcmills bcmills commented Mar 1, 2021

Older versions of Go will choke before getting to the go 1.N directive.

That doesn't particularly matter, because older versions of Go won't know how to interpret a prefixless user module anyway. 😅

@icholy
Copy link

@icholy icholy commented Mar 2, 2021

warning, heavy speculation:

If this gets accepted/implemented, prefix-less mode will become the preferred choice for non-library modules. Once/if that becomes the case, people will also want to write prefix-less library code. This will become a point of contention.

@rsc
Copy link
Contributor

@rsc rsc commented Mar 3, 2021

If we do this I think it should probably be the suggested way to migrate from GOPATH to modules incrementally, at which point module GOPATH seems like the right name. But then it should also apply the GOPATH-mode-specific "vendor import rewrites" (https://golang.org/s/go15vendor). And then GO111MODULE=off mode would maybe transform to an imagined immutable module GOPATH go.mod in GOPATH/src. That would provide a way to keep GOPATH mode around for improved compatibility without holding modules back.

One detail I don't quite understand is what to do about imports of code from GOPATH that is itself covered by other go.mod files. For example if we have

GOPATH/src/go.mod: module GOPATH, no mention of example.com/b
GOPATH/src/a/a.go: imports "example.com/b"

then it seems clear that the import should resolve to GOPATH/src/example.com/b/b.go.
But what if GOPATH/src/example.com/b/go.mod exists?
It seems like we just ignore it?

@ohir

This comment was marked as off-topic.

@bcmills
Copy link
Member Author

@bcmills bcmills commented Mar 3, 2021

@ohir, this proposal is not intended to address simultaneous editing of multiple modules. That will need to be addressed separately.

This proposal is specifically intended to address module migration for existing projects (large or small) that rely on packages that, for whatever reason, already cannot be fetched using go get, and therefore cannot be published or consumed as modules using go get.

That may be because they are hobby programs, or programs that generate additional code using make or bazel or similar third-party build tools, or just large private codebases that never needed to be accessible to go get.

@ohir
Copy link

@ohir ohir commented Mar 3, 2021

@ohir, this proposal is not intended to address simultaneous editing of multiple modules. That will need to be addressed separately.

Ah, indeed. I humbly apologize for my off-topic comment above.

@bcmills
Copy link
Member Author

@bcmills bcmills commented Mar 3, 2021

what if GOPATH/src/example.com/b/go.mod exists?
It seems like we just ignore it?

Hmm, good question. Normally we would prune out the nested module, but if we're applying the legacy GOPATH-mode vendor behavior, then I think we would also need to ignore go.mod files found within the GOPATH module (and pull those packages into the GOPATH module itself).

I'm tempted to suggest that we wire in nested modules using replace directives instead, but I think that would make the vendor behavior too complicated.

@rsc
Copy link
Contributor

@rsc rsc commented Mar 3, 2021

Hmm, good question. Normally we would prune out the nested module, but if we're applying the legacy GOPATH-mode vendor behavior, then I think we would also need to ignore go.mod files found within the GOPATH module (and pull those packages into the GOPATH module itself).

I think I agree.

If GOPATH/src/go.mod says 'module GOPATH' and GOPATH/src/example.com/b/go.mod says 'module example.com/b' and I cd into 'example.com/b' and run a go command, then I assume it uses example.com/b/go.mod and not GOPATH/src/go.mod? That is, the command runs in the example.com/b module and not the GOPATH module?

@bcmills
Copy link
Member Author

@bcmills bcmills commented Mar 3, 2021

Yes, I think that's correct. If you want to run a command in the GOPATH module, you need to be outside of any other (nested) module.

Changes to the source code found in of any of those nested modules would be reflected in the GOPATH module itself, but changes to their module requirements would not be. That's maybe a bit confusing, but not really any worse than switching in and out of GOPATH mode today.

@rsc rsc changed the title proposal: cmd/go: reserve a distinguished module path for a user module without an import-path prefix proposal: cmd/go: introduce 'module GOPATH' to allow transition from GOPATH to modules Mar 10, 2021
@rsc rsc moved this from Incoming to Active in Proposals Mar 10, 2021
@rsc
Copy link
Contributor

@rsc rsc commented Mar 10, 2021

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@meling
Copy link

@meling meling commented Mar 11, 2021

I for sure don’t understand the full extent of this issue, but I think modules is already very complex, and this seems to add more complexity with yet another variant... I think it is a mistake to add more features to the standard tooling to support legacy style.

This issue is about transitioning to modules. Would it not be possible to write a tool to assist with conversion to modules instead, perhaps with some manual steps?

@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Mar 11, 2021

I think what I like about this proposal is that it lets people who are used to GOPATH continue working with a GOPATH-like workflow within modules if their projects aren't dependencies for other modules. I imagine that covers a lot of small projects and examples.

I don't think this can realistically serve as a compatible replacement for GOPATH. There are too many small differences to emulate: multiple GOPATH directories, multiple vendor directories, ignoring go.mod files, minimal module compatibility, relative imports, to name a few.

@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Mar 11, 2021

This issue is about transitioning to modules. Would it not be possible to write a tool to assist with conversion to modules instead, perhaps with some manual steps?

@meling That mostly exists as go mod init followed by go mod tidy, though it may be necessary to add major version suffixes in imports. That generally works for an individual project though, not all the projects collected within a GOPATH; it would be difficult to automate that in a way that would work for everyone.

@bcmills
Copy link
Member Author

@bcmills bcmills commented Mar 15, 2021

A general conversion tool would also have to deal with import paths that cannot be fetched via go get. If each top-level subdirectory of GOPATH/src has to be its own module and a package in one subdirectory imports a package from another, then they have to be stitched together with a replace and a require directive. In the general case, that may entail up to O(N²) replace directives for N top-level subdirectories.

In contrast, if GOPATH/src itself can be a module, then no replace directive is needed to allow packages in one subdirectory to import packages in another.

@benitogf
Copy link

@benitogf benitogf commented Mar 19, 2021

I for sure don’t understand the full extent of this issue, but I think modules is already very complex, and this seems to add more complexity with yet another variant... I think it is a mistake to add more features to the standard tooling to support legacy style.

Agree on this, but I wouldn't say that GOPATH is legacy I think that they are just different modes of building and maintaining a Go environment, In our team we use both go get for libraries and code that's not available through go get with ease, there are no makefiles and no go.mod files, I also keep public libraries where I do use go modules.

The issue to keep GOPATH has been closed and there's a blog post announcing the removal of it by 1.17, but if this proposal exists and #37755 has so many comments (and deleted comments) it would seem that there are more than a few people who prefer the functionality of GOPATH rather than modules.

We have a way of dealing with this, the ON/OFF switch for modules (GO111MODULE), which I would urge you to keep, not only so people can keep using GOPATH if they want to but also to keep modules as its intended and not trying to emulate what GOPATH does so well.

@rsc
Copy link
Contributor

@rsc rsc commented Mar 31, 2021

I think the big question here is whether it's worth putting any new effort into GOPATH mode when so few people are using it at this point (96% on modules).

@benitogf
Copy link

@benitogf benitogf commented Apr 1, 2021

I think the big question here is whether it's worth putting any new effort into GOPATH mode when so few people are using it at this point (96% on modules).

if we take percentages from this issue it doesn't reach 96% actually most people where asking for it not to be removed, as I said before I use go modules, so I'm in that 96% but I also use GOPATH and I think that many people still do as well and would like to continue having an option to build Go code without dealing with a package manager.

@complyue
Copy link

@complyue complyue commented Apr 10, 2021

I'd like to add a non-hobbyist use case for the demand, that I used to develop several projects at the same time - a few reusable libs plus 1~3 apps using them, translated to go.mod setup, I'll have to maintain local replaces in several go.mod files and take special care to not commit those replaces to shared repos, how burdensome!

Personally I like #44347 GOTINKER the most among alternative proposals by far. As for the special go.mod syntax with special semantics as proposed here, I feel it kinda confusing against simpler intuition with the go.mod construct. I wonder how the std trick was working, and why not to make similar practice public doable?

I would like to suggest a separate go.project or go.home file as a solution, with syntax and semantics on its own right. But I'm not developing with Go for 1~2 years or so, my knowledge and memory around Go seems outdated a lot, and don't have time for a full proposal.

Update: Maybe go.farm is a sensible name, for the metaphor of a place where we can grow multiple Go modules at the same time, while ultimately they are to be transported to the market (public repositories & registries).

Shame on me and it's just my 1 cent.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Linked pull requests

Successfully merging a pull request may close this issue.

None yet