cmd/go: modify the Go toolchain to work without GOPATH

[rasky]

Over the years, when helping people approaching existing Go projects (colleagues, friends, etc.) the number one problem is of course that they don't know about GOPATH, they don't have one configured, and they expect to be able to clone an existing project wherever they want on the disk, and be able to build it. I think the problem statement is clear and the problem is well known.

With the introduction and adoption of the vendoring folder, most Go applications do not even require go get to run, after the initial clone: all the source code required to compile them is already available in the source tree; it's just that the Go tool doesn't know where to look if GOPATH is not defined.

Instead of suggesting a default GOPATH (which does not solve the fact that people will need to find out about it and use it somehow), I suggest that we change the Go toolchain in multiple steps (each one can be independently released as an experiment):

• STEP 1: If GOPATH is not set, the Go tool will use any existing ./vendor directory as it was part of the GOPATH. (I think of this as defaulting GOPATH/src to ./vendor). At least go build and go run are guaranteed to work. Anything else, like go install and go get, would fail with an error pointing to the GOPATH documentation.
• STEP 2: if GOPATH is not set, the Go toolchain would be updated to use $HOME/.gopkg (or similar) as pkg directory. This would allow other commands to work (e.g.: go build -i or go install).
• STEP 3: if GOPATH is not set, go get (or the new package manager, if/when it gets released) would be modified to download packages directly into ./vendor

As shown in Step 2, I think this proposal would also interact well with the existing proposal/discussion of dropping pkg in favour of a hidden cache directory, as that would move things further into the direction of not needing GOPATH anymore for a whole class of development; in fact, this functionality could be prototyped in this specific scenario of "GOPATH not defined", where it would make sense to do it without facing the bigger discussion of deprecating pkg for a normal GOPATH scenario.

As shown in Step 3, I also think that this might interact well with the new package manager. I saw many people requested that the new package manager ought to be able to download dependencies directly into the vendor folder (like govendor and glide can do); if this functionality lands in the new package manager, it would be a perfect default for the "GOPATH not defined" scenario described here. Alternatively, it might make sense to discuss modifying go get to have this behaviour.

I think this proposal is incremental, backward-compatible, doesn't affect existing users that successfully use GOPATH, and allow beginners to approach existing Go codebases without stumbling too soon into GOPATH.

[davecheney]

If you don't want to use $GOPATH, there's always gb :)

But seriously, in my experience the problem newcomers have with $GOPATH is not its prescribed namespace layout, but the fact that they have to choose a location.

Using the go tool does have a learning curve, that is undeniable, but I believe the go tool, and the conventions of laying out source in the $GOPATH form aren't hard to learn, and once people understand the rationale behind the $GOPATH layout there is general approval.

My proposal in #17262 seeks to defer the additional cognitive load of choosing a $GOPATH value by choosing a reasonable one for the user. Users still need to understand how $GOPATH works, because that is also how import statements work in Go code because the compiler interprets an import statement as a path rooted at some location, that being a segment of the users' supplied $GOPATH. But that additional learning can take place after they've been able to write small Go programs and go get code from others.

[rasky]

But seriously, in my experience the problem newcomers have with $GOPATH is not its prescribed namespace layout, but the fact that they have to choose a location.

I respectfully disagree. I never had somebody complain to me that choosing GOPATH was difficult per-se (as in, how to call the directory). Their main objection is that there's no good GOPATH location because they usually have projects layed out as ~/Sources/prjname, and there is no good way of matching this layout with that imposed by GOPATH. Choosing a default GOPATH would swap one confusing error message with another. If you go with defaulting GOPATH with something, they would still have to realise that:

• There is a thing called GOPATH
• That thing has a default (that might point to a directory they don't even have)
• Checking out their project of choice in $GOPATH/projectname will still not work
• Checking out their project of choice in $GOPATH/src/projectname might still not work (in case the project refers to one of its subpackage as github.com/whatever/projectname/foo).

My proposal removes all four hurdles above; it erases the whole cognitive load associated with GOPATH for the purpose of a clone+modify+push+pull-request cycle, and thus lowers the barriers of entry much more.

Also, if we look at a modern Go application, with all its dependencies correctly vendored, I really can't see what GOPATH is buying to a user that's just trying to edit and recompile such application. Every source file that must be built is already there; it looks overly aggressive to ask them to shuffle their directories around and set environment variables just to be able for the Go tool to find a library that is stored in the official vendor subfolder. It did make more sense when they had to go get to fetch the dependencies, but that step is not so common anymore.

[dmage]

But which import path supposed to use for the project's packages if there is no GOPATH?

[sdwarwick]

The GOPATH issue has been a real stumbling block. We have multiple projects for multiple clients, each project uses a multitude of different languages and frameworks. They are all in different directories, different volumes etc.. We want all code associated with a specific project to be separately managed. It has been very difficult to see how the GOPATH architecture supports this. It is certainly not documented in the standard "getting started" information.

[rasky]

@dmage I think it should just be "." or whatever it is called the top-level path in the import hierarchy. This would mean that, to work without GOPATH, projects including sub-packages will have to refer to them with a relative path like "./foo". I think it's a reasonable compromise for projects that wish to build without GOPATH.

(a possibly crazy idea is to also use canonical import path for this, that is a way of teaching the Go toolchain the path of the current project being built in a no-GOPATH scenario)

[ngrilly]

Their main objection is that there's no good GOPATH location because they usually have projects layed out as ~/Sources/prjname, and there is no good way of matching this layout with that imposed by GOPATH.

As I work on several projects using Go alongside otherside programming languages, this is exactly my experience.

[kokes]

Since #17262 is going forward, the "if GOPATH isn't set, use '.'" can no longer apply. But a slight change to "if outside the GOPATH, use '.'" would do the trick - mimicking gb's behaviour and allowing people to just download/copy a piece of code and compile it then and there (possibly with a friendly warning message, that the user is outside the GOPATH).

Then again, this sounds an awful like #12488, which was declined.

[adg]

Regarding the resolution of #17262: while this proposal takes advantage of an unset GOPATH, I don't see that as a key element. It is still possible to take advantage of a ./vendor outside an explicit or implicit GOPATH.

[rasky]

Well, yes, I guess it's possible to adapt a part of this proposal with a default GOPATH (though STEP3 probably will be impossible to implement, for instance), but I think this proposal (e.g. in this comment: #17271 (comment)) makes a point that a default GOPATH isn't really helping, and instead defines a different, more comprehensive workflow to help beginners work with Go without having to learn about GOPATH from the get go. If the decision on #17262 is final (which I consider very unfortunate, as I think this proposal is a superior and more comprehensive solution to the problem statement), I'll have to revisit this proposal.

[justinfx]

I have had to deal with the problem that this proposal offers to solve. That being, making it easy for the non-go developers at my company to be able to check out a Go project to a random location and build it.

Because I can't expect every developer to have a GOPATH set up, or to clone to a specific location, I have needed to design Go logic into our waf-based build system. This has to try and do extra smarts such as forming a GOPATH on the fly if it can. Telling them to use gb is not an option for multiple reasons. They would need a special tool as a non go developer, and gb takes its own approach to the vendor directory (last I checked) in having vs not having a src subdirectory.

It would be fantastic if the standard go tool could offer sane default behavior for trying to build a project that is outside gopath, before failing. If it at least tried to assume the project is self contained with a vendor directory, it would have a chance to build.

Isn't this following the same goal as #17262 in having default behavior that makes it easier for beginners?

[davecheney]

The go tool requires GOPATH, it's built into its design and backed by its
interpretation of the import statements. They are inseperable.

What is it specifically that you don't like about gb? It seems like its
design of a self contained project structure is what you need. How can I
help make gb a better fit for your requirements?

On Wed, 26 Oct 2016, 05:42 Justin Israel notifications@github.com wrote:

I have had to deal with the problem that this proposal offers to solve.
That being, making it easy for the non-go developers at my company to be
able to check out a Go project to a random location and build it.

Because I can't expect every developer to have a GOPATH set up, or to
clone to a specific location, I have needed to design Go logic into our
waf-based build system. This has to try and do extra smarts such as forming
a GOPATH on the fly if it can. Telling them to use gb is not an option for
multiple reasons. They would need a special tool as a non go developer, and
gb takes its own approach to the vendor directory (last I checked) in
having vs not having a src subdirectory.

It would be fantastic if the standard go tool could offer sane default
behavior for trying to build a project that is outside gopath, before
failing. If it at least tried to assume the project is self contained with
a vendor directory, it would have a chance to build.

Isn't this following the same goal as #17262
#17262 in having default behavior
that makes it easier for beginners?

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
#17271 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAcAwmMhO6OiiB5eMHxl36Y-d_we155ks5q3k2ogaJpZM4KJZPO
.

[justinfx]

What is it specifically that you don't like about gb? It seems like its
design of a self contained project structure is what you need. How can I
help make gb a better fit for your requirements?

It is the very fact that gb diverges from the official standard, that being the go tool. Problem is that I cannot force a developer to build their Go projects with gb, so that I can then assume I can use gb to build it later, in our build system layer. The most likely path is that a developer will use the official go tool and structure their project (including the vendor directory layout) to match. Then I have to make sure any other developer can clone that project and build it, regardless of where they clone it. To use gb, I would have to make it a standard that everyone follows.

[davecheney]

On Wed, 26 Oct 2016, 06:26 Justin Israel notifications@github.com wrote:

What is it specifically that you don't like about gb? It seems like its
design of a self contained project structure is what you need. How can I
help make gb a better fit for your requirements?

It is the very fact that gb diverges from the official standard, that
being the go tool.

That's fair, but I would suggest that the chances of the Go tool making
GOPATH optional are about the same likely hood as gb acquiring some
official status.

The important thing for me when I built gb was the realisation that the go
tool isn't the lanaguge or the compiler, it's just a driver that feeds
files to the compiler in the correct order, so in that respect gb is a peer
to the go tool.

Problem is that I cannot force a developer to build their Go projects with

gb, so that I can then assume I can use gb to build it later. The most
likely path is that a developer will use the official go tool and structure
their project (including the vendor directory layout) to match.

Yeah, but is it fair to say that it doesn't work for you, or them,
otherwise we wouldn't be having this conversation?

Then I have to make sure any other developer can clone that project and

build it, regardless of where they clone it. To use gb, I would have to
make it a standard that everyone follows.

Gb is only for projects, applications that produce binaries. It has no
story for library developers, but in saying that, the cost in terms of
perceived comparability, I feel, is not as strong when you approach the
problem on a application by application basis.

—

You are receiving this because you commented.

Reply to this email directly, view it on GitHub
#17271 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAcA5FC2Ia4SdHd28Q1Fjazln37huDtks5q3lf1gaJpZM4KJZPO
.

[justinfx]

Yeah, but is it fair to say that it doesn't work for you, or them,
otherwise we wouldn't be having this conversation?

I wouldn't say gb doesn't work on its own. Its more a case of gb not being able to be suggested as the solution to the problem being proposed here. I was just trying to make a case really for the go tool to adopt some of the concepts of gb, by making the zero value of the go tool more useful.

The most common scenario would be that people are following the official Go project structures and expect projects to be able to be built with the go tool. So with that being said, it would be amazing if the go tool could cope with more situations and try harder to build before failing. Just like gb is only for projects, applications that produce binaries, the go tool could also try to ensure that an application can be built regardless of GOPATH not being set, if it can do so by assuming some defaults.

[ironiridis]

It doesn't seem like these two proposals are in conflict. #17262 proposes a mechanism by which GOPATH can be inferred when the environment variable is missing. This proposes an algorithm for that inferred path. In fact it would seem this proposal actually depends on #17262.

[bradfitz]

Let's see how much of the problem is solved by automatic GOPATH=$HOME/go in Go 1.8 and revisit this in a release or two.

[lucsky]

@bradfitz I really don't see how an automatic GOPATH solves anything, this proposal is about the possible ability to do without the GOPATH.

[bradfitz]

I understand the difference. Hey, listen, I don't want GOPATH either. But I'm also realistic about how fast things move. I'd love to be see it happen sooner than later. Personally I'd like something like git where the "go" command would walk up the tree, looking for a magic directory that's the GOPATH. In Git, that's the .git directory. The Go project avoids dot files for the most part, and we're not going to hide user's code. I'd like to see a magic gopath directory that contains the gopath.

We also need to fix #4719 first and get rid of the $GOPATH/pkg directory entirely, making all build artifacts aggressively cached. That would fix another few open issues and common user confusions too.

Then we you could have structures like:

$HOME/my-projects/fooproj/cmd1
$HOME/my-projects/fooproj/cmd1/main.go
$HOME/my-projects/fooproj/cmd2
$HOME/my-projects/fooproj/cmd2/main.go
$HOME/my-projects/fooproj/gopath
$HOME/my-projects/fooproj/gopath/github.com/you/bar/bar.go
$HOME/my-projects/fooproj/gopath
$HOME/my-projects/fooproj/gopath/vendor/github.com/

And if you were in the "cmd2" directory and ran go install, it would walk up, file "gopath", and prepend that to your $GOPATH environment, which will likely be empty for new users in a GOPATH-less world.

"go get" and everything would work.

"go install" would need a bin directory to write to. $GOBIN exists and would be used first. If unset, it could be write to $GOPATH/bin. (yes, there's no gopatch/src in this proposal. Maybe it could exist, but it seems unnecessarily deep) Or we put the default $GOBIN elsewhere.

Anyway, the point is that I also don't like $GOPATH.

It's just a lot of design & implementation work, and nobody's on it, at least yet.

[bradfitz]

Or, a revised version of the above proposal:

Instead of the magic directory being gopath, instead make the magic directory be named src. When walking up the filesystem to the root, whenever /foo/bar/src is found, /foo/bar is prepended as a $GOPATH.

Then the implicit $GOPATH/bin would be $YourProjectRoot/bin, next to your src directory. Then you'd just .gitignore the bin directory, as is typical already.

So:

$HOME/my-projects/.gitignore          # with bin
$HOME/my-projects/fooproj/cmd1
$HOME/my-projects/fooproj/cmd1/main.go
$HOME/my-projects/fooproj/cmd2
$HOME/my-projects/fooproj/cmd2/main.go
$HOME/my-projects/fooproj/src
$HOME/my-projects/fooproj/src/github.com/you/bar/bar.go
$HOME/my-projects/fooproj/src
$HOME/my-projects/fooproj/bin/cmd1   # binary from 'go install' in the cmd1 dir
$HOME/my-projects/fooproj/bin/cmd2   # binary from 'go install' in the cmd2 dir

And there would never be a pkg directory, because we'd do #4719 as a dependency, which would replace pkg with a build artifact cache directory.

[kardianos]

@bradfitz I think using src as the magic would be work. I would propose something similar, but slightly different. We specify the root folder's virtual location with somehow*. We still install bin (and until pkg get's sorted out, pkg too) locally in this root.

$HOME/my-projects/fiiproj/special-file.json < "specifies github.com/you/bar"
$HOME/my-projects/fiiproj/cmd/cmd1/main.go < "imports github.com/you/bar/bears"
$HOME/my-projects/fiiproj/bears/black.go
$HOME/my-projects/fiiproj/bin/cmd1
$HOME/my-projects/fiiproj/pkg/... < until this can go away, have it live here
$HOME/my-projects/fiiproj/vendor < optional can still have vendor folder

This can live in tandem with existing $GOPATH.
Idea from here: https://docs.google.com/document/d/1GiSDQHFo5YfYMN5TBaYdjfWHI3KDyfaU1kTHzcLR2qE

In Java / C#, I also often expect a local "bin" and "obj" or similar directory, so I wouldn't make #4719 a dependency, just when that gets resolved, the pkg dir would no longer be needed. The local "bin" would stay.

A dependency of this would probably be a standard "go dep" tool to put the root path specifier.

A small side note, govendor and godep write down the path in their manifest files today so services such as heroku can build the project with just the vcs repo.

[mcluseau]

@bradfitz #14566 is not a duplicate of this... it's the case where we (may) have a GOPATH but want to build something from outside of it. This issue won't solve it at all. To solve #14566, we need to be able to specify the package we are in (for instance with a "--local-package=foo/bar" flag) and have everything else work by default (including the vendor directory).

[bradfitz]

It might not be an exact duplicate, but the issues are all closely related. Using this as the tracking issue is good enough.

[mcluseau]

Ok so now this issue also includes solving the need to override the local package's "reference" when outside the GOPATH. From the top of my head, 2 proposals where made: the flag solution I just put and the "godoc" like comment solution (ie: package quota // import "k8s.io/kubernetes/pkg/quota").

[dlsniper]

That's the sticky widget isn't it?
Transitive dependency management is the bane of many systems that have come before (maven, npm) In theory A can use B and C (release 1.2), and B can also use C but a just different version of it (HEAD). Oops.

I'm fully aware of the diamond problem, thanks for the oops right there :)

Yes, but if you also read what I've wrote, it shouldn't be up to go get to solve that.
The initial version can do whatever the current go get does so the problem is solved, the behavior is the same.

Should we also discuss about the complexity of resolving all dependencies in automatic way?
Let's do that in the next section.

I guess the question is - can/should a Go application be allowed to use multiple versions of a library? It seems like the answer should be no, but then how do you solve that?

In my opinion, no. That's a can of "goodies" that it's best left unhandled by automated tools in a "smart" way.

Again however, this shouldn't be a problem that go get shouldn't solve, at least not in an initial release.
This should be solved via the metadata file(s) present in the repository which will tell the tool go get tool to fetch the exact versions that you need. Will this tool be the same as go get? Maybe, I don't know.

We also have two problems here:

• people version their repositories and follow semver religiously
• people version their repositories and don't follow semver even remotely religiously

If the first thing happens, then you are lucky. But it's highly likely that a similar situation as the one you are describing, where a small change that technically doesn't effect the API but the behavior of the package.

For this and the latter case where a user doesn't follow semver practices you really have a single solution: ask the user to solve the conflict issue.
It's really the only way to guarantee to the user that the package that is in a version conflict state can be used.
Sure someone might say: but we can write tools for that. I agree, at some point we might have better tools that detect these things, but are you willing to wait until they get created and tested and proved to never be wrong?

We could argue for the third case where people don't version their repos but then we can also say: just fetch the HEAD and let the users complain about the missing versioning.
This will create enough pressure on the maintainers to start being better at maintaining their packages, which is totally ok.

This seems highly related to the behavior of the /vendor folder.

It's a problem today for get as well, with the GOPATH only, not just the vendor folder.

I'm sure there's someone that will soon point out even more issues, hypothetical or real, but again, I urge you, please take a couple of steps back and ask yourself:

• What's important? Having a way to work with an application focused workflow? Or keeping compatibility with GOPATH?
• (or in a rewording of the above): - What's important? Having a way to get the dependencies for your application in the vendor folder? Or having a way to have go get compatibility for applications?
• What's important? Having a way to cover 80%-90% of the cases out of the box and provide the users a way to reliably say go get ./... in a project root directory and fetch the dependencies at their expected, predefined versions? Or solving a lot of the corner cases that appear along the complex subject that versioning is, in an automatic way?

I'm not saying effort shouldn't be put into trying to solve as many issues as possible and make the life of the users as good as possible.
But please remember that Go itself doesn't have this approach to solve problems and we can approach this in an incremental manner while still offering something stable and usable to the users at all times.

[lloeki]

Coming from the ruby world and having dabbled with node, my personal practical experience as well as theoretical belief totally concurs that using a single version of a lib is the way to go. Go has the added benefit that an incompatible version has a greater chance of being detected statically at compile time compared to dynamic languages which have to rely on tests. If something arises from detecting an incompatible version whether through semver, static analysis, or tests, then it means some mess needs to be sorted out, and the node way is just trying to sweep it under the rug, only adding complexity to the table when (when!) you have to untangle object lifecycle through multiple versions of the same lib that may have subtly different expectations of data structures as well as behaviors.

[dlsniper]

@rsc this is the issue we've talked about at GopherCon.

I believe that the proposal to have a "virtual GOPATH" created based on the vendor/ directory presence (and probably the golang/dep manifest/lock files).

In my simplistic view, this could be done for Go 1.10 but I defer to your judgement if this could be approached in the proposed format, or otherwise, by then.

Thank you.

[rasky]

I think that it would be a good time to revisit this proposal that was put on hold 15 months ago (/cc @bradfitz @rsc).

My experience hasn't changed much since I opened this proposal, and the default GOPATH hasn't helped a bit, as I was describing in this comment. Given also the new cache (that was created outside GOPATH, and basically cancels the needs that were described in STEP2 in the proposal), we're really close to being able to let users play with a Go project (with vendored dependencies) without having a GOPATH set, that is STEP1 in my original proposal.

[bradfitz]

Unholding.

[target-san]

What I'd suggest may have been already proposed. Anyway

1. Make go tools respect and use per-user directory, like $HOME/.gocache by default
2. All packages pulled via go get land automatically into $GOCACHE/src/<packagepath>. Some versioning prefix or suffix may be added to package's path - disputable.
3. To build project, user needs to pull its sources and then go build, which will involve:
4. go dep ensure automatically and place all deps into source cache, as defined above
5. Actual go build which will look for dependencies in cache folder, not in $GOPATH/src or ./vendor
6. Make $GOPATH deprecated
7. go install may actually deploy binaries to $GOCACHE/bin for convenience
8. ...

This way, each project is self-contained and can be built right from sources, without managing dependencies and their locations manually

[kardianos]

This issue will be resolved should vgo or similar solution be adopted. The need for a GOPATH env var goes away when adding a go.mod file with the module root path.

[stevenroose]

@target-san

3. go dep ensure automatically and place all deps into source cache, as defined above
4. Actual go build which will look for dependencies in cache folder, not in $GOPATH/src or ./vendor

Different projects can have different versions for packages. I would strongly suggest to have dep ensure still use a project-specific cache: either ./vendor/ or something like $GOCACHE/build/$PROJECT/. This way, project-specific package sources go first, and then the regular Go cache (current Go path) is used.

[mathstuf]

Different projects can have different versions for packages.

Could the cache be keyed on the name and version of the package rather than just the name?

[DanielHeath]

Given that they're just files (and therefore could have been modified), I'd suggest keying on the hash of the package source instead of any surrogate key.

[devnev]

Hey all,

My experience covers work both inside a mono-repo and with individual libraries, with either fully vendored dependencies or with all the dependencies already inside the repository. The extra GOPATH structure has added little value while making it annoying to set up and manage clones of the repository under development.

I’ve been experimenting on implementing some of the changes necessary to work without a GOPATH (in a repository with fully vendored dependencies) to see what it might entail. As far as I can tell, this change is comparably small and localised, although I didn’t implement a complete solution.

A possible way to allow users to use the build tools without a GOPATH might be to walk up the filesystem from the starting directory until a source file with a specific directive (e.g. //go:prefix github.com/foo/bar) is found. If this is the case, the directory containing that source would be the root of a GOPATH-less source tree. This tree is placed in the import namespace at a prefix specified as part of the directive.

I think this allows new GOPATH-less-capable repositories to be fully compatible with older toolchain versions (the prefix directive can be ignored if it is located in a directory below a GOPATH root). Also, because the prefix scan occurs just once and picks the first prefix it finds, ambiguity from having multiple such prefixes is avoided and minimal build overhead is added. Because the source tree behaves as if placed inside a virtual GOPATH root, most of the build behaviour continues to work unchanged, and import statements do not need any adjustment.

Disregarding the particular details though, it would be great to see some movement on this issue.

[kardianos]

@devenv and any others: @rsc is currently working on a proposal that will not only take care of this issue (completly remove the need for GOPATH), but also take care of making imports version aware.

His papers on the subject: https://research.swtch.com/vgo
His placeholder proposal: https://golang.org/issue/24301

Right now his proposal issue is empty and used as a placeholder for drafting the proposal text CL.

[rsc]

We're certainly moving toward "modify the Go toolchain to work without GOPATH", so accepting that idea, although not the details in the top post above. It will take a few releases to get there. The steps are

1. Add modules.
2. Stop writing anything to GOPATH/pkg (aka cmd/go: drop $GOPATH/pkg #4719; implies disallowing GOCACHE=off or at least making that very slow).

We're not ready to do 2 yet.

[rsc]

This is really just blocked on #24301 and #4719. After they're done, we get this for free. Closing as a duplicate of #4719.

[rsc]

Duplicate of #4719.