proposal: incrementally modify the Go toolchain to work without GOPATH #17271

Open
rasky opened this Issue Sep 28, 2016 · 36 comments

Projects

None yet
@rasky
Contributor
rasky commented Sep 28, 2016 edited

Over the years, when helping people approaching existing Go projects (colleagues, friends, etc.) the number one problem is of course that they don't know about GOPATH, they don't have one configured, and they expect to be able to clone an existing project wherever they want on the disk, and be able to build it. I think the problem statement is clear and the problem is well known.

With the introduction and adoption of the vendoring folder, most Go applications do not even require go get to run, after the initial clone: all the source code required to compile them is already available in the source tree; it's just that the Go tool doesn't know where to look if GOPATH is not defined.

Instead of suggesting a default GOPATH (which does not solve the fact that people will need to find out about it and use it somehow), I suggest that we change the Go toolchain in multiple steps (each one can be independently released as an experiment):

  • STEP 1: If GOPATH is not set, the Go tool will use any existing ./vendor directory as it was part of the GOPATH. (I think of this as defaulting GOPATH/src to ./vendor). At least go build and go run are guaranteed to work. Anything else, like go install and go get, would fail with an error pointing to the GOPATH documentation.
  • STEP 2: if GOPATH is not set, the Go toolchain would be updated to use $HOME/.gopkg (or similar) as pkg directory. This would allow other commands to work (e.g.: go build -i or go install).
  • STEP 3: if GOPATH is not set, go get (or the new package manager, if/when it gets released) would be modified to download packages directly into ./vendor

As shown in Step 2, I think this proposal would also interact well with the existing proposal/discussion of dropping pkg in favour of a hidden cache directory, as that would move things further into the direction of not needing GOPATH anymore for a whole class of development; in fact, this functionality could be prototyped in this specific scenario of "GOPATH not defined", where it would make sense to do it without facing the bigger discussion of deprecating pkg for a normal GOPATH scenario.

As shown in Step 3, I also think that this might interact well with the new package manager. I saw many people requested that the new package manager ought to be able to download dependencies directly into the vendor folder (like govendor and glide can do); if this functionality lands in the new package manager, it would be a perfect default for the "GOPATH not defined" scenario described here. Alternatively, it might make sense to discuss modifying go get to have this behaviour.

I think this proposal is incremental, backward-compatible, doesn't affect existing users that successfully use GOPATH, and allow beginners to approach existing Go codebases without stumbling too soon into GOPATH.

@davecheney
Contributor
davecheney commented Sep 28, 2016 edited

If you don't want to use $GOPATH, there's always gb :)

But seriously, in my experience the problem newcomers have with $GOPATH is not its prescribed namespace layout, but the fact that they have to choose a location.

Using the go tool does have a learning curve, that is undeniable, but I believe the go tool, and the conventions of laying out source in the $GOPATH form aren't hard to learn, and once people understand the rationale behind the $GOPATH layout there is general approval.

My proposal in #17262 seeks to defer the additional cognitive load of choosing a $GOPATH value by choosing a reasonable one for the user. Users still need to understand how $GOPATH works, because that is also how import statements work in Go code because the compiler interprets an import statement as a path rooted at some location, that being a segment of the users' supplied $GOPATH. But that additional learning can take place after they've been able to write small Go programs and go get code from others.

@rasky
Contributor
rasky commented Sep 28, 2016 edited

But seriously, in my experience the problem newcomers have with $GOPATH is not its prescribed namespace layout, but the fact that they have to choose a location.

I respectfully disagree. I never had somebody complain to me that choosing GOPATH was difficult per-se (as in, how to call the directory). Their main objection is that there's no good GOPATH location because they usually have projects layed out as ~/Sources/prjname, and there is no good way of matching this layout with that imposed by GOPATH. Choosing a default GOPATH would swap one confusing error message with another. If you go with defaulting GOPATH with something, they would still have to realise that:

  • There is a thing called GOPATH
  • That thing has a default (that might point to a directory they don't even have)
  • Checking out their project of choice in $GOPATH/projectname will still not work
  • Checking out their project of choice in $GOPATH/src/projectname might still not work (in case the project refers to one of its subpackage as github.com/whatever/projectname/foo).

My proposal removes all four hurdles above; it erases the whole cognitive load associated with GOPATH for the purpose of a clone+modify+push+pull-request cycle, and thus lowers the barriers of entry much more.

Also, if we look at a modern Go application, with all its dependencies correctly vendored, I really can't see what GOPATH is buying to a user that's just trying to edit and recompile such application. Every source file that must be built is already there; it looks overly aggressive to ask them to shuffle their directories around and set environment variables just to be able for the Go tool to find a library that is stored in the official vendor subfolder. It did make more sense when they had to go get to fetch the dependencies, but that step is not so common anymore.

@dmage
dmage commented Sep 29, 2016

But which import path supposed to use for the project's packages if there is no GOPATH?

@sdwarwick

The GOPATH issue has been a real stumbling block. We have multiple projects for multiple clients, each project uses a multitude of different languages and frameworks. They are all in different directories, different volumes etc.. We want all code associated with a specific project to be separately managed. It has been very difficult to see how the GOPATH architecture supports this. It is certainly not documented in the standard "getting started" information.

@rasky
Contributor
rasky commented Sep 29, 2016

@dmage I think it should just be "." or whatever it is called the top-level path in the import hierarchy. This would mean that, to work without GOPATH, projects including sub-packages will have to refer to them with a relative path like "./foo". I think it's a reasonable compromise for projects that wish to build without GOPATH.

(a possibly crazy idea is to also use canonical import path for this, that is a way of teaching the Go toolchain the path of the current project being built in a no-GOPATH scenario)

@bradfitz bradfitz added the Proposal label Oct 4, 2016
@quentinmit quentinmit modified the milestone: Proposal Oct 4, 2016
@ngrilly
ngrilly commented Oct 25, 2016

Their main objection is that there's no good GOPATH location because they usually have projects layed out as ~/Sources/prjname, and there is no good way of matching this layout with that imposed by GOPATH.

As I work on several projects using Go alongside otherside programming languages, this is exactly my experience.

@kokes
kokes commented Oct 25, 2016

Since #17262 is going forward, the "if GOPATH isn't set, use '.'" can no longer apply. But a slight change to "if outside the GOPATH, use '.'" would do the trick - mimicking gb's behaviour and allowing people to just download/copy a piece of code and compile it then and there (possibly with a friendly warning message, that the user is outside the GOPATH).

Then again, this sounds an awful like #12488, which was declined.

@adg
Contributor
adg commented Oct 25, 2016

Regarding the resolution of #17262: while this proposal takes advantage of an unset GOPATH, I don't see that as a key element. It is still possible to take advantage of a ./vendor outside an explicit or implicit GOPATH.

@rasky
Contributor
rasky commented Oct 25, 2016

Well, yes, I guess it's possible to adapt a part of this proposal with a default GOPATH (though STEP3 probably will be impossible to implement, for instance), but I think this proposal (e.g. in this comment: #17271 (comment)) makes a point that a default GOPATH isn't really helping, and instead defines a different, more comprehensive workflow to help beginners work with Go without having to learn about GOPATH from the get go. If the decision on #17262 is final (which I consider very unfortunate, as I think this proposal is a superior and more comprehensive solution to the problem statement), I'll have to revisit this proposal.

@justinfx

I have had to deal with the problem that this proposal offers to solve. That being, making it easy for the non-go developers at my company to be able to check out a Go project to a random location and build it.

Because I can't expect every developer to have a GOPATH set up, or to clone to a specific location, I have needed to design Go logic into our waf-based build system. This has to try and do extra smarts such as forming a GOPATH on the fly if it can. Telling them to use gb is not an option for multiple reasons. They would need a special tool as a non go developer, and gb takes its own approach to the vendor directory (last I checked) in having vs not having a src subdirectory.

It would be fantastic if the standard go tool could offer sane default behavior for trying to build a project that is outside gopath, before failing. If it at least tried to assume the project is self contained with a vendor directory, it would have a chance to build.

Isn't this following the same goal as #17262 in having default behavior that makes it easier for beginners?

@davecheney
Contributor

The go tool requires GOPATH, it's built into its design and backed by its
interpretation of the import statements. They are inseperable.

What is it specifically that you don't like about gb? It seems like its
design of a self contained project structure is what you need. How can I
help make gb a better fit for your requirements?

On Wed, 26 Oct 2016, 05:42 Justin Israel notifications@github.com wrote:

I have had to deal with the problem that this proposal offers to solve.
That being, making it easy for the non-go developers at my company to be
able to check out a Go project to a random location and build it.

Because I can't expect every developer to have a GOPATH set up, or to
clone to a specific location, I have needed to design Go logic into our
waf-based build system. This has to try and do extra smarts such as forming
a GOPATH on the fly if it can. Telling them to use gb is not an option for
multiple reasons. They would need a special tool as a non go developer, and
gb takes its own approach to the vendor directory (last I checked) in
having vs not having a src subdirectory.

It would be fantastic if the standard go tool could offer sane default
behavior for trying to build a project that is outside gopath, before
failing. If it at least tried to assume the project is self contained with
a vendor directory, it would have a chance to build.

Isn't this following the same goal as #17262
#17262 in having default behavior
that makes it easier for beginners?

โ€”
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
#17271 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAcAwmMhO6OiiB5eMHxl36Y-d_we155ks5q3k2ogaJpZM4KJZPO
.

@justinfx
justinfx commented Oct 25, 2016 edited

What is it specifically that you don't like about gb? It seems like its
design of a self contained project structure is what you need. How can I
help make gb a better fit for your requirements?

It is the very fact that gb diverges from the official standard, that being the go tool. Problem is that I cannot force a developer to build their Go projects with gb, so that I can then assume I can use gb to build it later, in our build system layer. The most likely path is that a developer will use the official go tool and structure their project (including the vendor directory layout) to match. Then I have to make sure any other developer can clone that project and build it, regardless of where they clone it. To use gb, I would have to make it a standard that everyone follows.

@davecheney
Contributor

On Wed, 26 Oct 2016, 06:26 Justin Israel notifications@github.com wrote:

What is it specifically that you don't like about gb? It seems like its
design of a self contained project structure is what you need. How can I
help make gb a better fit for your requirements?

It is the very fact that gb diverges from the official standard, that
being the go tool.

That's fair, but I would suggest that the chances of the Go tool making
GOPATH optional are about the same likely hood as gb acquiring some
official status.

The important thing for me when I built gb was the realisation that the go
tool isn't the lanaguge or the compiler, it's just a driver that feeds
files to the compiler in the correct order, so in that respect gb is a peer
to the go tool.

Problem is that I cannot force a developer to build their Go projects with

gb, so that I can then assume I can use gb to build it later. The most
likely path is that a developer will use the official go tool and structure
their project (including the vendor directory layout) to match.

Yeah, but is it fair to say that it doesn't work for you, or them,
otherwise we wouldn't be having this conversation?

Then I have to make sure any other developer can clone that project and

build it, regardless of where they clone it. To use gb, I would have to
make it a standard that everyone follows.

Gb is only for projects, applications that produce binaries. It has no
story for library developers, but in saying that, the cost in terms of
perceived comparability, I feel, is not as strong when you approach the
problem on a application by application basis.

โ€”

You are receiving this because you commented.

Reply to this email directly, view it on GitHub
#17271 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAcA5FC2Ia4SdHd28Q1Fjazln37huDtks5q3lf1gaJpZM4KJZPO
.

@justinfx

Yeah, but is it fair to say that it doesn't work for you, or them,
otherwise we wouldn't be having this conversation?

I wouldn't say gb doesn't work on its own. Its more a case of gb not being able to be suggested as the solution to the problem being proposed here. I was just trying to make a case really for the go tool to adopt some of the concepts of gb, by making the zero value of the go tool more useful.

The most common scenario would be that people are following the official Go project structures and expect projects to be able to be built with the go tool. So with that being said, it would be amazing if the go tool could cope with more situations and try harder to build before failing. Just like gb is only for projects, applications that produce binaries, the go tool could also try to ensure that an application can be built regardless of GOPATH not being set, if it can do so by assuming some defaults.

@ironiridis

It doesn't seem like these two proposals are in conflict. #17262 proposes a mechanism by which GOPATH can be inferred when the environment variable is missing. This proposes an algorithm for that inferred path. In fact it would seem this proposal actually depends on #17262.

@bradfitz
Member
bradfitz commented Nov 7, 2016

Let's see how much of the problem is solved by automatic GOPATH=$HOME/go in Go 1.8 and revisit this in a release or two.

@lucsky
lucsky commented Dec 8, 2016

@bradfitz I really don't see how an automatic GOPATH solves anything, this proposal is about the possible ability to do without the GOPATH.

@bradfitz
Member
bradfitz commented Dec 9, 2016

I understand the difference. Hey, listen, I don't want GOPATH either. But I'm also realistic about how fast things move. I'd love to be see it happen sooner than later. Personally I'd like something like git where the "go" command would walk up the tree, looking for a magic directory that's the GOPATH. In Git, that's the .git directory. The Go project avoids dot files for the most part, and we're not going to hide user's code. I'd like to see a magic gopath directory that contains the gopath.

We also need to fix #4719 first and get rid of the $GOPATH/pkg directory entirely, making all build artifacts aggressively cached. That would fix another few open issues and common user confusions too.

Then we you could have structures like:

$HOME/my-projects/fooproj/cmd1
$HOME/my-projects/fooproj/cmd1/main.go
$HOME/my-projects/fooproj/cmd2
$HOME/my-projects/fooproj/cmd2/main.go
$HOME/my-projects/fooproj/gopath
$HOME/my-projects/fooproj/gopath/github.com/you/bar/bar.go
$HOME/my-projects/fooproj/gopath
$HOME/my-projects/fooproj/gopath/vendor/github.com/

And if you were in the "cmd2" directory and ran go install, it would walk up, file "gopath", and prepend that to your $GOPATH environment, which will likely be empty for new users in a GOPATH-less world.

"go get" and everything would work.

"go install" would need a bin directory to write to. $GOBIN exists and would be used first. If unset, it could be write to $GOPATH/bin. (yes, there's no gopatch/src in this proposal. Maybe it could exist, but it seems unnecessarily deep) Or we put the default $GOBIN elsewhere.

Anyway, the point is that I also don't like $GOPATH.

It's just a lot of design & implementation work, and nobody's on it, at least yet.

@bradfitz
Member

Or, a revised version of the above proposal:

Instead of the magic directory being gopath, instead make the magic directory be named src. When walking up the filesystem to the root, whenever /foo/bar/src is found, /foo/bar is prepended as a $GOPATH.

Then the implicit $GOPATH/bin would be $YourProjectRoot/bin, next to your src directory. Then you'd just .gitignore the bin directory, as is typical already.

So:

$HOME/my-projects/.gitignore          # with bin
$HOME/my-projects/fooproj/cmd1
$HOME/my-projects/fooproj/cmd1/main.go
$HOME/my-projects/fooproj/cmd2
$HOME/my-projects/fooproj/cmd2/main.go
$HOME/my-projects/fooproj/src
$HOME/my-projects/fooproj/src/github.com/you/bar/bar.go
$HOME/my-projects/fooproj/src
$HOME/my-projects/fooproj/bin/cmd1   # binary from 'go install' in the cmd1 dir
$HOME/my-projects/fooproj/bin/cmd2   # binary from 'go install' in the cmd2 dir

And there would never be a pkg directory, because we'd do #4719 as a dependency, which would replace pkg with a build artifact cache directory.

@kardianos
Contributor

@bradfitz I think using src as the magic would be work. I would propose something similar, but slightly different. We specify the root folder's virtual location with somehow*. We still install bin (and until pkg get's sorted out, pkg too) locally in this root.

$HOME/my-projects/fiiproj/special-file.json < "specifies github.com/you/bar"
$HOME/my-projects/fiiproj/cmd/cmd1/main.go < "imports github.com/you/bar/bears"
$HOME/my-projects/fiiproj/bears/black.go
$HOME/my-projects/fiiproj/bin/cmd1
$HOME/my-projects/fiiproj/pkg/... < until this can go away, have it live here
$HOME/my-projects/fiiproj/vendor < optional can still have vendor folder

This can live in tandem with existing $GOPATH.
Idea from here: https://docs.google.com/document/d/1GiSDQHFo5YfYMN5TBaYdjfWHI3KDyfaU1kTHzcLR2qE

In Java / C#, I also often expect a local "bin" and "obj" or similar directory, so I wouldn't make #4719 a dependency, just when that gets resolved, the pkg dir would no longer be needed. The local "bin" would stay.

A dependency of this would probably be a standard "go dep" tool to put the root path specifier.

A small side note, govendor and godep write down the path in their manifest files today so services such as heroku can build the project with just the vcs repo.

@MikaelCluseau

@bradfitz #14566 is not a duplicate of this... it's the case where we (may) have a GOPATH but want to build something from outside of it. This issue won't solve it at all. To solve #14566, we need to be able to specify the package we are in (for instance with a "--local-package=foo/bar" flag) and have everything else work by default (including the vendor directory).

@bradfitz
Member
bradfitz commented Jan 4, 2017

It might not be an exact duplicate, but the issues are all closely related. Using this as the tracking issue is good enough.

@MikaelCluseau

Ok so now this issue also includes solving the need to override the local package's "reference" when outside the GOPATH. From the top of my head, 2 proposals where made: the flag solution I just put and the "godoc" like comment solution (ie: package quota // import "k8s.io/kubernetes/pkg/quota").

@sanmai
sanmai commented Jan 5, 2017 edited

Don't see anyone suggesting this, but what if we get away from all magic altogether?

Consider there is ~/.goconf in which any developer defines his preferences, plus a per-project .goconf for settings specific to a project. Go tools are expected to look for this file not unlike Git looks for .git/config, .gitconfig and /etc/gitconfig in succession.

This approach gives great flexibility and great transparency for all. There come shorter onboarding times for new developers. There comes understanding. There come things people are craving for.
But hatred or heated discussions, endless issues or proposals? Not anymore! Things of past.

No more looking for multiples of $GOPATH set in multitude of dot files. No more questions without answers. Just type go config and here's it all before your eyes.

Even if there's such file, it doesn't mean that there aren't sensible defaults, be it $HOME/go or else. In other words, what was working will continue to work.

(These files are not meant for dependency management. Current approach with vendor already comes with needed flexibility and transparency.)

@davecheney
Contributor
@MikaelCluseau

@sanmai I'm in favor of anything that can make my use case work ;) the dot-file is another solution.

@ascotan
ascotan commented Jan 5, 2017

O.K. I'm going to bring the now defunct #14566 into this because it has been requested that all discussions regarding that proposal be continued here.

Use cases for #17271 (this proposal):

  • (case 1) user checks out a go project, hits 'go build' and it magically builds without having a set GOPATH environment variable because the dependencies are in the ./vendor folder.
  • (case 2) user checks out a go project, hits 'go get' on a dependency and it magically installs it into the local ./vendor directory with no GOPATH set

This proposal does not address what happens when a user calls 'go install' with no GOPATH set. (probably a large oversight)

Use cases for #14566 (locked/dead proposal):

  • (case 3) user checks out a go project, hits 'go build' and it magically builds WITH having a set GOPATH environment variable. This works because a local ./vendor directory overrides lookups in the configured GOPATH

Proposal #14566 doesn't say anything about the functionality of go get and how that works.

For my purposes I have a GOPATH, but when I use a tool like glide outside of the GOPATH it doesn't work because of the requirement that the entire project must reside in the GOPATH.

The similarity between these proposals is in that the code can live outside of GOPATH and work. Proposal #17262 doesn't change the fact that these are still a valid proposals. There is still a desire to run the go toolchain on code that lives outside of the GOPATH.

In my mind, #17262 actually solves some of the issues with this proposal because it defines where pkg/ and bin/ would show up if you ran 'go install' on code outside of the GOPATH.

Personally I think that if you add the ability to:

  1. Run the toolchain on code outside of the GOPATH where it defaults to $home/GO if it's not defined
  2. Allow local vendor/ folder to take precedence over installed dependencies in GOPATH

this move the ball forward.

The issue that this proposal is proposing that #14566 was not, is that 'go get' should be modified to install into a local ./vendor directory. The problem is that even with a $home/GO, if you install a dependency it ends up in $home/GO not in ./vendor and when you go check in the code, the dependency is missing.

There are a number of vendoring tools (like glide) that currently solve this problem already outside of the toolchain. However a 'phase 2 approach' might be able to get 'go get' to install either 'local' or 'global' (cough npm), so that you can toolchain install into the local ./vendor folder.

@skelterjohn
Contributor

@bradfitz have you seen https://github.com/skelterjohn/wgo ? I believe it does very close to what you suggest, with some extra stuff on top that you probably don't care about for revision pinning. I mean, I'm sure you care, but I'm also sure it's a different issue.

That is, it discovers the go workspace in the same way that git finds .git. It has worked well, in my experience, and even has a few users that aren't me.

@lloeki
lloeki commented Jan 19, 2017 edited

I am personally using a cd hook that automatically sets (and unsets) both GOPATH and PATH according to the presence of a .gopath file or directory at or above the working directory:

  • if it's a file and it's empty the containing directory is the GOPATH
  • if it's a file and it's not empty the content is the path to the GOPATH
  • if it's a directory, it is the GOPATH

With this in place I can cd into various projects and have things work the way I want by being creative in the way I place the .gopath items for each one (or group of ones) without ever having to set a GOPATH by hand. When it's a "self-contained" project (.gopath directory at the root of the project) I do have to symlink the root of the project at the proper namespace in .gopath/src (as extracted from git remote get-url).

EDIT: my point is, the really hacky part is that symlink, but otherwise that construct allows for me to have things work out of the box with what I find to be a useful convention with a file that can be included into a repo without extra tools nor requirements or assumptions for that repo (such as with wgo: "github repositories that are made to work with go get do not work as wgo workspaces").

@sanmai
sanmai commented Jan 21, 2017

Probably you have seen it already, but anyway...
https://github.com/cloudflare/hellogopher

"just clone and make" any Go project

It would be much better if we wouldn't have to resort to makefiles.

@FiloSottile
Contributor

https://github.com/cloudflare/hellogopher

_o/ I wrote hellogopher above and meant to comment here about it.

It's not the goal of hellogopher to be a definitive solution, but to show that the user flow is desired and useful, and to act as a stopgap.

I was very happy to find this proposal. I think what's missing from @rasky's is a way to define the import path "cutoff point", which in hellogopher is a Makefile with IMPORT_PATH defined, while here could be a .IMPORT_PATH file, maybe? I'm not as enthusiast about @bradfitz's because it sounds like exactly setting GOPATH to the project root (while standard and more convenient), which forces a user to choose between the "go get" style and this.

@mvdan
Member
mvdan commented Jan 23, 2017

while here could be a .IMPORT_PATH file, maybe?

If dot files are to be avoided, perhaps a magic header similar to +build in a determined Go file like doc.go would be a better option. That would also mean that an extra file could be avoided.

@skelterjohn
Contributor
@sdboyer
Member
sdboyer commented Jan 26, 2017 edited

Chiming in here quickly, without having really read the details (sorry!), to note that I think there's a very sane path to getting rid of GOPATH - or at least making it not horrible - that's intertwined with the experimental dep tool. We haven't explicitly put it in a roadmap or anything - things are still a bit of a whirlwind - but it makes sense to treat these things together.

@davecheney
Contributor
@sdboyer
Member
sdboyer commented Jan 26, 2017 edited

I know, I'm sorry, I'm really not trying to be coy or withhold information. I almost didn't say anything, but that seemed worse than dropping a ton of info out of the blue later.

It might take a little while to get it written up - a couple weeks? - as it's fairly involved.

@frankcarey frankcarey added a commit to ahoy-cli/ahoy that referenced this issue Jan 31, 2017
@frankcarey frankcarey Add requirements for building ahoy from the /home/fcarey/go.
* The vendor folder isn't supported outside of the $GOPATH or without $GOPATH being set. See golang/go#17271
00d418b
@frankcarey frankcarey added a commit to ahoy-cli/ahoy that referenced this issue Jan 31, 2017
@frankcarey frankcarey Add requirements for building ahoy from the $GOPATH.
* The vendor folder isn't supported outside of the $GOPATH or without $GOPATH being set. See golang/go#17271
65bd613
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment