Compatibility with tools at golang.org/x/tools

[ngrilly]

I understand that gb tries to replace go build and go test and replace or avoid go get. But what about the other tools at golang.org/x/tools? godoc, oracle, goimports, callgraph, eg, gomvpkg, gorename, etc. can be very useful and rely on a GOPATH being set. Thanks!

[davecheney]

That's a good question. For most you should be able to emulate GOPATH with

GOPATH=$PROJECT:$PROJECT/vendor godoc

For example. This is also what the plugin interface is for, gb doc $PATH
could be written in 3 lines of shell.

On Tue, May 5, 2015 at 3:51 AM, Nicolas Grilly notifications@github.com
wrote:

I understand that gb tries to replace go build and go test and replace or
avoid go get. But what about the other tools at golang.org/x/tools?
godoc, oracle, goimports, callgraph, eg, gomvpkg, gorename, etc. can be
very useful and rely on a GOPATH being set. Thanks!

—
Reply to this email directly or view it on GitHub
#42.

[ngrilly]

Hello Dave,

To be frank, this is the answer I expected, and this is honestly a good solution. :-)

But, as we are there, why not do the same for go build and go test? This is not trick question. I'm trying to understand the added value of gb.

[davecheney]

I think the fact that gb does not just wrap the go tool is one if its distinguishing features.

Technically there is nothing that gb does that couldn't be done with a shell script. In fact several tools have been proposed over the last few years that do just this, all have been completely overlooked by the market.

Why is this? Well probably for one, Go programmers want to use tools written in Go. This seems like a bizarre statement, but it's true, not just for Go programmers, but pretty much most programming communities. Ansible exists because Python programmers didn't want to manage their servers with Ruby.

The second reason is, while you can construct an equivalent line of shell to what gb does, generalising that with $PROJECT detection from you working directory and handling the case you may be several levels deep what was a one line shell script becomes a rather complicated shell script, which adds further weight to programmers not wanting to use a tool not written in their language.

Basically, any solution to this problem will be

a. written in Go
b. at least wrap the Go tool.

I think there is also value in convention. I chose vendor/src arbitrarily because that's what it was called in Rails. Without this convention people would use 3pp, external, etc, ie everyone would come up with their own naming convention, and suddenly your not talking one shell script, but one shell script per project, along with various configuration flags, etc.

So, what does gb do ? Nothing that you couldn't do by hand. But people don't want to do things by hand, they want someone else to do it for them; hence gb.

[ukiahsmith]

Dependency management in Go is a problem. It's inconsistent and cumbersome. There are competing tools and processes all of which sort of work. I like gb because it is solving the dependency issue in a Go-like way.

Yes everything (at the moment) that gb does could be done with a shell script; but as Dave mentioned, the has not been widely adopted. Solidify this practice in a tool, and hardcoding specific directories makes it easier for users to adopt this pattern. Having hardcoded src and vendor/src directories will bring consistency to the Go ecosystem, and ease of use; similar to the philosophy behind go fmt.

[ngrilly]

I understand that gb, or any solution similar to gb, should at least 1) detect the project root by walking up from the current directory and 2) wrap the go tool. I fully agree it is simpler and better to write such a tool in Go rather than in shell. But it is beyond my point. By glancing at the source, it's clear gbis more than a wrapper. I'm wondering what are the other features of gb beyond detecting the project root and wrapping the go tool (by "wrapping" I mean something equivalent to setting the GOPATH and invoking the go tool)?

As an aside, I'm a user of Ansible since a long time (and even a minor contributor), and I doubt its success is related to Python programmers wanting to use devops tools written in Python. I'd say Ansible is successful because of its learning curve (it's a lot simpler than Chef and Puppet), its YAML recipes, and using SSH instead of dedicated agents and servers. I'm happy that Ansible is written in Python, but I would have chosen it over Chef and Puppet even if it was written in Ruby ;-)

[ngrilly]

@Supermighty, I agree that dependency management is still a problem in Go. But it's probably more a problem for newcomers, lost in the sea of existing solutions, than for experienced Go developers, that can easily use one GOPATH per project (like Digital Ocean [1]) or import path rewriting (like godeps, nuts or kardianos/vendor). And I'd say that even in languages like Python, Ruby, Node or Java which have established dependency management solutions, it's not all roses, and these tools come with their own set of problems and limitations.

I agree that gb can be a useful improvement to the status quo. By the way, I use an almost identical approach for my internal projects, using almost the same GOPATH structure than gb, and setting the GOPATH in a Makefile before invoking the go tool. For this reason, I welcome the birth of gb which can be an improvement over my Makefile :-)

But I'm worried about the scope of gb, about gb going too far in trying to replace the go tool (I read the slides and watched Dave's talk). I'm worried because 1) it's very ambitious and I'm not sure there are enough resources for implementing another go tool outside the Go team and 2) what we win in terms of build simplicity could be lost in integration complexity with other tools relying on the GOPATH (oracle, godoc, goimports, gorename, callgraph, etc.).

Again, I really appreciate the effort, but we need to think forward before risking to "fragment" the community for a small benefit in return. In your comment, you complained there are many competing dependency management tools and processes. It would be worse to have several competing build systems and processes.

[1] https://www.digitalocean.com/company/blog/taming-your-go-dependencies/

[davecheney]

Nicholas, forgive my bluntness, but you are arguing to not do something
because it is hard. You are arguing to not do something because possibly it
won't work out.

Obviously I don't share your concerns and feel the task is achievable.

Even if gb is not a success, the idea of standardising on a project layout
rather than a single GOPATH layout, is far more valuable than my
implementation. And I have shown that a project based layout is totally
workable and can be made backwards compatible for existing tools that
expect a GOPATH layout.

On Tue, 5 May 2015 06:17 Nicolas Grilly notifications@github.com wrote:

@Supermighty https://github.com/Supermighty, I agree that dependency
management is still a problem in Go. But it's probably more a problem for
newcomers, lost in the sea of existing solutions, than for experienced Go
developers, that can easily use one GOPATH per project (like Digital Ocean
[1]) or import path rewriting (like godeps, nuts or kardianos/vendor). And
I'd say that even in languages like Python, Ruby, Node or Java which have
established dependency management solutions, it's not all roses, and these
tools come with their own set of problems and limitations.

I agree that gb can be a useful improvement to the status quo. By the
way, I use an almost identical approach for my internal projects, using
almost the same GOPATH entries, and setting the GOPATH in a Makefile before
invoking the go tool (this is similar to what Peter Bourgon does in
github.com/soundcloud/roshi). For this reason, I welcome the birth of gb
which can be an improvement over my Makefile :-)

But I'm worried about the scope of gb, about gb going too far in trying
to replace the go tool (I read the slides and watched Dave's talk). I'm
worried because 1) it's very ambitious and I'm not sure there are enough
resources for implementing another go tool outside the Go team and 2) what
we win in terms of build simplicity could be lost in integration complexity
with other tools relying on the GOPATH (oracle, godoc, goimports, gorename,
callgraph, etc.).

Again, I really appreciate the effort, but we need to think forward before
risking to "fragment" the community for a small benefit in return. In your
comment, you complained there are many competing dependency management
tools and processes. It would be worse to have several competing build
systems and processes.

[1] https://www.digitalocean.com/company/blog/taming-your-go-dependencies/

—
Reply to this email directly or view it on GitHub
#42 (comment).

[ngrilly]

you are arguing to not do something because it is hard. You are arguing to not do something because possibly it won't work out.

I reread my comment and I don't think this is what I've been arguing.

I'm not arguing "to not do it because it's hard". I'm arguing about making sure the resources dedicated to gb are important enough. Writing a build tool that aims to be an alternative to the standard built tool is not a small project. I think this a legitimate concern, considering that even the Go team, which is composed of several people working full-time on the matter, needs time to fix bugs like the ones related to builds broken because of stale dependencies (caused by the use of multiple GOPATH entries, by source files being deleted, or by build tags). I'm just trying to be realistic and to understand if it's reasonable to start using and contributing to gb, or not.

I'm not arguing "to not do it because it won't work out". I'm arguing about making sure to not create more confusion about the issues of managing its GOPATH and managing dependencies. One of the stated goal is to simplify the life of Go developers by not having to setup the GOPATH manually to build a project. I concur with that. But the truth is that Go developers use many other tools (goimports, godoc, oracle, gorename, etc.) and they have to setup their GOPATH anyway. I'm just wondering what have we gained?

To be very clear, I'm not against the idea of gb. But as a Go developer and a potential user and contributor, I'm trying to gain a clear understanding of the goals, the benefits, and the costs. I think this a fair request. Since you've gone public about this project at GDG Berlin, I think it's normal people start to ask questions :)

[davecheney]

I must apologise for the tone of my previous remarks. However I would like to correct some misunderstandings

I'm not arguing "to not do it because it's hard". I'm arguing about making sure the resources dedicated to gb are important enough.

I'm dedicating my time to making gb the build tool I want to use. Apart from a personal desire to fix this problem, I hope one day that I'll be able to use this on juju/juju, but that's a bridge I'll cross when I come to it.

I guess the question of is gb the right tool for your project, is really a question of do you trust me to get it right and not abandon to the tool. The best answer I can give is to point to my 5 years of service to the Go project, and my record of speaking on this issue both in these comments and at conferences and meetups over the last few years. You can also look at the blog posts on my website and the go-pm mailing list for additional validation of how seriously I take this issue.

I'm not arguing "to not do it because it won't work out". I'm arguing about making sure to not create more confusion about the issues of managing its GOPATH and managing dependencies.

@nathany compiled a list of several dozen competing tools, and this was over a year ago, the field is even more crowded since then. I'm afraid that I don't see how me writing another tool could make things any more confusing than they already are.

One of the stated goal is to simplify the life of Go developers by not having to setup the GOPATH manually to build a project. I concur with that. But the truth is that Go developers use many other tools (goimports, godoc, oracle, gorename, etc.) and they have to setup their GOPATH anyway. I'm just wondering what have we gained?

You are correct that today none of those tools work with gb projects, because up until a few weeks ago, none of those tools were aware of gb projects. Given how similar a gb project is to a multi element $GOPATH, something those tools have to already support, if gb were to gain sufficient market traction, adding support for gb projects to those tools is trivial (I say, knowing that I may be signing someone else up to do the work). In summary, the concern that those tools don't understand a gb path is valid, but certainly not a showstopper in my opinion.

In summation. You've asked in several threads why I chose to write a new tool. The best answer I can give is because none of the others had achieved significant market share. I hope, by trying something different, by not just wrapping the go tool and working within the structure of $GOPATH, gb will achieve more success.

[ChrisHines]

Technically there is nothing that gb does that couldn't be done with a shell script. In fact several tools have been proposed over the last few years that do just this, all have been completely overlooked by the market.

Why is this? Well probably for one, Go programmers want to use tools written in Go.

My reason is that a shell script is less portable than Go code. When I investigate a proposed solution to this problem and I see the phrase "shell script" I just stop and move on.

My use case requires building binaries for a heterogeneous environment (Windows & Linux). Hence the capability to cross compile or compile on multiple platforms or do both, using a single build tool and a single project repo, interests me far more than a platform specific script.

[tsuna]

I'm going to ask what's perhaps a stupid question, but which I think paraphrases what @ngrilly is saying.

What if gb was really just doing this:

• Detect the project root
• Export GOPATH=projectroot/src:projectroot/vendor/src
• Execute (as in execv) an arbitrary other command.

So one could do gb go build ./... or gb goimports etc. Of course it's not great to have to prefix every Go-related command with gb, but if this way of organizing Go projects is successful, then this behavior could be just ported to the go tool and it would become the de-facto standard way to establish the GOPATH.

If this whole problem boils down to managing GOPATH and vendoring dependencies under a well-established path (vendor/src) then isn't the above what gb would boil down to?

I think the only exception is that gb go get would just need to clone its stuff under vendor/src instead, as opposed to cloning it under the first GOPATH entry, and also people might not wanna keep the .git / .hg around when they vendor the dependency, just the revision at which the dependency was obtained, a la godep.

[davecheney]

@tsuna it's certainly not a stupid question. The reason gb does not wrap the go tool is two fold

a. divorcing itself from the go tool gives gb a clean slate to work from, not to be bound to simple orchestrate the movements of the go tool. There is a fair point to be made that in #50 I made the decision to retain some compatibility with source arranged in the traditional GOPATH layout, but this was for compatibility with tools like goimports, godoc, golint, etc which work with GOPATH. It was not for compatibility with the go tool itself.

b. Personally I don't see any value in a tool that wraps the go tool. Others have tried, most notably godep (without -r), and it received little mindshare. I'm going to go to the extreme of saying I felt that Go developers do not want another wrapper, they want an opinionated tool.

[kidoman]

I wonder if we could have achieved the solution to the core issue (fix the vendoring problem and get mindshare while at it) in a "opinionated" way without having to potentially fragment the Go community around the usage of the Go tool.

@tsuna has made a very valid argument about how much of what we are trying to achieve can also be achieved without having to reinvent the wheel. While no one is doubting your (as in @davecheney) commitment to the project and what you have done for the Go community thus far, I am a bit worried about effort being spent towards the wrong thing.

So the questions which come to my mind are:

• What opinion(s) is gb trying to encourage?
• Could we do what we are trying to achieve (src/ and vendor/src/) without having to rewrite the build process? (there are heavy costs on both sides of the coin)
• Is gaining mindshare anyway related to throwing away the current approach (the go tool) ? If not, why do it?

Also, I would say that https://github.com/tools/godep has significant mindshare. Almost every big Go project (docker, kubernetes come to mind) use it in almost exactly the same way as gb (except for the name of the actual vendoring folder).

[davecheney]

@kidoman

Could we do what we are trying to achieve (src/ and vendor/src/) without having to rewrite the build process? (there are heavy costs on both sides of the coin)

No, I do not believe so.

Is gaining mindshare anyway related to throwing away the current approach (the go tool) ? If not, why do it?

My philosophy is go get has artificially constrained the solution space for solving the go dependency management problem. You can wrap other tools around go get and go build to work around this, but like I say, they are workarounds and continue to live within the limitations of the go tool.

None of these approaches that wrap the go tool have been successful, even godeps, which you mentioned has abandoned it's wrapping (godeps go build) in favor of vendoring code via rewrites. I do not believe another tool that wraps the go tool would have an appreciable impact on the market, so it's time for something more radical.

[davecheney]

I am a bit worried about effort being spent towards the wrong thing.

@kidoman if this is your way of asking "why didn't you work with the go team to change the go tool?", I'll remind you that the Go team have chosen vendoring via import rewriting. I have spelled out in my presentation why I am not in favor of this approach, so there is no common ground to be had here.

[kidoman]

Side note: While godeps might be heading towards the rewriting approach, most of the projects are still using it w/o rewriting their dependencies. See: https://github.com/GoogleCloudPlatform/kubernetes/blob/master/Godeps/_workspace/src/golang.org/x/net/context/context.go

On topic:

I feel that we are conflating a few slightly unrelated things:

• Need solution to vendoring, and rewriting dependencies is not ok (a)
• Solution has not gained market share (b)
• Solution has wrapped the go tool (c)
• Vendoring is done by manipulating the GOPATH (d)
• Tool is written by a very respected member of the Go community (e)
• Solution has gained market share (f)
• Tool is written in Go and not as a shell script (g)

Let's assume for a moment (as it is a separate discussion) that (a) is a fact.

Now, what you are saying is: (c) -> (b) (-> = implies)

This is something I am not able to agree with. I believe this is what should happen: (e) + (c) + (d) + (g) -> (f)

I love that you are tackling this problem head on, and you are on the right track when it comes to the eventual "vendoring" approach, but I do not see the connection between (c) (wrapping the Go tool) and (b) (not gaining market share).

If you were to write a tool which brings out a opinionated approach to vendoring w/o disrupting the currently established go build process, the tool would gain market share purely because of (e), (g) and (a) (irrespective of whether it does (d) or not). And that would not have the side effect of fragmenting the Go community and leading to a lot of bikeshedding (I can foresee a lot of discussions in teams as they start out on a Go project about whether to pick a vendoring tool which uses the standard Go build tool or to pick a tool which replaces the standard Go build pipeline).

Would having two competing Go build tools help the Go community? The jury is out on that one.

[davecheney]

@kidoman you wrote a lot, so forgive me for not writing a lot in return. Here is the shortest answer to all your questions

I wrote gb because I did not feel that another tool wrapping the go tool would have enough impact to justify the effort put into it, and ultimately would be unsuccessful in actually fixing the problem of reproducible builds that I had.

People who agree with my approach will use gb, people who wont, will not use gb. That's all there is to it.

Would having two competing Go build tools help the Go community? The jury is out on that one.

Competition is good for consumers.

[tsuna]

What does gb do to fix the problem of reproducible builds that simply doing export GOPATH=projectroot/src:projectroot/vendor/src followed by an execv(3) doesn't do? Code under vendor/src doesn't change unless you run a command to update it, so the only dependent code that isn't pinned to the project is Go's standard library (and from your slides this is deemed out of scope for gb anyway).

I guess I'm still failing to see what's fundamentally different about gb than an established convention to vendor dependent code.

I just re-read the slides of your talk and my understanding on your take is:

• The notion of project and having GOPATH per project.
• Check in the source of your dependencies under vendor/src to have reproducible builds (dependency pinned to your tree + you no longer depend on the upstream repo existing or not).
• No import rewriting or duplicates (since each dependency must appear only once under vendor/src).
• A configuration-less tool that detects the current project using the directory hierarchy.

Isn't the above achievable using all existing tools simply by setting GOPATH automatically depending on the current project? Let's say the go command had a -gb flag that didn't honor any existing GOPATH environment variable but rather detected the current project using the directory hierarchy and set GOPATH=projectroot/src:projectroot/vendor/src before proceeding normally. Wouldn't that be equivalent to what gb does today?

I could do go -gb get <url> and that would vendor a new dependency under vendor/src that I would git add and git commit. If I want to update my dependency I'd do go -gb get -u <url> and commit the resulting diff. My builds would be isolated and reproducible since they only use my src/ and my vendor/src/.

[davecheney]

@tsuna what you have described it Keith Rarick's godep, used without the -r flag to save.

gb is an alternative to that that does not involve wrapping the go tool and goes not use GOPATH. It does this to make project owners decide how they want to manage the dependencies of their project. They will have to choose one of these options:

a. use vendoring via import rewriting as proposed by the Go team
b. use vendoring via wrapping the go tool as proposed by godep
c. use vendoring with a new project layout that is not constrained by a requirement to work around the go tool.

It is worth noting that, prior to gb, project owners only have choices a and b.

[mkmik]

@davecheney

c. use vendoring with a new project layout that is not constrained by a requirement to work around the go tool.

a. divorcing itself from the go tool gives gb a clean slate to work from

From this I infer you are preparing the ground for behavioural changes and don't want to be tied to the limitations that being just a go build wrapper would bring.

Do you have anything specific in mind ?
(or pointers to issues exploring it, I'm new to the project and too lazy to read all history, just want to make it work with goimports)

Given how similar a gb project is to a multi element $GOPATH, something those tools have to already support, if gb were to gain sufficient market traction, adding support for gb projects to those tools is trivial (I say, knowing that I may be signing someone else up to do the work).

I might try to give it a stab. However, what would be an acceptable way for e.g. goimports to detect that it's running inside a gb project ? Just scan for src + vendor/src.

My editor invokes goimports on save. I don't want to change the confs of my several editor environments just to pass some flag that enables gb detection. Env vars would be also suboptimal.

I know that it has been decided to go on with no configuration file, so we cannot rely on the presence of a .gb file or something like that.

The reliability of the test might be crucial for making the support land in the official goimport (and friends).

[davecheney]

From this I infer you are preparing the ground for behavioural changes and don't want to be tied to the limitations that being just a go build wrapper would bring.

The major, and only change is a move away from $GOPATH, to the project structure described elsewhere. This is change enough, I only want to boil one ocean at a time.

I might try to give it a stab. However, what would be an acceptable way for e.g. goimports to detect that it's running inside a gb project ? Just scan for src + vendor/src.

I've proposed we do this as a gb plugin, see #61

My editor invokes goimports on save. I don't want to change the confs of my several editor environments just to pass some flag that enables gb detection. Env vars would be also suboptimal.

I'm afraid I have no solution for you at the moment. However gb is backwards compatible with $GOPATH workspaces, because it fulfils the project autodetction rules, see #53. gb treats $GOPATH as one giant project with no vendored code.

I know that it has been decided to go on with no configuration file, so we cannot rely on the presence of a .gb file or something like that.

No, see #53 for an explanation on how to detect a gb project.

[kevinjos]

That's a good question. For most you should be able to emulate GOPATH with

GOPATH=$PROJECT:$PROJECT/vendor godoc

For example. This is also what the plugin interface is for, gb doc $PATH
could be written in 3 lines of shell.

How do you feel about including a bin directory in the project root to setup the environment for golang.org/x/tools compatibility, similar to Python virtualenv's activate/deactivate? This could be done by gb-init. Or should this be left as an exercise for the end user?

[davecheney]

How do you feel about including a bin directory in the project root to setup the environment for golang.org/x/tools compatibility, similar to Python virtualenv's activate/deactivate? This could be done by gb-init. Or should this be left as an exercise for the end user?

I'm not a fan of this style of workflow; http://go-talks.appspot.com/github.com/davecheney/presentations/reproducible-builds.slide#18

I think we can do better via plugins.

[kevinjos]

I think we can do better via plugins.

How do you suggest existing plugins that use go tooling like vim-go adapt to this proposed change? My suggestion above is mostly concerned with code already written that assumes a specific environment.

[davecheney]

How do you suggest existing plugins that use go tooling like vim-go adapt to this proposed change? My suggestion above is mostly concerned with code already written that assumes a specific environment.

It's simple to write a gb plugin that constructs a GOPATH from a gb project and calls out to the tool.

https://github.com/constabulary/gb/blob/gb-goimports/cmd/gb-goimports/main.go

It's crude, but it works, and can be improved later on.

[kevinjos]

It's simple to write a gb plugin that constructs a GOPATH from a gb project and calls out to the tool.

Agreed, but unless I am missing something Vim script would have to be written to call gb instead of go for vim-go to be fully functional in a gb project environment. I imagine there other other IDE plugins that are also programmed to call go tools on certain triggers.

[nathany]

@kevinjos vim-go support is already started, though @fatih may split it out a separate vim-gb plugin. fatih/vim-go@4262864

[tsuna]

Why did you choose to have gb-goimports instead of gb-exec goimports? I guess the only thing that would change is that c := exec.Command("goimports", os.Args[1:]...) would become c := exec.Command(os.Args[1:]...)?

[kevinjos]

@nathany That's great, thanks for sharing.

[fatih]

@kevinjos Here is the PR fatih/vim-go#420 with :GbBuild and :GbBuildAll support. It works perfectly as @davecheney fixed some of the error outputs so I'm able to parse them now. I'm working on also fixing the other tools within vim-go waiting for clarification from #79

[fatih]

Some other clarification, I'm working on adding support for godep to vim-go, which is by far very interesting. I'll do an automatic detection, if it's a godep project vim-go will automatically change the GOPATH and invoke the necessary tool. I have a local branch with an initial support already and it works perfect.

I've laid out the code, so it also supports gb by default. So by adding support to godep I can see the shortcomings on my side. That's why I've opened the issue #93, to make it clear to distinguish a gb project. Once I have a clear understanding of it, it'll be very easy for me, because I'll just add another case into the existing code that will handle gb and all the current tools in vim-go will play nicely with it.

There are only some problems I have in my mind, basically what to do if something has a Godeps folder inside a gb project, example layout:

~/foo
~/foo/src/github.com/fatih/color
~/foo/src/github.com/fatih/color/Godeps
~/foo/vendor/src/...

Assuming ~foo is our $PROJECT and github.com/fatih/color is our source code which still has the Godeps folder. Now this will confuse vim-go, because this is a broken project layout and needs to be fixed by the project owner. It's just these kind of cases could happen and I need to be prepared so vim-go can handle it in a sane way.

[davecheney]

@tsuna

Why did you choose to have gb-goimports instead of gb-exec goimports? I guess the only thing that would change is that c := exec.Command("goimports", os.Args[1:]...) would become c := exec.Command(os.Args[1:]...)?

No good reason, we're still figuring out how to write plugins.

[davecheney]

I'm going to mark this as closed (answered). If you want to continue the discussion, please consider opening a new issue.