Add a summary of the differences with standard go tool

[ngrilly]

Could you add a summary of the differences with the standard go tool?

Here is the two main differences I noted:

1. The GOPATH is automatically determined depending on the current working directory by searching directories src or vendor/src.
2. The tool makes sure that deleted packages are not used anymore, even in their compiled form, unlike the standard go tool that still uses remaining .a files.

Regarding the first difference, which is the main selling point of gb, I'm wondering why it couldn't be implemented as a wrapper around the standard tool. This is what godep does, in some way. But I'm probably missing something :)

Even if it cannot be implemented as a wrapper, it would be great to reuse as much code as possible from the standard library, instead of reimplementing the whole tool?

Regarding the second difference, is there a reason for not fixing the behavior directly in the standard go tool?

[ngrilly]

Another question: What happens when a project using the gb layout (with its source under a src directory inside the project) tries to vendor a library that also uses the gb layout? Are you advocating a future in which every Go project will have a src directory that contains the Go source?

[davecheney]

Another question: What happens when a project using the gb layout (with its source under a src directory inside the project) tries to vendor a library that also uses the gb layout? Are you advocating a future in which every Go project will have a src directory that contains the Go source?

gb projects are not go get'able, they are the end of the line. See #20

[ngrilly]

I understand that gb projects are not go get'able (the premise of your project being to not try to stay compatible with the go tool), but could they be "gb" get'able? I think that the distinction between "end of line" applications and libraries is quite arbitrary. It would be very useful, in a project that uses gb, to be able to vendor a library that uses gb itself. In other words, being able to vendor transitive dependencies seems necessary.

[davecheney]

I agree, but don't have a solution for packages, not projects yet. This is
issue #20.

On Fri, 1 May 2015 23:02 Nicolas Grilly notifications@github.com wrote:

I understand that gb projects are not go get'able (the premise of your
project being to not try to stay compatible with the go tool), but could
they be "gb" get'able? I think that the distinction between "end of line"
applications and libraries is quite arbitrary. It would be very useful, in
a project that uses gb, to be able to vendor a library that uses gb itself.
In other words, being able to vendor transitive dependencies seems
necessary.

—
Reply to this email directly or view it on GitHub
#23 (comment).

[ngrilly]

FYI, it looks like Go version 1.5 will fix the issue about "deleting a file from a package doesn't cause a rebuild":

golang/go#10509

[davecheney]

Yes. That issue was addressed after I made my presentation. Competition is
good for the consumer.

On Sat, May 2, 2015 at 10:51 PM, Nicolas Grilly notifications@github.com
wrote:

FYI, it looks like Go version 1.5 will fix the issue about "deleting a
file from a package doesn't cause a rebuild":

golang/go#10509 golang/go#10509

—
Reply to this email directly or view it on GitHub
#23 (comment).

[ngrilly]

Competition is good for the consumer.

Well said. :-)

[davecheney]

I'll address this issue by adding more introductory text. The video of my GDG presentation will be available shortly as well.

For the record, gb does nothing more than GOPATH=$PROJECT/src:$PROJECT/vendor/src go build. But, as nobody has been able to popularise that manual method, I think there is value in wrapping it in a tool. Let's let the market decide.

With respect to the request to enumerate the differences between gb and other tools, I'll respectfully decline as it would be yet another incomplete list. Here is one I found, https://github.com/golang/go/wiki/PackageManagementTools, I know others are being written all the time.

[ngrilly]

For the record, gb does nothing more than GOPATH=$PROJECT/src:$PROJECT/vendor/src go build.

But why is there so much code in gb if it does nothing more than finding the project root, setting the GOPATH and invoking the go tool? It looks like gb does something more complex than simply setting the GOPATH and running go build. gb seems to reimplement a part of the toolchain (analysis of stale dependencies, management of the different build phases, etc.).

With respect to the request to enumerate the differences between gb and other tools, I'll respectfully decline as it would be yet another incomplete list.

I haven't requested a list of differences between gb and other tools. I was just interested in differences between gb and the standard go tool.

[davecheney]

But why is there so much code in gb if it does nothing more than finding the project root, setting the GOPATH and invoking the go tool? It looks like gb does something more complex than simply setting the GOPATH and running go build. gb seems to reimplement a part of the toolchain (analysis of stale dependencies, management of the different build phases, etc.).

This is what is required to write a built tool for Go. I'd love to reuse more of the code, but a lot of it is locked away in cmd/go and cannot be reused.

[davecheney]

I haven't requested a list of differences between gb and other tools. I was just interested in differences between gb and the standard go tool.

gb uses a different layout for your Go code, based on a project, not a single $GOPATH, beyond that it aims to provide parity with all the other features of the Go tool.

You can emulate everything gb does with shell scripting and wrapping of the Go tool, and of course documenting this on a per project basis, but I think Go programmers will prefer a tool that does this for them.

[ngrilly]

gb uses a different layout for your Go code, based on a project, not a single $GOPATH, beyond that it aims to provide parity with all the other features of the Go tool.

Thanks for following-up on my questions. Your answer is very clear. I'm starting to get it.

You can emulate everything gb does with shell scripting and wrapping of the Go tool, and of course documenting this on a per project basis, but I think Go programmers will prefer a tool that does this for them.

I agree that using a tool that does it well is better than implementing and documenting it on a per project basis.

This is what is required to write a built tool for Go. I'd love to reuse more of the code, but a lot of it is locked away in cmd/go and cannot be reused.

This is the part I find hard to follow. Why not just invoke the go tool using os.exec.Cmd, like what gb already does for its plugins? What are the benefits and the costs?

[davecheney]

Why not wrap the go tool and shell out? Several reasons

1. There are plenty of tools that have pitched that and failed to achieve a
   self sustaining amount of mine share.
2. To call out to the go tool would be to work within its limitations, see
   http://go-talks.appspot.com/github.com/davecheney/presentations/reproducible-builds.slide#26
3. More fundamentally, working around the go tool hasn't worked for 3
   years, another solution that proposed just that, but slightly different
   would probably fail to achieve significant market share. I felt a more
   radical solution was called for.

On Tue, 5 May 2015 08:22 Nicolas Grilly notifications@github.com wrote:

gb uses a different layout for your Go code, based on a project, not a
single $GOPATH, beyond that it aims to provide parity with all the other
features of the Go tool.

Thanks for following-up on my questions. Your answer is very clear. I'm
starting to get it.

You can emulate everything gb does with shell scripting and wrapping of
the Go tool, and of course documenting this on a per project basis, but I
think Go programmers will prefer a tool that does this for them.

I agree that using a tool that does it well is better than implementing
and documenting it on a per project basis.

This is what is required to write a built tool for Go. I'd love to reuse
more of the code, but a lot of it is locked away in cmd/go and cannot be
reused.

This is the part I find hard to follow. Why not just invoke the go tool
using os.exec.Cmd, like what gb already does for its plugins? What are
the benefits and the costs?

—
Reply to this email directly or view it on GitHub
#23 (comment).

[ngrilly]

There are plenty of tools that have pitched that and failed to achieve a self sustaining amount of mind share.

Agreed, but it doesn't explain why they failed. Maybe it is because these tools appeared too early in the history of the Go language, and the community was not ready to embrace such tools. Maybe it was because they were implemented in sh/bash instead of Go, and I agree with you that Go is better for such a tool. Maybe they were half-baked. Etc…

To call out to the go tool would be to work within its limitations

Earlier in this thread, you wrote "gb does nothing more than GOPATH=$PROJECT/src:$PROJECT/vendor/src go build". And now you're saying that gb cannot rely on the go tool because of its limitations. These are contradictory claims. This "contradiction" is the main reason why I have some difficulties following your reasoning about the goals of gb.

Could you clarify if "gb does nothing more than GOPATH=$PROJECT/src:$PROJECT/vendor/src go build" (I'm quoting you), yes or no?

Regarding the limitations of the go tool listed in your presentation:

• It looks like the two issues about deleted source files could be fixed in Go 1.5.
• It's perfectly possible to replace go get with something more powerful in terms of DVCS usage (ssh, etc.), without divorcing ourselves from the whole go tool.
• I have nothing to say about tags because I rarely use them. But I have one question: what is the rationale behind fixing this in a new tool, instead of trying to fix it by contributing to the go tool? Is there some "inertia" that makes difficult to fix this directly in the go tool?

More fundamentally, working around the go tool hasn't worked for 3 years, another solution that proposed just that, but slightly different would probably fail to achieve significant market share. I felt a more radical solution was called for.

I see what you mean. I felt the pain too. I'm sometimes annoyed by some limitations of the go tool, limitations that can feel arbitrary. The fact that stale dependencies are not recompiled across multiple GOPATH entries is a good example (it will hopefully be fixed in Go 1.5). The fact that symlinks are not followed by some tools (godoc) can be frustrating. The fact that compilation is not incremental by default. Etc…

Sometimes gb is presented as a solution to "reproducible builds" (for example in the slides). Sometimes it is presented as a "new project-based build tool for Go" (for example in GitHub). These are two very different ways to "market" the project:

1/ Saying "gb wants to fix dependency management in Go, and as a side-effect we have to replace the whole go tool" sounds weird: you start with a relatively well defined and limited problem, and you end up with writing a new build tool.

2/ Saying "gbis a new project-based build tool for Go" is okay, but in that case it must be associated to a very good explanation of why we need an alternative to the mainstream tool and what the new tool will bring to the table. Writing a new build tool is a much larger endeavour than writing a dependency management tool. And replacing the standard build tool with a new one requires a lot more trust than replacing my dependency management tool.

In order to clarify the goals of gb, is it leaning toward the first or the second statement? In other words, is it "a solution to reproducible builds, which requires a new build tool" or is it a "new and better build tool, project-based, for Go"?

[davecheney]

There are plenty of tools that have pitched that and failed to achieve a self sustaining amount of mind share.
Agreed, but it doesn't explain why they failed. Maybe it is because these tools appeared too early in the history of the Go language, and the community was not ready to embrace such tools. Maybe it was because they were implemented in sh/bash instead of Go, and I agree with you that Go is better for such a tool. Maybe they were half-baked. Etc…

That's really not something I can or feel comfortable answering. The fact is I'm trying to write a tool that gives me repeatable builds, and none of those existing tools, I felt, gave me the reproducibility I needed, for the reasons I stated in my GDG presentation.

Could you clarify if "gb does nothing more than GOPATH=$PROJECT/src:$PROJECT/vendor/src go build" (I'm quoting you), yes or no?

gb build is functionally equivalent to that command.

It looks like the two issues about deleted source files could be fixed in Go 1.5.

Yes, and you'll have to wait for August to get those fixes, and some time afterwards if your group doesn't want to change their Go version. gb will add those features shortly.

It's perfectly possible to replace go get with something more powerful in terms of DVCS usage (ssh, etc.), without divorcing ourselves from the whole go tool.

In theory yes, in practice, nobody has done so. People know that go get is just a thin wrapper over git clone, but they choose to use it anyway.

I have nothing to say about tags because I rarely use them. But I have one question: what is the rationale behind fixing this in a new tool, instead of trying to fix it by contributing to the go tool?

The Go team have stated they prefer vendoring via import rewriting. I don't agree that is the best solution, so I wrote a competing tool.

Is there some "inertia" that makes difficult to fix this directly in the go tool?

See above.

Sometimes gb is presented as a solution to "reproducible builds" (for example in the slides). Sometimes it is presented as a "new project-based build tool for Go" (for example in GitHub). These are two very different ways to "market" the project:

I don't think they are, although I admit my presentation and slides may need to be improved.

1/ Saying "gb wants to fix dependency management in Go, and as a side-effect we have to replace the whole go tool" sounds weird: you start with a relatively well defined and limited problem, and you end up with writing a new build tool.

gb fixes dependency management by adopting a project based approach, other solutions work around the go tool with its single $GOPATH workspace. The idea of a project based workflow, and a tool that uses it are inseparable in my mind.

2/ Saying "gb is a new project-based build tool for Go" is okay, but in that case it must be associated to a very good explanation of why we need an alternative to the mainstream tool and what the new tool will bring to the table. Writing a new build tool is a much larger endeavour than writing a dependency management tool. And replacing the standard build tool with a new one requires a lot more trust than replacing my dependency management tool.

I did the best job I could to provide justification for creating gb in my GDG presentation, I don't have anything more to add at this point.

I agree it isn't a small job, but it's an important job. I've addressed issues of my suitability for this task in #42, but it basically comes down to

a. do you agree with my position
b. do you trust me, noting of course that gb is an open source tool, and you can fork it if you don't like they way i'm doing things -- the idea of a project based workflow is far more important than my implementation.

[davecheney]

#42 has become a duplicate of the general "why build a new build tool" debate. I'm closing this issue as a duplicate.

There is also the go-pm mailing list which may be more appropriate for debate and advocacy.