Consider making this package go gettable?

[dmitshur]

Right now, this package requires custom manual steps for its installation:

Run go get -d github.com/libgit2/git2go to download the code and go to your $GOPATH/src/github.com/libgit2/git2go dir. From there, we need to build the C code and put it into the resulting go binary.

git submodule update --init # get libgit2
make install

will compile libgit2 and run go install such that it's statically linked to the git2go package.

This is inconvenient, it makes it harder to get than other Go packages (which are just go get). Worst of all, this means that any other Go package that imports this one, even if the upstream package is completely go gettable, will no longer be 1 command to install.

I think it may be possible for you to change that, and thus improve the experience of anyone using this packages, and make it easy for others to import it and no incur the cost of manual dependency installation steps.

The way you can do that would be similar to how we did it with glfw3 package (currently on a branch, to be merged into master once GLFW 3.1 is released), which is a Go binding for a C library. It was discovered by @slimsag and you can see the discussion in the following issue and PR:

go-gl/glfw#82
go-gl/glfw#83

The general idea is that you vendor the C library source in a subfolder (potentially as a git submodule) and use cgo directives to link to it. Basically, everything you're doing now, except without needing the Makefile and scripts.

The main consideration is that you will need to replace the function of CMake with multiple .go files with build tags (which may require some work and maintenance), which is what we've done in glfw3:

Less configurable? Unable to modify CMAKE parameters?

Actually -- We can make build tags for any CMAKE configurable feature that we want (e.g. tell people to compile their application with go install -tags "wayland" my/pkg and a wayland client will be built instead of an X11 one, you can see how I did this here: https://github.com/go-gl/glfw3/blob/devel_glfw3.1/build.go)

But the end result is a go gettable package that is much easier for users. Just imagine:

To install git2go, run go get -u github.com/libgit2/git2go. Done.

What do you think? If this is something you're amendable to, I can assist with questions/answers about the approach and potentially try to make a PR.

[carlosmn]

AFAICT there is no way to make decisions with Go directives based on what packages are installed. While I we could build a no-dep version of libgit2, it would not have any network support other than raw TCP, but that would require custom installation to have TLS/SSH transports anyway.

The larger problem however is that this would require a lot of work just to get to the same state as we have now. Getting a package which go will build directly requires rebuilding the build system we have on the libgit2 side and it does not let us automatically handle optional dependencies. So we'd likely have to go all-or-nothing, or have different capabilities by default depending on what we know about the OS, which is ugly.

The flat structure of .c files is going to be a pain, but it's also quite likely to increase the effort needed to update between versions of libigt2.

The one sure way to make it go-gettable is to go back to dynamic linking by default, but that has the issues that you then have to make sure the version of libgit2 matches on your system, which is more work.

If you'd like to send a PR I can take a look, but I don't think I have the time to work on this myself.

[dmitshur]

Thanks for getting back to me. I see, I didn't expect those complications.

I will take them into account and keep thinking about this more.

[carlosmn]

It is deceptively complex to build libgit2, since Git is primarily a local system, and we do want to provide that without bringing in unnecessary dependencies for network. And there are all sorts of quirks for some OSs.

The go tool does have more tricks than I thought though, and it's easier than I thought to provide a way to link to a built libgit2.a without having to use the Makefile. You would still have to recompile libgit2 via make or whatever when libgit2 is updated, but compiling an updated git2go itself as a dependency should keep working, assuming that it remembers the tags which were passed on build.

[dmitshur]

The go tool does have more tricks than I thought though, and it's easier than I thought to provide a way to link to a built libgit2.a without having to use the Makefile.

I think any improvement to the process to get it closer to 100% go gettable would be a huge improvement.

You would still have to recompile libgit2 via make or whatever when libgit2 is updated,

Sorry, I don't quite understand what you mean. Couldn't you just update the Go package to have a new commit with the new prebuilt libgit2.a whenever there is an update to libgit2, so when the user does go get -u github.com/libgit2/git2go they get the new version?

[carlosmn]

There is no precompiled version of libgit2, partly for the same reasons as it's complex to build it via 'go build'. It needs to link to libraries which are available in the system and 'go build' does not let us discover that.

[dmitshur]

So what were you referring to when you said this:

The go tool does have more tricks than I thought though, and it's easier than I thought to provide a way to link to a built libgit2.a without having to use the Makefile.

[carlosmn]

I mean that once libgit2.a is built, you can make the 'go build' link to it without the need for all those variables in the file. Whenever libgit2 changes, you still have to run 'make' (or the script directly) since AFAIK you can't make 'go' execute an arbitrary command, but for any change in the git2go code itself, 'go build'/'go test' would be enough to build/test the changed Go code.

[carlosmn]

There is of course the possibility of using 'go generate' to run some go command (e.g. the "script" to check for thread-locking) and we could create one which calls 'make' or CMake directly, but generate won't be run automatically, so we'd just be changing one extra command with a different one that's spelt slightly differently.

[carlosmn]

Now that libgit2 v0.22 is released, I've made a v22 branch which will dynamically link to libgit2. You can use

import "gopkg.in/libgit2/git2go.v22"

to get a version of git2go which will work against the system libraries and thus the go tool will like.

[kron4eg]

@carlosmn so, with merge v22 into master now static linking is eliminated... which is very unfortunately.

[kron4eg]

@carlosmn any suggestions to workaround?

[carlosmn]

As master follows the releases, there is no need for static linking. If you would like that, you can use the next branch at whichever commit you prefer.

[dmitshur]

Have you considered this and decided it won't be done?

[carlosmn]

See this issue and the README for how this package can be built with the standard Go toolchain.