Native cgo support infrastructure

[shadowmint]

So, as a summary for this issue, I tried to use https://github.com/therecipe/qt, with dep, and it didn't work; although it downloaded the dependency, it didn't generate all the required files to actually be able to run.

In order to work, the qt binding uses a setup script qtbuild to setup various go packages and generate the various files that cgo can bind to.

This is not an uncommon issue; http://akrennmair.github.io/golang-cgo-slides summarizes the issue with cgo succinctly:

Usually, you want to access functions outside of libc.
Linking to external libraries required; two possibilities:

- Add CGO_LDFLAGS to Makefile
- Embed in .go source file using #cgo directive (preferred)

...and that's pretty much the state of play.

For example:

• The SDL2 library requires you to have system level SDL libraries installed and on the (https://github.com/veandco/go-sdl2/blob/master/sdl/sdl_wrapper.h)

• The sqlite binding embeds the entire sqlite library as an amalgamation in the source of the library (https://github.com/mattn/go-sqlite3/blob/master/sqlite3-binding.c)

• The port audio binding uses pkg-config to find and link to the system library (https://github.com/gordonklaus/portaudio/blob/master/portaudio.go#L15)

• The tensor flow binding requires you to run some custom scripts to install things before you get started with it (https://github.com/tensorflow/tensorflow/blob/master/tensorflow/go/README.md)

• As mentioned above, QT uses a custom build script (https://github.com/therecipe/qt/blob/master/cmd/qtsetup/qtsetup.go)

...solving all the problems with cgo is entirely out of the scope of what dep sets out to do, but perhaps we can standardize on a way of supporting the basic cgo workflow.

What do other languages do?

If you look at python (setup.py), rust (build.rs), node (post-install scripts), even java (maven plugins) you'll see a commonality in the package managers; arbitrary post-install script execution to do various things, including compile C dependencies.

However, if we look closely at a recent example, rust, this model hasn't been particularly successful; although build.rs can execute arbitrary code, and you can have 'build dependencies' to import helper scripts, ultimately the configuration and usage of these scripts is brittle and difficult to get right.

The guidelines in http://doc.crates.io/build-script.html are complex; and people get it wrong. Despite the best efforts of its contributors https://github.com/PistonDevelopers/Piston continues to generate a stream of 'I tried it but it didn't compile...' problems for people.

What do we actually want?

Before looking at any specific solution, perhaps we can look at what we actually want to achieve.

In a nutshell, what we need is some kind of 'extra step':

• 1. Dep installs packages in vendor/
• 2. ??? <--- Extra step here
• 3. You can issue a build command against code using cgo and it works.

That 'extra step' needs to be able to, potentially any of:

• Check if build tools are available on the system.
• Actually build c libraries.
• Generate go code.
• Prompt the user with custom install instructions to install system level libraries.

For example, ages ago I wrote this demo for generating a static C library and embedding it using cgo to generate a 'single binary' go executable that embeds its own c dependency.

However, to do that you need to do this:

mkdir build
cd build
cmake ..
cmake --build .
cd ..
go build
./demo

There's no facility to turn this into a package that you can go get; there's no means to inform the user that there is a custom step they need to run, and no means to automatically run it.

So what now?

Well, so that's where we are now.

You could argue that supporting this is beyond the scope of dep, and that this is an issue that needs to be raised at a high level in the go toolchain itself, and that's a fair argument.

However, I think we could plausibly look at a way that dep can help standardize the way packages with dependencies are crafted, since the goal of dep is, itself, to standardize the way we handle dependencies in general.

[shadowmint]

One possible solution...

So, now I've kind of framed the issue, here's one possible solution I imagine:

It really seems like 'arbitrary code execution' is the only feasible solution to the 'extra step'; but without both the security risks that dep ensure might do something unknown, and the hideous mess of having a setup.py that is both a data file and arbitrary code execution at the same time.

If we consider a dependency chain as being something akin to a migration list to a database; you need to run the migrations before you can use the code you've installed in vendor, but you can pick when and which ones you run; then perhaps we can standardize on allowing packages to specify arbitrary executable commands that need to be executed in a specific order, without actually executing them automatically.

I imagine a workflow, something along the lines of:

dep ensure

dep status
PROJECT                         CONSTRAINT  VERSION  REVISION  LATEST  PKGS USED DEPS
github.com/go-sql-driver/mysql  *           v1.3     a0583e0   v1.3    1         1

dep deps
PROJECT                         PENDING SCRIPT
github.com/go-sql-driver/mysql  true    ./bin/000_go_sql_driver_config

dep deps execute github.com/go-sql-driver/mysql
Building: ./bin/000_go_sql_driver_config
Executing: ./bin/000_go_sql_driver_config

dep deps
PROJECT                         PENDING SCRIPT
github.com/go-sql-driver/mysql  false    ./bin/000_go_sql_driver_config

Where a project can optionally choose to specify some path in it's manifest with a main() which can perform some arbitrary execution step; perhaps tell the user how to install a system library, perhaps build and generate some go code.

dep would purely have the responsibility in this case to:

• Generate specifically (package named) binaries in a specific folder when dep deps build X was invoked.

• Track what named binaries had been executed.

• Reset the execution flag for a package's build when a new version is installed.

You could then:

• Install all dependencies using dep without executing any build steps.
• Generate a specific list of build steps that are required by each package.
• Execute the required setup for each package as a one time option as required.

That's just my $0.02 on how I'd like to see it work, but its not a trivial problem and it'd be really good to at least start a conversation about how to handle it.

One thing I would love to see is dep as having a consistent behavior that all packages can conform to, and stop the current 'do whatever' approach that libraries using cgo seem to take.

[ghost]

the qt binding uses a setup script qtbuild to setup

you probably mean qtsetup.

Let him know about the issue 😸 @therecipe

[therecipe]

@m8m5
Thanks for letting me know :)

Yes, I would love to be able to specify a build script to make things more simple to setup.

I think these build script(s) will be mainly used to solve these two cgo related problems:

1. To download (and compile) additional C dependencies such as C headers, libs and maybe (cross) compilers (if these dependencies can't live inside the repo)
2. To generate *.go files with cgo CFLAGS, LDFLAGS, ... (if one can't use pkg-config)

I don't know if this adds anything to the discussion, as this is the wrong repo.
But actually, all I would need is just a hook that gets called automatically after "go get ...", to make the repo go gettable.

[sdboyer]

related questions in rust-lang/cargo#3816 and rust-lang/cargo#3815

(also ugh how has it been 2.5 months since @shadowmint opened this already 😢 )

[sdboyer]

eh, as long as i'm commenting, i should also probably make explicit two things:

1. this isn't on the immediate-term roadmap, as this is something that can likely later be added without changing existing logic, and we have a bunch of critical path things
2. i haven't really explored the implications of the ideas @shadowmint laid out, but my gut says they're pointing us in the right direction