cmd/cgo: add support for pre-compiled cgo packages

[eliasnaur]

This is the GitHub issue for my half-baked proposal posted on golang-dev.

Inspired by #35721 (comment)
and my own desire for easily cross-compiling Gio programs, I wonder whether it is
feasable to re-use cgo processing through pre-generation of the cgo tool output.

Packages that import "C" require quite a lot of support software:
a native toolchain, development libraries, headers. Invoking the host compiler
is also slow. For example, cross-compiling Cgo programs for macOS and iOS is particularly
difficult (and perhaps legally impossible), and cross-compiling for Android requires
a ~700MB NDK.

On the other hand, it's not always possible to avoid C dependencies: Gio uses
system libraries that are impossible to implement in Go.

One option is .syso files (https://github.com/golang/go/wiki/GcToolchainTricks),
pre-compiled native code that the Go linker knows how to link in. However .syso
files are too low level: they can't link to shared (system) libraries and there is no
Go<->C bridging for making calls across the language barrier safe.

So I wonder: can you imagine a Goldilocks option where a package can have access
all Cgo features through pre-generated cgo files? It's similar to the old binary-only package
feature, but only for the Cgo parts. I hope that C's much more stable ABI makes a pre-generated
cgo file format viable than pre-built Go binary packages.

I wouldn't mind if the hypothetical cgo file format changes once in a while, say every Go
release, as long as I could supply multiple files covering the Go releases I support. If none
of the pre-generated files are usable, the Go toolchain falls back to just running cgo processing.

As a strawman, I suggest a way to invoke

$ go tool cgo build <package>

which then outputs, say, <package_go115>.cgo. If is later built, possibly on a different
machine, the Go toolchain will look for *.cgo files in the package directory and use one of them
if possible, skipping cgo processing altogether.

Advantages

With pre-compiled Cgo the building process is faster, and I can avoid having the headers and development libraries available for my target platform, needing only a native toolchain for external linking.

What's really interesting is then adding support for -linkmode=internal when Cgo is used, alleviating the need for a native toolchain completely.

In other words, pre-generated cgo could give Cgo programs some of the nice cross-platform properties of pure Go programs.

[FiloSottile]

This is exactly what I wanted to do ad-hoc for SQLite, would love to see it!

https://twitter.com/FiloSottile/status/1245831899290820608

I think there wouldn't even be any need for new .cgo file types, it should be possible to just generate .go and .syso files that build with the current toolchain. It might even be relatively stable: the ABI is fixed, so the only unstable internal is asmcgocall.

This actually means it could be a fully external tool, at least for prototyping. Indeed, the thing that needs toolchain support is #38918, which is where things get interesting.

[eliasnaur]

@FiloSottile I had your SQLite and x509 work in mind when filing this proposal :) Perhaps you'd like to join the next golang-tools call May 13th for further discussion?

[eliasnaur]

I think there wouldn't even be any need for new .cgo file types, it should be possible to just generate .go and .syso files that build with the current toolchain. It might even be relatively stable: the ABI is fixed, so the only unstable internal is asmcgocall.

This actually means it could be a fully external tool, at least for prototyping.

Indeed. What I'd like to achieve first is some indication that work towards pre-generation may end up in the main repository. I don't fancy maintaining an out-of-tree tool.

[ianlancetaylor]

I don't really understand how to do this in general.

cgo packages can and do refer to external libraries; presumably we would have to find all of those external libraries and make them available somewhere. But the libraries are going to vary a lot. How do we find them and where do we put them?

Some of those libraries are written in C++. C++ code requires extra work from the linker to support things like global constructors and destructors and exception handling. If we can't invoke the external linker, we have to implement all of that code in cmd/link, and we have to maintain it as C++ changes.

Maybe I'm missing something, but while I can see how this could work in the simplest cases, I don't see how it could work in general.

[eliasnaur]

I don't really understand how to do this in general.

cgo packages can and do refer to external libraries; presumably we would have to find all of those external libraries and make them available somewhere. But the libraries are going to vary a lot. How do we find them and where do we put them?

Can we ask the native toolchain while pre-pregenerating, in a similar way Cgo currently asks the toolchain about the structure of native types, functions etc.?

Some of those libraries are written in C++. C++ code requires extra work from the linker to support things like global constructors and destructors and exception handling. If we can't invoke the external linker, we have to implement all of that code in cmd/link, and we have to maintain it as C++ changes.

Maybe I'm missing something, but while I can see how this could work in the simplest cases, I don't see how it could work in general.

My use-case is system libraries, which are usually designed to have simple C (or Objective-C) interfaces as well as being impossible to replace with Go by definition. Having pre-generation work on a best-effort basis would be ok for me, I would simply not provide pre-generated files for the platforms/libraries that are not (yet) supported.

[dosgo]

I am looking forward to this feature, but there seems to be no progress.

[dosgo]

Is there a simple way to implement it? Pack the intermediate files generated by gcc into a zip, and continue to use a third-party linker. The main trouble with C language cross-compilation is various dependencies and libraries. I don't know much about compiling and linking, and I might understand it incorrectly.

[ianlancetaylor]

If you have to have the third party linker and the third party libraries anyhow, then I don't think this idea is going to help very much.