cmd/go: optimize CGO caching and dead code removal

[Keithcat1]

I'm writing this with the assumption that each CGO-generated wrapper for each C function is stored in its own C file.
Editing and compiling a CGO package can take a minute or 2, even for one little change.
It should be possible to cache the generated .o files and only recompile them if the code for that function changes.
When linking the CGO wrappers, Go should avoid linking unused wrappers to reduce size.

[ianlancetaylor]

One C file is generated for each Go file that uses import "C". It is not the case that we use a separate C file for each C function wrapper. Though I suppose we could if it seems useful.

CC @bcmills @jayconrod

[jayconrod]

Related #9887

Currently, I think we cache the generated .c files but not the .o files produced by the C compiler.

We should have a separate action for each .c file compilation, and we should cache the outputs of those actions.

[ianlancetaylor]

Note that proper caching of C object files require detecting whether any included .h files change. This does arise when people update to a newer version of some C dependency.

[jayconrod]

Good point. We should be incorporating hashes of included .h files into the cache keys for compiled .a files at least. I don't believe we currently are.

[Keithcat1]

Maybe we could only run the C compiler when the package is actually being compiled in order to get the values of C constants and make sure C functions are called properly.
Then, at link time, the Go linker linkes the entire program instead of using the C linker. We could probably use syscall to call C functions at runtime from CGO Go code wrappers.
This would have several benefits. Faster linking speed since no C linker is being invoked, the Go linker probably wouldn't need to spend time building a C linker compatible object file, dead code elimination could probably be done even for statically linked C code (at least on a per object file basis), and cross compiling might be supported if you have static libraries for other platforms. The Go package "github.com/veandco/go-sdl2/sdl" provides static libraries for all the platforms SDL2 runs on.
The linker would need support for reading a few different formats of object files though.

[ianlancetaylor]

We run the C linker, rather than using only the Go linker, because the Go linker doesn't support C++ global constructors or C++ exception handling or a few other cases that the C linker handles. We could support those things in the Go linker, but it's not trivial.

[docxplusgmoon]

@Keithcat1 How you solved this issue?

[Keithcat1]

By putting CGO code in sub-packages so if you make a change to the main package, it won't recompile those packages that use CGO unless you changed them

[cherrymui]

The Go linker does support reading C object files in various formats (ELF, Mach-O, PE, etc.) on some platforms. But as @ianlancetaylor pointed out, the support of C linking is limited, and it doesn't support fancy C/C++ features. If you're sure your C code doesn't use any of those, and you're targeting a platform that supports internal linking, you can try passing -ldflags=-linkmode=internal, then the Go linker will link the C code and produce an executable without using the C linker. (Note: that linking may fail, use with caution.)

That does not get deadcode elimination on C code automatically, as C object semantics is section based, not function based. Perhaps you could try using C compiler options -ffunction-sections and -fdata-sections. (Go linker's support for it is not well-tested, use with caution.)