cmd/cgo: make identical C types identical Go types across packages

[bcmills]

https://golang.org/cmd/cgo/ says:

Cgo translates C types into equivalent unexported Go types. Because the translations are unexported, a Go package should not expose C types in its exported API: a C type used in one Go package is different from the same C type used in another.

While that's a convenient workaround for allowing access to struct fields and other names that would otherwise be inaccessible as Go identifiers, it greatly complicates the process of writing Go APIs for export to C callers. The Go code to produce and/or manipulate C values must be essentially confined to a single package.

It would be nice to remove that restriction: instead of treating C types as unexported local types, we should treat them as exported types in the "C" package (and similarly export the lower-case names that would otherwise be unexported).

[mdempsky]

Somewhat related: does C require types to be defined the same way across translation units? Reading through C99, it seems to only require that objects and functions with external linkage need to have the same type across translation units (6.2.7), but I can't find anything per se that disallows for example "typedef int foo;" in one translation unit and "typedef unsigned foo;" in another (assuming they don't in turn lead to incompatible object/function declarations).

(Not to say that cgo needs to support that.)

[ianlancetaylor]

C and C++ permit the same name to designate different types in different compilation units.

[joegrasse]

I definitely agree that this restriction should be lifted. It makes it very hard to break up your code to promote maintainability. It would make more sense for a C.int, etc to be a C.int everywhere, just as an int is an int everywhere.

[rsc]

I don't believe we should lift this restriction. It is explicitly not a goal to make it possible to expose C types directly in Go package APIs. As Ian said, it's not even clear this is sound.

[bcmills]

It is explicitly not a goal to make it possible to expose C types directly in Go package APIs.

The point of this request is not to write general-purpose Go packages using C types. It is to enable the creation of support libraries for Go packages that call C functions and/or Go packages that export C APIs with richer structure than primitive pointers and integers. (For example: one might want to return a protocol buffer from a Go function to a C or C++ caller without making more than one copy of the marshaled data. That operation is complex enough that it needs a support library, and because it needs to manipulate Go types it must be written in Go.)

As Ian said, it's not even clear this is sound.

The request is to make "identical" C types identical Go types, not to make C types "with the same name" identical Go types. I believe it is sound provided that we enforce that the C types are actually identical.

[rsc]

OK, I'm happy to reopen this, but I have no idea how to do it. It seems fundamentally at odds with Go's package system. I'm happy to look at implementation proposals though.

[14rcole]

For the time being, is there a workaround for this? Maybe using an interface that can represent the same struct from different packages?

[bcmills]

You can work around it in one direction (going from the C types in another package to the C types in the current package) using reflect and unsafe.Pointer. The same technique may be possible in the other direction too.

If you want to add back some of the type-safety at run time, you can use the reflect package to iterate over the struct fields to verify that they're compatible.

[joegrasse]

Just pondering if aliases or whatever comes out of #18130 would help with this problem.

[bcmills]

@joegrasse I don't think type aliases per se would help with the general problem: either way, you end up needing one "canonical definition" for each type, and if you've already got a canonical definition then you don't need to be able to refer to it by different names.

However, it might at least solve the subproblem of making the Go types for C typedefs have the same aliasing structure as the C types. (I'm honestly not sure whether that's currently the case: it hadn't even occurred to me to check.)

[rsc]

@joegrasse, probably not, but if we do #16623 (let compiler know more about cgo) then the compiler would be in a position to resolve this, if we wanted to.

(Fixed issue number, sorry.)

[joegrasse]

@rsc, do you mind double checking the issue number. I think you might have mistyped it.

[ianlancetaylor]

@joegrasse Russ meant #16623.

[joegrasse]

Thanks

[14rcole]

@bcmills Forgive my backtracking, but I don't understand why aliasing wouldn't solve the problem. I thought that the problem with C structs in Go was that the compiler views a C struct within a package differently as the same C struct outside of the package. Therefore you don't have a "cannonical definition" for that type. Wouldn't aliasing the C struct as a Go struct help with the problem?

[bcmills]

@14rcole Consider this program:

foo.h:

typedef struct {
  int i;
} Foo;

foo/foo.go:

package foo

// #include "foo.h"
import "C"

func Frozzle(x *C.Foo) {
  …
}

bar.h:

typedef struct {
  int i;
} Bar;

bar/bar.go:

package bar

// #include "bar.h"
import "C"

func Bozzle(y *C.Bar) { foo.Frozzle(y) }

This program should compile: C.Foo in package bar is the same C type (a typedef of a struct with the same definition) as C.Bar in package foo. However, that would require cgo to write the definitions of C.Foo and C.Bar such that they are aliases for the same underlying type. Since the type includes the x field which is currently unexported, there is no package in which that type could be defined.

There are other possible ways to solve the problem (e.g. by rewriting field names so that they are always exported and combining all of the C declarations into one package), but they involve more than just a suitable application of aliases.

[rsc]

Also, an alias has to exist in one package P and point to another package Q. That implies P imports Q (to point at it). In the general version of the problem in this issue, both P and Q define some C type and don't know about each other at all. Then some other package M (for main) imports both and tries to mix one with the other. There's no way for aliases per se to solve this problem, because P and Q need to continue not knowing about each other, and M can't change the definitions in P and Q.

[mdempsky]

Following the latest proposed solution in #16623 (but not strictly dependent upon it), the compilers could treat declarations for _Cfoo_bar as though they were declared from a synthetic "C" package. I believe we could also easily turn off symbol visibility rules for this package (e.g., so that lowercase struct fields are still accessible).

Then I think usual type identity rules would just work as desired, and usual type-reexporting information would help to catch ODR (one-definition rule) violations across C compilation units.

[bcmills]

@mdempsky

the compilers could treat declarations for _Cfoo_bar as though they were declared from a synthetic "C" package

It can't be just one package, unfortunately. In a valid program, a type C.X can legitimately mean two different things in two different compilation units.

We could perhaps do some sort of name-mangling to disambiguate, though. For example, we could encode the complete C type definition in the mangled name and have the compiler treat all C-mangled names as being in the same package.

The remaining concern with that approach is what to do with reflect. (If we've mangled the names to avoid collisions, should reflect report the mangled names, the colliding names, or something else entirely?)

[minux]

What if two packages both define their own version of incompatible type T in cgo preamble? I think any solutions to this issue must check to make sure the two cgo types are indeed compatible.

[joegrasse]

Here is a very contrived example of how I came across this issue. I believe it to be a more simplistic case then what @bcmills and @rsc have both discussed above (although I could be wrong).

Consider the package:
cp/cp.go

package cp

import "C"

func CTest(name *C.char) string {
	return C.GoString(name)
}

and the program:
ct/main.go

package main

import (
	"fmt"

	"cp"
)
import "C"

func main() {
	s := C.CString("Hello World") // This needs to be freed later
	fmt.Println(cp.CTest(s))
}

When you try and build ct/main.go, you get the following error.

# ct
./main.go:12: cannot use s (type *C.char) as type *cp.C.char in argument to cp.CTest

Before coming across this isssue, I would have thought that this program should compile, because cp.CTest takes a *C.char and s is a *C.char. I am not creating any new types, just using the basic C char type. For some reason though, *C.char in package cp becomes a *cp.C.char.

[rsc]

After #16623 we can figure out what the semantics should be here. It could be that we only support this for built-in C types like char/int/etc.

[AlexRouSg]

After #16623 we can figure out what the semantics should be here. It could be that we only support this for built-in C types like char/int/etc.

What about for types in external C libs? For external libs we can be pretty sure C.foo will always be C.foo, maybe even prefix it MyLib so like C.MyLib.foo or C.MyLib_foo?

[AlexRouSg]

@MaVo159

Was just about to describe a very similar problem and you beat me to it.

[jimmyfrasche]

A related issue is one large library that has a number of "modules" defined by optional header files.

It's natural to want to make these true separate packages in Go. This is doable with an internal/ package that implements everything which is used by packages that expose the actual API.

But this means that the implementation of every optional module is included in the build artifact regardless of what actually gets imported, which depending on the library/module can be rather large.

It's not a show-stopping issue, generally, but it sounds like this would fix it.

[AlexRouSg]

My use case is more like an union of the problems described by @MaVo159 and @jimmyfrasche.

A package using multiple intercommunicating C libraries that wants users to expand on it's functionality by calling the C library's functions directly. Something similar to plugin behaviour.

[AlexRouSg]

Just thought of a problem that will affect the implementation and its usefulness.

It is common for some C libraries to hide the fields of some structs. For example SDL defines typedef struct SDL_Window SDL_Window; in the public headers and struct SDL_Window{...} in the internal headers so you can't directly access the fields.

I think in these cases we would have no choice but to use unsafe.Pointer if we want to pass them around? Unless there is some way cgo can get the strut definition without including the internal headers as we would most likely be missing the proper #defines for it.

[bcmills]

@AlexRouSg

I think in these cases we would have no choice but to use unsafe.Pointer if we want to pass them around?

Those should be fine, as long as the compiler treats blank-named fields as exported for the purpose of computing type identity (but I should verify that).

The we can define an “opaque” struct type for each incomplete type, and as long as the type is incomplete in all of the Go packages they'll remain equivalent.

The generated code would look something like:

type _Ctype_struct_SDL_Window = struct {
	_ struct{} `cgo: struct SDL_Window`
}

(But see also #19487: defining Go types at all for incomplete types is a bit of a thorny problem.)

[cavokz]

Came across this issue while writing a Python extension in Go, packaging the heavy lifting of the direct Python C API usage into something more idiomatic and easier to (re)use from Go.

Hypothetical Python minimal (dummy) generic and reusable package:

package python

// #cgo pkg-config: python3-embed
// #include <Python.h>
import "C"

type Module struct {
	Name string
	...
}

// other helpers for exporting to Python

func (m Module) Create() *C.PyObject {
	var PyModuleDef_HEAD_INIT = C.PyModuleDef_Base{
		C.PyObject{1, nil}, nil, 0, nil,
	}

	var module = C.PyModuleDef{
		PyModuleDef_HEAD_INIT,
		C.CString(m.Name), nil, -1, nil,
		nil, nil, nil, nil,
	}

	return C.PyModule_Create2(&module, C.PYTHON_API_VERSION)
}

Hypothetical user:

package main

import "python"

// #cgo pkg-config: python3-embed
// #include <Python.h>
import "C"

....

//export PyInit_shape
func PyInit_shape() *C.PyObject {
	m := new(python.Module)
	m.Name = "shape"

	// export stuff to Python

	return m.Create()
}

func main() {}

The build fails:

./shape.go:29:9: cannot use m.Create() (value of type *python._Ctype_struct__object) as type *_Ctype_struct__object in return statement

func PyInit_shape() *C.PyObject {...} must be defined by the user, the name PyInit_shape is known only to the user, depends on the name of the extension being built, and cannot be delegated to the "Python" package.

I fail to see a way to improve the Go-Python relationship with this limitation in place.

EDIT: I used unsafe.Pointer to successfully work around this inconvenient.

[hwittenborn]

I'm about to write some bindings to a fairly big Go library, and it's looking like I'm going to have to include ~80% of my code in a single file, because I can't pass C types across different Go packages in my project. This is quite a burden on the maintainability for my project going forward - languages like Rust can pass C types across packages, so I don't see much of a reason why Go can't be supporting it.

[hwittenborn]

Just adding on an example, but if I define all of my C types in a separate file like shown below:

// `ctypes/ctypes.go`
package ctypes

// #include <stdlib.h>
import "C"
import "fmt"

func PrintChar(str *C.char) {
    fmt.Println(C.GoString(str))
}

I think it'd be reasonable to assume I can call them in a separate file like so:

// `main.go`
package main

// #include <stdlib.h>
import "C"
import "example.com/ctypes"

func main() {
    ctypes.PrintChar(C.CString("Hello World!"))
}

But alas this fails with a type error:

# example.com
./main.go:8:22: cannot use (_Cfunc_CString)("Hello World!") (value of type *_Ctype_char) as *ctypes._Ctype_char value in argument to ctypes.PrintChar

Obviously these types are the same. And again, in a language like Rust this would be trivial to implement.

[cavokz]

What you can do is wrapping your C types in Go types:

type MyString struct {
  str *C.char
}

You can then pass them around different modules/packages as long as you modify them only from the package defining them.

Note: do not forget to free memory allocated by C.CString!

[hwittenborn]

I should've mentioned it, but I'm exporting the functions in question via FFI.

The situation I'm running into is more like the code below (I'd share the code I'm using directly, but I'm trying to keep it private right now for personal reasons - it's quite close to the below code in terms of how functions are called though):

// `ctypes/ctypes.go`
package ctypes

// #include <stdlib.h>
import "C"
import "fmt"

func PrintChar(str *C.char) {
    fmt.Println(C.GoString(str))
}

// `main.go`
package main

// #include <stdlib.h>
import "C"
import "example.com/ctypes"

func main() {
}

//export PrintString
func PrintString(str *C.char) {
    ctypes.PrintChar(str)
}

And then I'd like to access PrintString from a C program. But alas the above example fails to compile with the same error I had in my previous comment.

What you can do is wrapping your C types in Go types:

That also creates a compile-time error about the Go struct not being exportable. I should've mentioned how I was using my data previously though.

You can then pass them around different modules/packages as long as you modify them only from the package defining them.

I also need it able to be accessed across packages (packages are just different files, correct?), but it's of my knowledge that such isn't possible yet, correct?