Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/cgo: make identical C types identical Go types across packages #13467

Open
bcmills opened this issue Dec 3, 2015 · 58 comments
Open

cmd/cgo: make identical C types identical Go types across packages #13467

bcmills opened this issue Dec 3, 2015 · 58 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime.
Milestone

Comments

@bcmills
Copy link
Contributor

bcmills commented Dec 3, 2015

https://golang.org/cmd/cgo/ says:

Cgo translates C types into equivalent unexported Go types. Because the translations are unexported, a Go package should not expose C types in its exported API: a C type used in one Go package is different from the same C type used in another.

While that's a convenient workaround for allowing access to struct fields and other names that would otherwise be inaccessible as Go identifiers, it greatly complicates the process of writing Go APIs for export to C callers. The Go code to produce and/or manipulate C values must be essentially confined to a single package.

It would be nice to remove that restriction: instead of treating C types as unexported local types, we should treat them as exported types in the "C" package (and similarly export the lower-case names that would otherwise be unexported).

@ianlancetaylor ianlancetaylor changed the title cgo: make identical C types identical Go types across packages cmd/cgo: make identical C types identical Go types across packages Dec 3, 2015
@mdempsky
Copy link
Contributor

mdempsky commented Dec 3, 2015

Somewhat related: does C require types to be defined the same way across translation units? Reading through C99, it seems to only require that objects and functions with external linkage need to have the same type across translation units (6.2.7), but I can't find anything per se that disallows for example "typedef int foo;" in one translation unit and "typedef unsigned foo;" in another (assuming they don't in turn lead to incompatible object/function declarations).

(Not to say that cgo needs to support that.)

@ianlancetaylor
Copy link
Contributor

C and C++ permit the same name to designate different types in different compilation units.

@joegrasse
Copy link

I definitely agree that this restriction should be lifted. It makes it very hard to break up your code to promote maintainability. It would make more sense for a C.int, etc to be a C.int everywhere, just as an int is an int everywhere.

@rsc
Copy link
Contributor

rsc commented Dec 28, 2015

I don't believe we should lift this restriction. It is explicitly not a goal to make it possible to expose C types directly in Go package APIs. As Ian said, it's not even clear this is sound.

@rsc rsc closed this as completed Dec 28, 2015
@bcmills
Copy link
Contributor Author

bcmills commented Jan 4, 2016

It is explicitly not a goal to make it possible to expose C types directly in Go package APIs.

The point of this request is not to write general-purpose Go packages using C types. It is to enable the creation of support libraries for Go packages that call C functions and/or Go packages that export C APIs with richer structure than primitive pointers and integers. (For example: one might want to return a protocol buffer from a Go function to a C or C++ caller without making more than one copy of the marshaled data. That operation is complex enough that it needs a support library, and because it needs to manipulate Go types it must be written in Go.)

As Ian said, it's not even clear this is sound.

The request is to make "identical" C types identical Go types, not to make C types "with the same name" identical Go types. I believe it is sound provided that we enforce that the C types are actually identical.

@rsc
Copy link
Contributor

rsc commented Jan 4, 2016

OK, I'm happy to reopen this, but I have no idea how to do it. It seems fundamentally at odds with Go's package system. I'm happy to look at implementation proposals though.

@rsc rsc reopened this Jan 4, 2016
@rsc rsc added this to the Unplanned milestone Jan 4, 2016
@14rcole
Copy link

14rcole commented Jun 27, 2016

For the time being, is there a workaround for this? Maybe using an interface that can represent the same struct from different packages?

@bcmills
Copy link
Contributor Author

bcmills commented Jun 27, 2016

You can work around it in one direction (going from the C types in another package to the C types in the current package) using reflect and unsafe.Pointer. The same technique may be possible in the other direction too.

If you want to add back some of the type-safety at run time, you can use the reflect package to iterate over the struct fields to verify that they're compatible.

@joegrasse
Copy link

Just pondering if aliases or whatever comes out of #18130 would help with this problem.

@bcmills
Copy link
Contributor Author

bcmills commented Dec 2, 2016

@joegrasse I don't think type aliases per se would help with the general problem: either way, you end up needing one "canonical definition" for each type, and if you've already got a canonical definition then you don't need to be able to refer to it by different names.

However, it might at least solve the subproblem of making the Go types for C typedefs have the same aliasing structure as the C types. (I'm honestly not sure whether that's currently the case: it hadn't even occurred to me to check.)

@rsc
Copy link
Contributor

rsc commented Dec 6, 2016

@joegrasse, probably not, but if we do #16623 (let compiler know more about cgo) then the compiler would be in a position to resolve this, if we wanted to.

(Fixed issue number, sorry.)

@joegrasse
Copy link

@rsc, do you mind double checking the issue number. I think you might have mistyped it.

@ianlancetaylor
Copy link
Contributor

@joegrasse Russ meant #16623.

@joegrasse
Copy link

Thanks

@14rcole
Copy link

14rcole commented Dec 6, 2016

@bcmills Forgive my backtracking, but I don't understand why aliasing wouldn't solve the problem. I thought that the problem with C structs in Go was that the compiler views a C struct within a package differently as the same C struct outside of the package. Therefore you don't have a "cannonical definition" for that type. Wouldn't aliasing the C struct as a Go struct help with the problem?

@bcmills
Copy link
Contributor Author

bcmills commented Dec 6, 2016

@14rcole Consider this program:

foo.h:

typedef struct {
  int i;
} Foo;

foo/foo.go:

package foo

// #include "foo.h"
import "C"

func Frozzle(x *C.Foo) {
  …
}

bar.h:

typedef struct {
  int i;
} Bar;

bar/bar.go:

package bar

// #include "bar.h"
import "C"

func Bozzle(y *C.Bar) { foo.Frozzle(y) }

This program should compile: C.Foo in package bar is the same C type (a typedef of a struct with the same definition) as C.Bar in package foo. However, that would require cgo to write the definitions of C.Foo and C.Bar such that they are aliases for the same underlying type. Since the type includes the x field which is currently unexported, there is no package in which that type could be defined.

There are other possible ways to solve the problem (e.g. by rewriting field names so that they are always exported and combining all of the C declarations into one package), but they involve more than just a suitable application of aliases.

@rsc
Copy link
Contributor

rsc commented Dec 6, 2016

Also, an alias has to exist in one package P and point to another package Q. That implies P imports Q (to point at it). In the general version of the problem in this issue, both P and Q define some C type and don't know about each other at all. Then some other package M (for main) imports both and tries to mix one with the other. There's no way for aliases per se to solve this problem, because P and Q need to continue not knowing about each other, and M can't change the definitions in P and Q.

@mdempsky
Copy link
Contributor

mdempsky commented Dec 6, 2016

Following the latest proposed solution in #16623 (but not strictly dependent upon it), the compilers could treat declarations for _Cfoo_bar as though they were declared from a synthetic "C" package. I believe we could also easily turn off symbol visibility rules for this package (e.g., so that lowercase struct fields are still accessible).

Then I think usual type identity rules would just work as desired, and usual type-reexporting information would help to catch ODR (one-definition rule) violations across C compilation units.

@bcmills
Copy link
Contributor Author

bcmills commented Dec 6, 2016

@mdempsky

the compilers could treat declarations for _Cfoo_bar as though they were declared from a synthetic "C" package

It can't be just one package, unfortunately. In a valid program, a type C.X can legitimately mean two different things in two different compilation units.

We could perhaps do some sort of name-mangling to disambiguate, though. For example, we could encode the complete C type definition in the mangled name and have the compiler treat all C-mangled names as being in the same package.

The remaining concern with that approach is what to do with reflect. (If we've mangled the names to avoid collisions, should reflect report the mangled names, the colliding names, or something else entirely?)

@minux
Copy link
Member

minux commented Dec 7, 2016 via email

@joegrasse
Copy link

Here is a very contrived example of how I came across this issue. I believe it to be a more simplistic case then what @bcmills and @rsc have both discussed above (although I could be wrong).

Consider the package:
cp/cp.go

package cp

import "C"

func CTest(name *C.char) string {
	return C.GoString(name)
}

and the program:
ct/main.go

package main

import (
	"fmt"

	"cp"
)
import "C"

func main() {
	s := C.CString("Hello World") // This needs to be freed later
	fmt.Println(cp.CTest(s))
}

When you try and build ct/main.go, you get the following error.

# ct
./main.go:12: cannot use s (type *C.char) as type *cp.C.char in argument to cp.CTest

Before coming across this isssue, I would have thought that this program should compile, because cp.CTest takes a *C.char and s is a *C.char. I am not creating any new types, just using the basic C char type. For some reason though, *C.char in package cp becomes a *cp.C.char.

@rsc
Copy link
Contributor

rsc commented Dec 7, 2016

After #16623 we can figure out what the semantics should be here. It could be that we only support this for built-in C types like char/int/etc.

@AlexRouSg
Copy link
Contributor

AlexRouSg commented Jun 1, 2017

After #16623 we can figure out what the semantics should be here. It could be that we only support this for built-in C types like char/int/etc.

What about for types in external C libs? For external libs we can be pretty sure C.foo will always be C.foo, maybe even prefix it MyLib so like C.MyLib.foo or C.MyLib_foo?

@AlexRouSg
Copy link
Contributor

@MaVo159

Was just about to describe a very similar problem and you beat me to it.

@jimmyfrasche
Copy link
Member

A related issue is one large library that has a number of "modules" defined by optional header files.

It's natural to want to make these true separate packages in Go. This is doable with an internal/ package that implements everything which is used by packages that expose the actual API.

But this means that the implementation of every optional module is included in the build artifact regardless of what actually gets imported, which depending on the library/module can be rather large.

It's not a show-stopping issue, generally, but it sounds like this would fix it.

@AlexRouSg
Copy link
Contributor

My use case is more like an union of the problems described by @MaVo159 and @jimmyfrasche.

A package using multiple intercommunicating C libraries that wants users to expand on it's functionality by calling the C library's functions directly. Something similar to plugin behaviour.

@AlexRouSg
Copy link
Contributor

AlexRouSg commented Sep 21, 2017

Just thought of a problem that will affect the implementation and its usefulness.

It is common for some C libraries to hide the fields of some structs. For example SDL defines typedef struct SDL_Window SDL_Window; in the public headers and struct SDL_Window{...} in the internal headers so you can't directly access the fields.

I think in these cases we would have no choice but to use unsafe.Pointer if we want to pass them around? Unless there is some way cgo can get the strut definition without including the internal headers as we would most likely be missing the proper #defines for it.

@bcmills
Copy link
Contributor Author

bcmills commented Sep 21, 2017

@AlexRouSg

I think in these cases we would have no choice but to use unsafe.Pointer if we want to pass them around?

Those should be fine, as long as the compiler treats blank-named fields as exported for the purpose of computing type identity (but I should verify that).

The we can define an “opaque” struct type for each incomplete type, and as long as the type is incomplete in all of the Go packages they'll remain equivalent.

The generated code would look something like:

type _Ctype_struct_SDL_Window = struct {
	_ struct{} `cgo: struct SDL_Window`
}

(But see also #19487: defining Go types at all for incomplete types is a bit of a thorny problem.)

02p01r added a commit to 02p01r/gorocksdb that referenced this issue Apr 20, 2020
02p01r added a commit to 02p01r/gorocksdb that referenced this issue Apr 20, 2020
mstemm added a commit to mstemm/libsinsp-plugin-sdk-go that referenced this issue Aug 2, 2021
Instead of passing events as unsafe.Pointer, use C.ss_event_struct
when possible.

It's not possible to use C.ss_struct_event in the conversion function
as the C.* pseudo types are package-specific (see
golang/go#13467)
mstemm added a commit to mstemm/libsinsp-plugin-sdk-go that referenced this issue Aug 3, 2021
Since it's a pain to use the C pseudo-types that result from the C
package across golang packages (see
golang/go#13467), make the helper function
more general, working with golang types.

The plugin needs some shim code to convert between the C types for the
plugin_extract_* functions and the functions used by the async
helper. We can possibly automate that, will try soon.
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jul 13, 2022
@cavokz
Copy link

cavokz commented Aug 18, 2022

Came across this issue while writing a Python extension in Go, packaging the heavy lifting of the direct Python C API usage into something more idiomatic and easier to (re)use from Go.

Hypothetical Python minimal (dummy) generic and reusable package:

package python

// #cgo pkg-config: python3-embed
// #include <Python.h>
import "C"

type Module struct {
	Name string
	...
}

// other helpers for exporting to Python

func (m Module) Create() *C.PyObject {
	var PyModuleDef_HEAD_INIT = C.PyModuleDef_Base{
		C.PyObject{1, nil}, nil, 0, nil,
	}

	var module = C.PyModuleDef{
		PyModuleDef_HEAD_INIT,
		C.CString(m.Name), nil, -1, nil,
		nil, nil, nil, nil,
	}

	return C.PyModule_Create2(&module, C.PYTHON_API_VERSION)
}

Hypothetical user:

package main

import "python"

// #cgo pkg-config: python3-embed
// #include <Python.h>
import "C"

....

//export PyInit_shape
func PyInit_shape() *C.PyObject {
	m := new(python.Module)
	m.Name = "shape"

	// export stuff to Python

	return m.Create()
}

func main() {}

The build fails:

./shape.go:29:9: cannot use m.Create() (value of type *python._Ctype_struct__object) as type *_Ctype_struct__object in return statement

func PyInit_shape() *C.PyObject {...} must be defined by the user, the name PyInit_shape is known only to the user, depends on the name of the extension being built, and cannot be delegated to the "Python" package.

I fail to see a way to improve the Go-Python relationship with this limitation in place.

EDIT: I used unsafe.Pointer to successfully work around this inconvenient.

@hwittenborn
Copy link

I'm about to write some bindings to a fairly big Go library, and it's looking like I'm going to have to include ~80% of my code in a single file, because I can't pass C types across different Go packages in my project. This is quite a burden on the maintainability for my project going forward - languages like Rust can pass C types across packages, so I don't see much of a reason why Go can't be supporting it.

@hwittenborn
Copy link

hwittenborn commented Feb 16, 2023

Just adding on an example, but if I define all of my C types in a separate file like shown below:

// `ctypes/ctypes.go`
package ctypes

// #include <stdlib.h>
import "C"
import "fmt"

func PrintChar(str *C.char) {
    fmt.Println(C.GoString(str))
}

I think it'd be reasonable to assume I can call them in a separate file like so:

// `main.go`
package main

// #include <stdlib.h>
import "C"
import "example.com/ctypes"

func main() {
    ctypes.PrintChar(C.CString("Hello World!"))
}

But alas this fails with a type error:

# example.com
./main.go:8:22: cannot use (_Cfunc_CString)("Hello World!") (value of type *_Ctype_char) as *ctypes._Ctype_char value in argument to ctypes.PrintChar

Obviously these types are the same. And again, in a language like Rust this would be trivial to implement.

@cavokz
Copy link

cavokz commented Feb 16, 2023

What you can do is wrapping your C types in Go types:

type MyString struct {
  str *C.char
}

You can then pass them around different modules/packages as long as you modify them only from the package defining them.

Note: do not forget to free memory allocated by C.CString!

@hwittenborn
Copy link

I should've mentioned it, but I'm exporting the functions in question via FFI.

The situation I'm running into is more like the code below (I'd share the code I'm using directly, but I'm trying to keep it private right now for personal reasons - it's quite close to the below code in terms of how functions are called though):

// `ctypes/ctypes.go`
package ctypes

// #include <stdlib.h>
import "C"
import "fmt"

func PrintChar(str *C.char) {
    fmt.Println(C.GoString(str))
}
// `main.go`
package main

// #include <stdlib.h>
import "C"
import "example.com/ctypes"

func main() {
}

//export PrintString
func PrintString(str *C.char) {
    ctypes.PrintChar(str)
}

And then I'd like to access PrintString from a C program. But alas the above example fails to compile with the same error I had in my previous comment.

What you can do is wrapping your C types in Go types:

That also creates a compile-time error about the Go struct not being exportable. I should've mentioned how I was using my data previously though.

You can then pass them around different modules/packages as long as you modify them only from the package defining them.

I also need it able to be accessed across packages (packages are just different files, correct?), but it's of my knowledge that such isn't possible yet, correct?

arkadijs added a commit to arkadijs/go-webrtcvad that referenced this issue Feb 7, 2024
To use VAD instance in some contexts, for example: a struct field, `sync.Pool`, etc. it's necessary to reference the type explicitly.
golang/go#13467

Signed-off-by: Arkadi Shishlov <arkadi.shishlov@gmail.com>
arkadijs added a commit to arkadijs/go-webrtcvad that referenced this issue Feb 7, 2024
To use VAD instance in some context, for example: a struct field, `sync.Pool`, etc. it's necessary to reference the type explicitly.
golang/go#13467

Signed-off-by: Arkadi Shishlov <arkadi.shishlov@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime.
Projects
None yet
Development

No branches or pull requests