Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/cgo: document rules for pointer sharing #12116

Closed
rsc opened this issue Aug 11, 2015 · 11 comments

Comments

@rsc
Copy link
Contributor

commented Aug 11, 2015

We need to document what the rules are.

@rsc rsc added this to the Go1.6 milestone Aug 11, 2015

@cznic

This comment has been minimized.

Copy link
Contributor

commented Aug 11, 2015

If not already considered a part of the issue, let me please ask for one more, I believe closely related thing above the subject: Pinning guarantees rules, ie. either defining an explicit mechanism, maybe an annotation //go:pinned, for making things with static addresses; or having an indirect mechanism, like eg. that b := make([]byte, n, m) under certain specified conditions produces an immovable &b[0], etc. Actually supporting a byte slice is enough (for me). What about automatically, always and everywhere pinning any unsafe.Pointer???

@cespare

This comment has been minimized.

Copy link
Contributor

commented Aug 11, 2015

Does this issue supersede #8310?

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Aug 11, 2015

@cznic Do you have a need for pinning beyond passing pointers between Go and C?

I don't think we are likely to implement a //go:pinned annotation. For one thing, pinning is a concept associated with specific pointers, and annotations in comments are a concept associated with specific variables (or functions).

@dr2chase

This comment has been minimized.

Copy link
Contributor

commented Aug 12, 2015

If we were going to pin things, I would pin any memory allocated that had a cgo-related type (C.uint, C.char, etc) or array of cgo-related type. So your example might be
b := make([]C.char, n, m)
Since the purpose of these C types is to pass them (and pointers to them) to C, and passing them to C is the main reason for pinning, this seems to me like a good match.

@chai2010

This comment has been minimized.

Copy link
Contributor

commented Aug 12, 2015

@dr2chase Good idea! +10086

@cznic

This comment has been minimized.

Copy link
Contributor

commented Aug 12, 2015

@ianlancetaylor

Do you have a need for pinning beyond passing pointers between Go and C?

Yes I do. I'm looking into a cgo alternative which translates, when possible, C code to Go code with equivalent semantics. The translated code uses the unsafe package, yet it has only the same inherent memory unsafeness of the original C code, nothing to lose here; but the call-into-C overhead is gone. Also, installing such translated-to-Go C library is as easy as go get.

In this scenario, p = malloc(n); can be translated to something like p = unsafe.Pointer(&make([]byte, n)[0]), plus keeping p reachable. *p escapes and will not be put in the stack and thus not moved by the current Go compilers, AFAIK. But there are no guarantees that the same will hold for any future/other Go implementations.

I assume that moving the backing array of the above example by the Go runtime at GC is actually fine as long as p stays as a translated-from-C variable only - then it would get correctly updated. The problem arises when p itself is stored in the C heap space, now a Go byte slice backing array, and becomes invisible to the Go runtime wrt to updating due to moving heap items around, which is so far a completely legal thing to do.

@dr2chase

This comment has been minimized.

Copy link
Contributor

commented Aug 12, 2015

I'm not sure how much experience you have with this sort of thing, but it might be worth looking at how this was done in other languages (e.g., Modula-3 or Modula-2+, Susan Owicki wrote a paper on this a long long time ago http://dl.acm.org/citation.cfm?id=567539 ). There is a standard pattern that has worked in other languages that use copying (object-moving) garbage collectors -- stuff for native code is allocated by malloc, if the GC is supposed to backstop its lifetime management you wrap that malloc-pointer into a heap-allocated object with a finalizer that calls free.

One thing to be aware of is that Go stacks can move, and Cgo contains a handshake to ensure that stuff passed to C is stored on the GC'd heap (because right now, that memory does not move) -- this means that if you write code that (you think) avoids a heap allocation for something you pass to C, that actually you are getting both a copy AND a heap allocation.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Aug 12, 2015

@cznic That's a pretty specialized use case, and you can, and probably should, resolve it by implementing your own syscall.Mmap based memory allocator (after all, the C heap is not garbage collected, so why should your translation of the C heap be garbage collected). At the moment I don't see it as a convincing reason to add a complex feature to the Go GC.

@cznic

This comment has been minimized.

Copy link
Contributor

commented Aug 13, 2015

@ianlancetaylor

resolve it by implementing your own syscall.Mmap based memory allocator...

Yep. Of course it would be easier to use the existing allocator. But not only that. The intention is to, in the next step, be able to also produce (slower) code, which however is not using package unsafe and qualifies (some of) the translated C libs to be used in restricted environments like the GAE/GCE.

I don't have yet a working proof of concept for the 'next step', but so far I haven't figured out any other blocking issue - except for the pinning problem (only with future/other Go implementations). It can be worked around, but at the cost of slowing down the "safe" mode even more.

Don't get me wrong, I'm not really pushing for the pinning feature. I am asking for it, but please read it as a request for it to be considered and if seen fit for future implementation, then its specification should be, IMO, part of the rules @rsc asked to be defined in this issue.

While I do agree that the implementation might be complex, there exist GC languages which support pinning, proving it's feasible.

@dr2chase
I have no access to the Owicki's paper. Maybe her technique is the same as discussed elsewhere?

... if you write code that (you think) avoids a heap allocation for something you pass to C ...

What I am trying to do does not use cgo/does not pass things to C.

@dr2chase

This comment has been minimized.

Copy link
Contributor

commented Aug 13, 2015

@cznic Owicki's paper discusses how you interoperate traced and untraced stuff in a language without dropping down into Unsafe. It influenced the design of Modula-2+, which influenced the design of Modula-3, then along came Java, and we forgot all that stuff because who needs anything better than JNI? The rules follow from the needs of the garbage collector and tracing -- traced can contain pointers to untraced, but not vice-versa, because the GC (in most designs) does not include untraced memory in its root set -- untraced memory is treated as a likely source of bugs and is not to be relied on (and decades of experience confirms this), and this is especially true in the usual use case for untraced memory, which is "interfacing with some other language that doesn't enforce type safety or array bounds". It's a perfectly reasonable model for Cgo.

I can't quite figure out how you end up needing pinning without using unsafe, if that's your eventual goal. I suspect you'll find yourself wanting to use unsafe to convert pointer-to-foo into a singleton slice of foo, also, but that doesn't create a need for pinning.

@rsc

This comment has been minimized.

Copy link
Contributor Author

commented Nov 10, 2015

Duplicate of #12416

@rsc rsc closed this Nov 10, 2015

@golang golang locked and limited conversation to collaborators Nov 10, 2016

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
7 participants
You can’t perform that action at this time.