proposal: runtime: explicitly allow pinning of strings and slices

[mknyszek]

The cgo pointer passing rules (and previously the Pinner documentation) states something like:

[Only pointers that are] the result of calling new, taking the address of a composite literal, or taking the address of a local variable[ may be passed to C and/or pinned].

This wasn't really accurate in practice and also placed restrictions on the API that made it harder to allow new cases in the future (see #62356 and #63768).

Even still, the Pinner API explicitly rejects non-pointer types today, but there are ways to work around that for some common use-cases like strings and slices:

• For slices, simply passing &s[0] works.
• For strings, passing unsafe.StringData(s) works.

While strings require unsafe, it's particularly surprising that slice elements just work given the documentation. Also, given that Go strings can already be passed as _GoString_ to C functions, it makes sense that they should be included as well.

I propose allowing both strings and slices as valid arguments to Pinner. The semantics are clear: pin the underlying memory for the string or the slice backing store. Callers are also able to pin any pointers inside the slice backing store (but Pinner will not do this for them automatically). We should make clear that C code is only allowed to access and manipulate the part of the slice backing store that has been pinned; going out-of-bounds on the slice passed to C is not OK.

[gopherbot]

Change https://go.dev/cl/558578 mentions this issue: runtime: support Pinner.Pin for string type

[gopherbot]

Change https://go.dev/cl/558715 mentions this issue: runtime: support Pinner.Pin for slice type

[qiulaidongfeng]

I wrote the implementation for that.
Unfortunately, since github PR couldn't do one CL to modify the code based on other CL, and I couldn't contribute with git codereview mail, the process of trying to split two small CL (the smaller the change, the easier the review) caused some noise.

[gopherbot]

Change https://go.dev/cl/559015 mentions this issue: doc/go1.23: document runtime.Pinner.Pin change

[dunglas]

I'm currently dealing with passing strings without doing a copy from Go to C to optimize FrankenPHP (Go and PHP use a similar data structure to store strings, so there is no need to copy the Go string to a C string to finally copy it to a PHP string).

An important point is that pinning string only works if the memory storing the string is allocated on the heap.

This panics (panic: runtime error: runtime.Pinner.Pin: argument is not a Go pointer):

package main

import (
	"runtime"
	"strings"
	"unsafe"
)

func main() {
	s := "s"

	p := runtime.Pinner{}
	p.Pin(unsafe.StringData(s))
}

Forcing allocation on the heap makes this snippet work:

package main

import (
	"runtime"
	"strings"
	"unsafe"
)

func main() {
	s := strings.Clone("s")

	p := runtime.Pinner{}
	p.Pin(unsafe.StringData(s))
}

This should probably be mentioned in the docs, the documentation change introduced in 916e6cd is misleading.

[randall77]

@dunglas, the "argument is not a Go pointer" part will be fixed in 1.22. See #62356.

[mknyszek]

@dunglas This'll be fixed in Go 1.22. It really should silently ignore "non-Go" pointers in general (in this case, some pointer to read-only memory in the binary). I agree it is a usability issue.

The decision in #63768 (comment) was to not backport this behavior since it's not technically a regression and there's a workaround. We could consider backporting it anyway, I suppose; the fix is pretty safe and the benefit is that the workarounds aren't necessary anymore.

[dunglas]

Awesome!

@mknyszek could you point me to the workaround, I can't find it and I have an immediate use case for it 😅

[mknyszek]

Ah, sorry, by workaround I meant strings.Clone in this particular case. Unfortunately the workaround is pretty situation-specific, but they're few in number in practice. I think it's mainly:

• Memory allocated in C. Unfortunately you just have to avoid trying to pin this memory blindly. :(
• Strings that are baked into the binary. You can work around this with strings.Clone.
• Globals that are linker-allocated. I think you can resolve this by not setting up globals directly (that is, repalce var x X = <value> with var x *X followed by an assignment to x in an init function. Note: var x *X = <value> won't work; that X value can be linker-allocated as well.

[dunglas]

Thanks for the clarification! I'll use the strings.Clone hack until we can upgrade to Go 1.22 then.