unsafe: add StringData, String, SliceData

[twmb]

Now that reflect.StringHeader and reflect.SliceHeader are officially deprecated, I think it's time to revisit adding function that satisfy the reason people have used these types. AFAICT, the reason for deprecation is that reflect.SliceHeader and reflect.StringHeader are commonly misused. As well, the types have always been documented as unstable and not to be relied upon.

We can see in Github code search that usage of these types is ubiquitous. The most common use cases I've seen are:

• converting []byte to string
• converting string to []byte
• grabbing the Data pointer field for ffi or some other niche use
• converting a slice of one type to a slice of another type

The first use case can also commonly be seen as *(*string)(unsafe.Pointer(&mySlice)), which is never actually officially documented anywhere as something that can be relied upon. Under the hood, the shape of a string is less than a slice, so this seems valid per unsafe rule (1), but this is all relying on undocumented behavior. The second use case is commonly seen as *(*[]byte)(unsafe.Pointer(&string)), which is by-default broken because the Cap field can be past the end of a page boundary (example here, in widely used code) -- this violates unsafe rule (1).

Regardless of the thought that people should never rely upon these types, people do, and they do so all over. People also rely on invalid conversions because Go has never made this easy. Part of the discussion on #19367 was about all the ways that people misuse these types today. These conversions are small tricks that can alleviate memory pressure and improve latencies and CPU usage in real programs. The use cases are real, and Go provides just enough unsafe and buggy ways of working around these problems such that now there is a large ecosystem of technically invalid code that just so happens to work.

Rather than consistently saying "don't use this", go veting somewhat, and then ducking all responsibility for buggy programs, Go should provide actual safe(ish) APIs that people can rely on in perpetuity. New functions can live in unsafe and have well documented rules around their use cases, and then Go can finally document what to do when people want this common escape hatch.

Concrete proposal

The following APIs in the unsafe package:

// StringToBytes returns s as a byte slice by performing a non-copying type conversion.
// Slices returned from this function cannot be modified.
func StringToBytes(s string) []byte

// BytesToString returns b as a string by performing a non-copying type conversion.
// The input bytes to this function cannot be modified while any string returned from
// this function is alive.
func BytesToString(b []byte) string

func DataPointer[T ~string|~[]E, E any](t T) unsafe.Pointer eliminating, because realistically a person can just do &slice[0] (although a corresponding analogue does not exist for strings)

I think unsafe.Slice covers the use case for converting between slices of different types, although I'm not 100% sure what the use case of unsafe.Slice is.

[ianlancetaylor]

Thanks, but I don't see an actual proposal here. If you want to discuss ideas, please use golang-nuts. The proposal process should be for a proposal. That is, suggest some specific functions that we should introduce. Thanks.

[twmb]

Sure, good point. I've edited my comment with an actual proposal of threetwo new functions. I think unsafe.Slice covers my the fourth common use case I mentioned above, but I'm not 100% sure.

[ianlancetaylor]

One of the main use cases of unsafe.Slice is to create a slice whose backing array is a memory buffer returned from C code or from a call such as syscall.MMap. I agree that it can be used to (unsafely) convert from a slice of one type to a slice of a different type.

[ianlancetaylor]

CC @mdempsky

[rsc]

Filed #53079. Perhaps we should undeprecate them for Go 1.19, since we don't have a replacement for all valid use cases yet. Thanks.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[bcmills]

FWIW, the proposed functions closely parallel the OfString and AsString functions in my unsafeslice package.

That package also provides best-effort mutation detection, since Go string variables are supposed to be immutable and Go programs may generally assume that variables passed as type string are never mutated until they are garbage-collected.

[twmb]

I've seen that package, and the one thing I'd not do is the mutation detection (which you gate behind the unsafe build tag). I'd expect mutation detection to be automatically done when testing / when built with -race, but not to automatically be applied in releases. I'm using the unsafe package, after all :).

I was thinking to link the package in my original writeup, since it is further evidence of the use case.

[bcmills]

At the very least, I think it's important for the documentation for the proposed functions to explicitly call out the lifetime issues, especially for BytesToString — a string used as a map key can cause arbitrary memory corruption if it is mutated while the map is still live.

[ianlancetaylor]

To fully replace reflect.StringHeader and reflect.SliceHeader, let's consider what they permit us to do. They can be used to read and/or to modify the contents of a string or a slice.

For reading, we can already extract all the elements of a slice, via &s[0], len(s), and cap(s). We can get the length of a string via len(s), but we can't get the pointer to the data.

For writing, we can create a new slice setting all the elements, via unsafe.Slice followed by a slice expression. But we can't create a new string.

So to me that suggests, in package unsafe,

// StringData returns a pointer to the bytes of a string.
// The bytes must not be modified; doing so can cause
// the program to crash or behave unpredictably.
func StringData(string) *byte

// String constructs a string value from a pointer and a length.
// The bytes passed to String must not be modified;
// doing so can cause the program to crash or behave unpredictably.
func String(*byte, int) string

These functions should remain meaningful even if we somehow change the representation of slices or strings in the future.

The restrictions on changing the bytes are unfortunate, but the fact is that people are doing these kinds of transformations today. Omitting these functions from the unsafe package doesn't mean that Go programs won't do them, it just means that they will do in ways that are sometimes even less safe.

It should be feasible to add a dynamic detector for any modifications to these bytes. This could perhaps be enabled when using the race detector. This would not be perfect but would detect egregious misuses.

The functions suggested above would then be written as

func StringToBytes(s string) []byte {
    return unsafe.Slice(unsafe.StringData(s), len(s))
}

func BytesToString(b []byte) string {
    return unsafe.String(&b[0], len(b))
}

(To be clear, the reverse is also possible: we can write String and StringData in terms of StringToBytes and BytesToString.)

[rsc]

It does seem like unsafe.String and unsafe.StringData match unsafe.Slice a bit better and are more fundamental operations
than providing StringToBytes and BytesToString as the primitives.
I wonder if we should add unsafe.SliceData as well (code often has to work around len 0 using &s[0]).

[mdempsky]

I'd like to suggest we reconsider the original proposal from #19367: to add new unsafe.StringHeader and unsafe.SliceHeader types, to handle the remaining advanced use cases that are supported by reflect.{String,Slice}Header but aren't covered by unsafe.Slice.

Concretely, I propose adding:

package unsafe

type StringHeader struct {
    Data *byte
    Len int
}

type SliceHeader[Elem any] struct {
    Data *Elem
    Len, Cap int
}

and allowing conversions between string and unsafe.StringHeader, and also between []Elem and unsafe.SliceHeader[Elem].

Converting an invalid unsafe.{String,Slice}Header (e.g., Len > Cap, or Data==nil and Len>0) into a normal string or slice type should fail, at least in -d=checkptr mode. I'm leaning towards making it a run-time panic (like unsafe.Slice) because the failure conditions are easy to specify/detect, but simply leaving it undefined (like unsafe.Add) seems not unreasonable too.

N.B., my original #19367 proposal also allowed conversions between *string and *unsafe.StringHeader, etc. I think we could still allow that (e.g., particularly to help users transition away from reflect.{String,Slice}Header), but I think it's less error-prone (and marginally better for escape analysis) if we encourage users to construct an unsafe.StringHeader, value convert it to string, and then store that into memory; rather than creating a *unsafe.StringHeader and individually assigning fields in memory.

[ianlancetaylor]

My concern about StringHeader and SliceHeader is that it locks all possible implementations into either using those exact headers or doing strange contortions in the compiler.

What can we do with StringHeader and SliceHeader that we can't do with Slice, SliceData, String, and StringData?

[mdempsky]

After thinking about it more, I'm inclined to agree that {Slice,String}{,Data} builtin functions are the way to go. As you say, they support the same functionality. I think that will be easier on tools authors than extending conversion semantics too.

[rsc]

It sounds like we've converged on considering unsafe.StringData, unsafe.String, and unsafe.SliceData.
Does anyone object to those?

[rsc]

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

[rsc]

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

[gopherbot]

Change https://go.dev/cl/428757 mentions this issue: reflect: deprecate (Slice|String)Header

[gopherbot]

Change https://go.dev/cl/429755 mentions this issue: all: transfer reflect.{SliceHeader, StringHeader} to unsafeheader.{Slice, String}

[cespare]

@mdempsky is this going to go in for 1.20? I've tried to follow the large number of associated PRs but I can't really tell what the current state is.

[ianlancetaylor]

@cespare As far as I know, they are in. What's missing is the docs. I just pinged that CL.

If there is anything else missing, please let me know. Thanks.

[gopherbot]

Change https://go.dev/cl/449537 mentions this issue: spec: document the new unsafe functions SliceData, String, and StringData

[griesemer]

Both the spec and the unsafe package documentation are now done. Closing.

[simon28082]

In practice I found out about bytes to string using the following method will result in converted values that are not as expected, to over debug my application I got this

unsafe.String(&bytes[0], byteLen)

[mitar]

I still miss that StringToBytes and BytesToString are not directly provided. As seen here, currently implementation of those functions requires:

import "unsafe"

func String2ByteSlice(str string) []byte {
	if str == "" {
		return nil
	}
	return unsafe.Slice(unsafe.StringData(str), len(str))
}

func ByteSlice2String(bs []byte) string {
	if len(bs) == 0 {
		return ""
	}
	return unsafe.String(unsafe.SliceData(bs), len(bs))
}

Those edge cases with empty strings/slices make current API very verbose as you have to check for those edge cases. Could those two functions be added to standard library?

On a related note, should examples like TextMarshalJSON one in json package use such conversion between strings and bytes? Or is that already automatically detected by the compiler and optimized?

[thepudds]

Hi @mitar, FWIW, I don't know that the standard library would want to make that more convenient.

Note that Go 1.22 will have some nice enhancements in this area, including recognizing in many cases when a []byte is not modified after a string->[]byte conversion to enable a safe zero-copy conversion (issue #2205).

Here is an example (with comparison of 1.21 vs. tip via godbolt):

func f() {
	s := "hello"
	x := []byte(s) // zero-copy string->[]byte conversion
	_ = x
}

[database64128]

Those edge cases with empty strings/slices make current API very verbose as you have to check for those edge cases.

@mitar I don't think these checks are needed. unsafe.StringData's documentation says:

For an empty string the return value is unspecified, and may be nil.

So it's perfectly safe to feed the returned pointer directly to unsafe.Slice. Same for unsafe.String(unsafe.SliceData(b), len(b)).

[go101]

@database64128
With the simple one-line way, converting different zero strings might result different slice values:

package main

import "unsafe"

func main() {
	{
		var s = ""
		println(s == "") // true
		var x = unsafe.Slice(unsafe.StringData(s), len(s))
		println(x == nil) // true
	}
	{
		var s = "go"[:0]
		println(s == "") // true
		var x = unsafe.Slice(unsafe.StringData(s), len(s))
		println(x == nil) // false
	}
}

It might satisfy the needs of some cases, but not all.

The situations of converting from zero-length slices to strings are comparatively simpler.
But the verbose way makes sure that the result string will never keep the pointer of the underlying array of blank slices
(it is hard to say this is a benefit, just a personal preference),

package main

import "unsafe"

func ByteSlice2String(bs []byte) string {
	if len(bs) == 0 {
		return ""
	}
	return unsafe.String(unsafe.SliceData(bs), len(bs))
}

func main() {
	var x = make([]byte, 0, 1024)
	var s = unsafe.String(unsafe.SliceData(x), len(x))
	println(s == "") // true
	println(*(*uintptr)(unsafe.Pointer(&s))) // <a non-zero addr>
	
	var s2 = ByteSlice2String(x)
	println(s2 == "") // true
	println(*(*uintptr)(unsafe.Pointer(&s2))) // 0
}