Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unsafe: add Slice(ptr *T, len anyIntegerType) []T #19367

Closed
mdempsky opened this issue Mar 2, 2017 · 125 comments
Closed

unsafe: add Slice(ptr *T, len anyIntegerType) []T #19367

mdempsky opened this issue Mar 2, 2017 · 125 comments

Comments

@mdempsky
Copy link
Member

@mdempsky mdempsky commented Mar 2, 2017

reflect.SliceHeader and reflect.StringHeader are clumsy to use because their Data fields have type uintptr instead of unsafe.Pointer.

This proposal is to add types unsafe.Slice and unsafe.String as replacements. They would be declared just like their package reflect analogs, except with unsafe.Pointer-typed Data fields:

type Slice struct {
    Data Pointer
    Len int
    Cap int
}

type String struct {
    Data Pointer
    Len int
}

Additionally, I suggest that for the purposes of type conversion, we treat that string and unsafe.String have the same underlying type, and also []T and unsafe.Slice. For example, these would be valid:

func makestring(p *byte, n int) string {
    // Direct conversion of unsafe.String to string.
    return string(unsafe.String{unsafe.Pointer(p), n})
}

func memslice(p *byte, n int) (res []byte) {
    // Direct conversion of *[]byte to *unsafe.Slice, without using unsafe.Pointer.
    s := (*unsafe.Slice)(&res)
    s.Data = unsafe.Pointer(p)
    s.Len = n
    s.Cap = n
    return
}

While the same results can be achieved using unsafe.Pointer conversions, by using direct conversions the compiler can provide a little extra type safety.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Mar 2, 2017

If we do this, we should figure out a way to exempt these new types from the Go 1 compatibility guarantee, so that we can change the representation of strings and slices in the future. I'm not sure how best to do that.

Loading

@cespare
Copy link
Contributor

@cespare cespare commented Mar 2, 2017

@ianlancetaylor reflect.SliceHeader and reflect.StringHeader already try:

It cannot be used safely or portably and its representation may change in a later release.

but the compat doc itself gives a strong exemption for all of unsafe:

Packages that import unsafe may depend on internal properties of the Go implementation. We reserve the right to make changes to the implementation that may break such programs.

ISTM that unsafe.{Slice,String} would already be exempted sufficiently.

Loading

@rsc
Copy link
Contributor

@rsc rsc commented Mar 6, 2017

Go 2 seems like the time to think about this (and reflect.SliceHeader etc).

-rsc for @golang/proposal-review

Loading

@bcmills
Copy link
Member

@bcmills bcmills commented Mar 23, 2017

This proposal seems a bit redundant with #13656.

How much of the use-case is "create a string or slice aliasing C memory" vs. "manipulate existing strings and slices by tweaking header fields unsafely"?

Loading

@mdempsky
Copy link
Member Author

@mdempsky mdempsky commented Oct 18, 2019

I'd like to suggest renewing consideration of this proposal for Go 1.14. I think it will be useful for users trying to address issues flagged by -d=checkptr.

I'll also offer a counter-proposal that I think better addresses most end user needs in a more ergonomic manner:

package unsafe

func Slice(ptr *ArbitraryType, len, cap int) []ArbitraryType
func String(ptr *byte, len int) string

[Edit: As discussed below, I'm now in favor of combining Slice's len/cap parameter into a single parameter.]

This is a little less versatile than exposing the Header types, but I think it will minimize typing for most users, while also providing better type safety.

We could also do both this proposal and my original one, if we want to still offer the full flexibility of the Header types. In that case, I would suggest renaming the types to SliceHeader and StringHeader, and reserve the shorter Slice and String identifiers for the constructor functions.

Loading

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Oct 18, 2019

I like that counter proposal API.

Loading

@mdempsky
Copy link
Member Author

@mdempsky mdempsky commented Oct 18, 2019

A few additional thoughts to add to my counter proposal:

  1. We should decide what happens when len < 0 or cap < len. I'm leaning towards panic, but maybe we should just leave it unspecified/undefined.

    Edit: ptr == nil && len > 0 is another case to consider.

    Edit 2: Also, len > MAXWIDTH / unsafe.Sizeof(*ptr).

  2. The functions would be builtins; in particular, users can't write f := unsafe.String; f(...).

  3. The cap argument to unsafe.Slice can be optional; if it's omitted, the len argument is used. (Just like make([]T, n) is shorthand for make([]T, n, n).)

  4. Perhaps the int parameters should actually follow the same goofy semantics that make([]T, n, m) follows. (I.e., make([]T, uint64(10), int8(20)) is valid, even though uint64 and int8 aren't normally assignable to int.)

  5. Since unsafe.String would be a builtin, it could evaluate to an untyped string.

Loading

@bcmills
Copy link
Member

@bcmills bcmills commented Oct 18, 2019

That API is closer to what I had suggested in https://golang.org/issue/13656#issuecomment-303216308, and we've been using that variant within Google for a couple of years now without complaints.

If the type desired for the slice does not match the pointer that the user has (for example, if one is a cgo-generated type and the other is a native Go type), I'm assuming that the caller could do something like:

	var s = unsafe.Slice((*someGoType)(unsafe.Pointer(cPtr)), len, cap)

to set the element type?

Loading

@bcmills
Copy link
Member

@bcmills bcmills commented Oct 18, 2019

We should decide what happens when len < 0 or cap < len. I'm leaning towards panic, but maybe we should just leave it unspecified/undefined.

I would leave it unspecified, but panic is a fine implementation of “unspecified”.

Perhaps the int parameters should actually follow the same goofy semantics that make([]T, n, m) follows.

That would certainly smooth out the call site in the (overwhelmingly common) case that len and/or cap is a C.size_t.

Loading

@mdempsky
Copy link
Member Author

@mdempsky mdempsky commented Oct 18, 2019

If the type desired for the slice does not match the pointer that the user has (for example, if one is a cgo-generated type and the other is a native Go type), I'm assuming that the caller could do something like:

	var s = unsafe.Slice((*someGoType)(unsafe.Pointer(cPtr)), len, cap)

to set the element type?

Yeah, that's my thought. If a user wants to convert *T into []U, then I think it's reasonable to require an explicit conversion there.

Loading

@mdempsky
Copy link
Member Author

@mdempsky mdempsky commented Oct 18, 2019

I would leave it unspecified, but panic is a fine implementation of “unspecified”.

Ack, though my concern is if we panic by default, then users might come to rely on it panicking and not write their own checking.

It would be easy to put the panic behind -d=checkptr though.

Loading

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Oct 18, 2019

func Slice(ptr *ArbitraryType, len, cap int) []ArbitraryType

Can we instead do:

func Slice(ptr *ArbitraryType, len int[, cap int]) []ArbitraryType

... with an optional cap. Where omitting cap means cap == len?

Loading

@mdempsky
Copy link
Member Author

@mdempsky mdempsky commented Oct 18, 2019

@bradfitz Yeah, that's my additional thought #3 above. :)

Loading

@bcmills
Copy link
Member

@bcmills bcmills commented Oct 18, 2019

if we panic by default, then users might come to rely on it panicking and not write their own checking.

Hmm, good point. We could make it a throw! 😉

Or we could make it a panic in ordinary code but a throw under -race or -d=checkptr. (The important thing, I think, is to vary it just enough that it causes tests to fail in some reasonably-common configuration.)

Loading

gopherbot pushed a commit that referenced this issue Apr 23, 2021
Updates #19367.
Updates #40481.

Change-Id: Id2b2d2e3e716f91f0dd9e5102689a1ba90a819e4
Reviewed-on: https://go-review.googlesource.com/c/go/+/312213
Run-TryBot: Matthew Dempsky <mdempsky@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Trust: Matthew Dempsky <mdempsky@google.com>
Trust: Robert Griesemer <gri@golang.org>
Reviewed-by: Robert Griesemer <gri@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
gopherbot pushed a commit that referenced this issue Apr 23, 2021
This is a port of CL 312212, CL 312591 (except check_test.go), and
CL 312790 to types2.

Updates #19367.
Updates #40481.

Change-Id: I58ba0b0dad157baba3f82c909d5eb1268b931be4
Reviewed-on: https://go-review.googlesource.com/c/go/+/312511
Trust: Matthew Dempsky <mdempsky@google.com>
Trust: Robert Griesemer <gri@golang.org>
Run-TryBot: Matthew Dempsky <mdempsky@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Robert Griesemer <gri@golang.org>
@mdempsky
Copy link
Member Author

@mdempsky mdempsky commented Apr 29, 2021

This is done.

Loading

@mdempsky mdempsky closed this Apr 29, 2021
gopherbot pushed a commit that referenced this issue May 2, 2021
Updates #19367.
Updates #40481.

Change-Id: Iabd2afdd0d520e5d68fd9e6dedd013335a4b3886
Reviewed-on: https://go-review.googlesource.com/c/go/+/312214
Run-TryBot: Matthew Dempsky <mdempsky@google.com>
Trust: Matthew Dempsky <mdempsky@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Cuong Manh Le <cuong.manhle.vn@gmail.com>
Reviewed-by: Keith Randall <khr@golang.org>
@yaxinlx
Copy link

@yaxinlx yaxinlx commented Jun 11, 2021

The docs for unsafe.Slice says:

If ptr is nil or len is negative at run time, a run-time panic occurs.

But should it returns a nil slice instead if ptr is nil and len is zero?

Loading

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jun 11, 2021

@yaxinlx I don't think so. That would complicate the code for a case that I think would rarely arise. For the cases where it does happen, it's easy enough for the code to check the pointer for nil before calling unsafe.Slice.

Loading

@yaxinlx
Copy link

@yaxinlx yaxinlx commented Jun 11, 2021

, it's easy enough for the code to check the pointer for nil before calling unsafe.Slice.

It could, but it is a little inconvenient. And it is some inconsistent.

Loading

@mdempsky
Copy link
Member Author

@mdempsky mdempsky commented Jun 11, 2021

But should it returns a nil slice instead if ptr is nil and len is zero?

I had considered that, and I still haven't ruled it out. But we had to pick something, and I don't think it's obvious returning nil in that case is necessarily better. It's also easier to relax an overly strict API than to restrict an overly lax one.

My intuition is it the inconvenience, if any, will be minor. However, if actual use demonstrates that returning nil for unsafe.Slice((*T)(nil), 0) is what users almost always want, I'm open to relaxing the rules to allow that.

Loading

@bcmills
Copy link
Member

@bcmills bcmills commented Jun 11, 2021

I don't think it's obvious returning nil in that case is necessarily better.

FWIW, I think it is obviously better — unsafe.Slice constructs a slice with the given pointer and length, and the slice with pointer 0 and length 0 is exactly the nil slice (https://play.golang.org/p/hsIOG1wYBhc).

(Compare Ousterhout ch. 10, although I agree that it's not always clear.)

It's also easier to relax an overly strict API than to restrict an overly lax one.

If that's the case, the documentation for unsafe.Slice should say “ptr must be non-nil and len must be non-negative”, rather than promising to panic. (A panic in Go is a defined control-flow construct, not a programmer error.)

That would leave open the possibility of changing the behavior in the future, whereas promising to panic does not.

Loading

@mdempsky
Copy link
Member Author

@mdempsky mdempsky commented Jun 14, 2021

The specified behavior for unsafe.Slice is that it returns (*[len]ArbitraryType)(unsafe.Pointer(ptr))[:], and the current behavior for evaluating (*[0]T)(nil)[:] is to panic: https://play.golang.org/p/v7UWr1QMR0T

I agree that unsafe.Slice((*T)(nil), 0) returning []T(nil) is what users most likely always want. What's not obvious to me is whether that justifies intentionally diverging from existing language semantics.

If others don't share my hesitancy and there's strong support for changing it to return []T(nil) in Go 1.17, I'm happy to update cmd/compile.

That would leave open the possibility of changing the behavior in the future, whereas promising to panic does not.

I think we can change the semantics as long as it's tied to the -lang used to compile the package where the unsafe.Slice call appeared? I'm also fine with changing the spec/docs wording to instead say ptr must be non-nil to leave the behavior unspecified.

(I agree panicking is defined behavior, but my feeling is we've been somewhat lenient about that when it comes to Go 1 compat. I don't have concrete examples off hand though to point to.)

Loading

@gopherbot
Copy link

@gopherbot gopherbot commented Jun 22, 2021

Change https://golang.org/cl/329925 mentions this issue: spec: unsafe.Add/Slice are not permitted in statement context

Loading

gopherbot pushed a commit that referenced this issue Jun 22, 2021
Add unsafe.Add and unsafe.Slice to the list of built-in functions
which are not permitted in statement context. The compiler and
type checker already enforce this restriction, this just fixes
a documentation oversight.

For #19367.
For #40481.

Change-Id: Iabc63a8db048eaf40a5f5b5573fdf00b79d54119
Reviewed-on: https://go-review.googlesource.com/c/go/+/329925
Trust: Robert Griesemer <gri@golang.org>
Run-TryBot: Robert Griesemer <gri@golang.org>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
Reviewed-by: Rob Pike <r@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
@twmb
Copy link
Contributor

@twmb twmb commented Jul 1, 2021

Is it worth it to add an example for how to use this in the context of converting a string to a slice? I have a feeling people will look at the API and just use

slice := unsafe.Slice((*byte)(unsafe.Pointer(&str)), len(str))

If I'm reading this correctly, the proper use is

slice := unsafe.Slice((*byte)(unsafe.Pointer((*reflect.StringHeader)(unsafe.Pointer(&str)).Data)), len(str))

which is a bit unwieldy. It may be useful to have examples on this function in general. Also, from reading the documentation, (*[len]ArbitraryType)(unsafe.Pointer(ptr))[:] does not read with any context, so is a little confusing in general. What does that line mean to me?

Loading

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jul 1, 2021

Converting a string to a slice is fairly unsafe, and not really what this function is for, so I'm would vote against an example of that.

The (*[len]ArbitraryType)(unsafe.Pointer(ptr))[:] just restates the text in code. What would you recommend to make it clearer?

Loading

@twmb
Copy link
Contributor

@twmb twmb commented Jul 1, 2021

If this is not for converting to a slice, it may be worth explicitly mentioning that.

I think garbage free string<->slice conversions are pretty valuable from a high-ish-performance perspective, e.g. hashing functions usually operate on slices, and often people have strings as input. In these cases, a library author can document that an input slice is not modified, and then offer corresponding non-copying string functions. In one of my own libraries, I explicitly document that a function uses unsafe and that the slice should not be modified by the user (docs here). I feel like this type of escape hatch is similar to Rust's unsafe code blocks, where a user of a library opts in up front to unsafe behavior: in Rust, users of unsafe functions often annotate their unsafe blocks with SAFETY: to describe how the user is not violating required invariants (example here).

I know I've seen many, many string<->slice implementations in Go libraries since 2012. Given that vet now checks for some safety of the conversion (#40701), and given the existence of go-safer, this conversion is clearly something that we know people do. I personally feel like it's well past time for Go to make this conversion a bit easier, because we know today that this is something people want, try to do, and make mistakes with (but I know this desire has been pushed back against all these years, so I don't really mind in the end).

That said, if this is still a pattern we don't want to support, then documenting that unsafe.Slice is not meant for this use case is valuable.

w.r.t. the code block, now that you mention it's just a restatement in code, it's really obvious. Maybe changing the preceding line from

The function Slice returns a slice whose underlying array starts at ptr and whose length and capacity are len:

to

The function Slice returns a slice whose underlying array starts at ptr and whose length and capacity are len. The function is equivalent to the following: may make that really obvious?

Loading

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jul 1, 2021

If this is not for converting to a slice, it may be worth explicitly mentioning that.

Well, maybe, but after all we don't usually document things to say what they are not for. We document them to say what they are for. I'm not denying that there are valid uses of converting a string to []byte, but that's not really the goal of unsafe.Slice and as you say it's awkward to use for that purpose, and it's considerably less safe than the intended use of unsafe.Slice for cgo code. So it kind of seems like a distraction. We don't to start saying all the things that unsafe.Slice isn't for. That's just my own view, of course.

I sent https://golang.org/cl/332409 with your suggestion, we'll see what other people think. Thanks.

Loading

@gopherbot
Copy link

@gopherbot gopherbot commented Jul 1, 2021

Change https://golang.org/cl/332409 mentions this issue: spec, unsafe: clarify unsafe.Slice docs

Loading

@twmb
Copy link
Contributor

@twmb twmb commented Jul 2, 2021

Thank you, that reads more clearly to me!

Good point on not mentioning what things are not for. I don't think the current docs imply what it is for (cgo?), more just that the function exists, but that's ok with me, I don't think I'm the intended user for the function at this point. :)

Loading

gopherbot pushed a commit that referenced this issue Jul 2, 2021
For #19367

Change-Id: If0ff8ddba3b6b48e2e198cf3653e73284c7572a3
Reviewed-on: https://go-review.googlesource.com/c/go/+/332409
Trust: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
Reviewed-by: Robert Griesemer <gri@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Aug 1, 2021

Change https://golang.org/cl/338949 mentions this issue: compiler, runtime: support unsafe.Add and unsafe.Slice

Loading

gopherbot pushed a commit to golang/gofrontend that referenced this issue Aug 2, 2021
For golang/go#19367
For golang/go#40481

Change-Id: I989d042a518a38fd24c0d14f9c78ae564e5799a2
Reviewed-on: https://go-review.googlesource.com/c/gofrontend/+/338949
Trust: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
@gopherbot
Copy link

@gopherbot gopherbot commented Aug 6, 2021

Change https://golang.org/cl/340549 mentions this issue: compiler: support export/import of unsafe.Add/Slice

Loading

gopherbot pushed a commit to golang/gofrontend that referenced this issue Aug 7, 2021
For golang/go#19367
For golang/go#40481

Change-Id: Id1aefd0696131842d480d9f9a5330c5ab221245a
Reviewed-on: https://go-review.googlesource.com/c/gofrontend/+/340549
Trust: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Proposals
Accepted
Linked pull requests

Successfully merging a pull request may close this issue.

None yet