Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unsafe: add Slice(ptr *T, len anyIntegerType) []T #19367

Open
mdempsky opened this issue Mar 2, 2017 · 105 comments
Open

unsafe: add Slice(ptr *T, len anyIntegerType) []T #19367

mdempsky opened this issue Mar 2, 2017 · 105 comments

Comments

@mdempsky
Copy link
Member

@mdempsky mdempsky commented Mar 2, 2017

reflect.SliceHeader and reflect.StringHeader are clumsy to use because their Data fields have type uintptr instead of unsafe.Pointer.

This proposal is to add types unsafe.Slice and unsafe.String as replacements. They would be declared just like their package reflect analogs, except with unsafe.Pointer-typed Data fields:

type Slice struct {
    Data Pointer
    Len int
    Cap int
}

type String struct {
    Data Pointer
    Len int
}

Additionally, I suggest that for the purposes of type conversion, we treat that string and unsafe.String have the same underlying type, and also []T and unsafe.Slice. For example, these would be valid:

func makestring(p *byte, n int) string {
    // Direct conversion of unsafe.String to string.
    return string(unsafe.String{unsafe.Pointer(p), n})
}

func memslice(p *byte, n int) (res []byte) {
    // Direct conversion of *[]byte to *unsafe.Slice, without using unsafe.Pointer.
    s := (*unsafe.Slice)(&res)
    s.Data = unsafe.Pointer(p)
    s.Len = n
    s.Cap = n
    return
}

While the same results can be achieved using unsafe.Pointer conversions, by using direct conversions the compiler can provide a little extra type safety.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Mar 2, 2017

If we do this, we should figure out a way to exempt these new types from the Go 1 compatibility guarantee, so that we can change the representation of strings and slices in the future. I'm not sure how best to do that.

@cespare
Copy link
Contributor

@cespare cespare commented Mar 2, 2017

@ianlancetaylor reflect.SliceHeader and reflect.StringHeader already try:

It cannot be used safely or portably and its representation may change in a later release.

but the compat doc itself gives a strong exemption for all of unsafe:

Packages that import unsafe may depend on internal properties of the Go implementation. We reserve the right to make changes to the implementation that may break such programs.

ISTM that unsafe.{Slice,String} would already be exempted sufficiently.

@rsc
Copy link
Contributor

@rsc rsc commented Mar 6, 2017

Go 2 seems like the time to think about this (and reflect.SliceHeader etc).

-rsc for @golang/proposal-review

@bcmills
Copy link
Member

@bcmills bcmills commented Mar 23, 2017

This proposal seems a bit redundant with #13656.

How much of the use-case is "create a string or slice aliasing C memory" vs. "manipulate existing strings and slices by tweaking header fields unsafely"?

@mdempsky
Copy link
Member Author

@mdempsky mdempsky commented Oct 18, 2019

I'd like to suggest renewing consideration of this proposal for Go 1.14. I think it will be useful for users trying to address issues flagged by -d=checkptr.

I'll also offer a counter-proposal that I think better addresses most end user needs in a more ergonomic manner:

package unsafe

func Slice(ptr *ArbitraryType, len, cap int) []ArbitraryType
func String(ptr *byte, len int) string

[Edit: As discussed below, I'm now in favor of combining Slice's len/cap parameter into a single parameter.]

This is a little less versatile than exposing the Header types, but I think it will minimize typing for most users, while also providing better type safety.

We could also do both this proposal and my original one, if we want to still offer the full flexibility of the Header types. In that case, I would suggest renaming the types to SliceHeader and StringHeader, and reserve the shorter Slice and String identifiers for the constructor functions.

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Oct 18, 2019

I like that counter proposal API.

@mdempsky
Copy link
Member Author

@mdempsky mdempsky commented Oct 18, 2019

A few additional thoughts to add to my counter proposal:

  1. We should decide what happens when len < 0 or cap < len. I'm leaning towards panic, but maybe we should just leave it unspecified/undefined.

    Edit: ptr == nil && len > 0 is another case to consider.

    Edit 2: Also, len > MAXWIDTH / unsafe.Sizeof(*ptr).

  2. The functions would be builtins; in particular, users can't write f := unsafe.String; f(...).

  3. The cap argument to unsafe.Slice can be optional; if it's omitted, the len argument is used. (Just like make([]T, n) is shorthand for make([]T, n, n).)

  4. Perhaps the int parameters should actually follow the same goofy semantics that make([]T, n, m) follows. (I.e., make([]T, uint64(10), int8(20)) is valid, even though uint64 and int8 aren't normally assignable to int.)

  5. Since unsafe.String would be a builtin, it could evaluate to an untyped string.

@bcmills
Copy link
Member

@bcmills bcmills commented Oct 18, 2019

That API is closer to what I had suggested in https://golang.org/issue/13656#issuecomment-303216308, and we've been using that variant within Google for a couple of years now without complaints.

If the type desired for the slice does not match the pointer that the user has (for example, if one is a cgo-generated type and the other is a native Go type), I'm assuming that the caller could do something like:

	var s = unsafe.Slice((*someGoType)(unsafe.Pointer(cPtr)), len, cap)

to set the element type?

@bcmills
Copy link
Member

@bcmills bcmills commented Oct 18, 2019

We should decide what happens when len < 0 or cap < len. I'm leaning towards panic, but maybe we should just leave it unspecified/undefined.

I would leave it unspecified, but panic is a fine implementation of “unspecified”.

Perhaps the int parameters should actually follow the same goofy semantics that make([]T, n, m) follows.

That would certainly smooth out the call site in the (overwhelmingly common) case that len and/or cap is a C.size_t.

@mdempsky
Copy link
Member Author

@mdempsky mdempsky commented Oct 18, 2019

If the type desired for the slice does not match the pointer that the user has (for example, if one is a cgo-generated type and the other is a native Go type), I'm assuming that the caller could do something like:

	var s = unsafe.Slice((*someGoType)(unsafe.Pointer(cPtr)), len, cap)

to set the element type?

Yeah, that's my thought. If a user wants to convert *T into []U, then I think it's reasonable to require an explicit conversion there.

@mdempsky
Copy link
Member Author

@mdempsky mdempsky commented Oct 18, 2019

I would leave it unspecified, but panic is a fine implementation of “unspecified”.

Ack, though my concern is if we panic by default, then users might come to rely on it panicking and not write their own checking.

It would be easy to put the panic behind -d=checkptr though.

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Oct 18, 2019

func Slice(ptr *ArbitraryType, len, cap int) []ArbitraryType

Can we instead do:

func Slice(ptr *ArbitraryType, len int[, cap int]) []ArbitraryType

... with an optional cap. Where omitting cap means cap == len?

@mdempsky
Copy link
Member Author

@mdempsky mdempsky commented Oct 18, 2019

@bradfitz Yeah, that's my additional thought #3 above. :)

@bcmills
Copy link
Member

@bcmills bcmills commented Oct 18, 2019

if we panic by default, then users might come to rely on it panicking and not write their own checking.

Hmm, good point. We could make it a throw! 😉

Or we could make it a panic in ordinary code but a throw under -race or -d=checkptr. (The important thing, I think, is to vary it just enough that it causes tests to fail in some reasonably-common configuration.)

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Sep 14, 2020

I also think that unsafe.Slice[T any](*T, int) []T is the right declaration. I understand why @rsc suggests that it is odd to write *T when you want a []T. However, you only call unsafe.Slice when you have a pointer, and you want to get a slice. Programs that are operating at this level are intimately familiar with the fact that a slice is in effect a bounded pointer. It seems to me to be entirely reasonable that there is an operation that takes a pointer and a bound and returns a slice. I don't think the fact that typical uses will explicitly say *T will lead to any confusion; the pointer is type *T, and the resulting bounded pointer is type []T.

@elichai
Copy link

@elichai elichai commented Sep 15, 2020

I'm afraid that involving generics in the API here will mean that this won't be added for at least the next year

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Sep 15, 2020

@elichai The actual implementation of this function would be entirely in the compiler, so generics are not really involved. It's just a way of picturing the declaration.

@elichai
Copy link

@elichai elichai commented Oct 5, 2020

I also think that unsafe.Slice[T any](*T, int) []T is the right declaration. I understand why @rsc suggests that it is odd to write *T when you want a []T. However, you only call unsafe.Slice when you have a pointer, and you want to get a slice. Programs that are operating at this level are intimately familiar with the fact that a slice is in effect a bounded pointer. It seems to me to be entirely reasonable that there is an operation that takes a pointer and a bound and returns a slice. I don't think the fact that typical uses will explicitly say *T will lead to any confusion; the pointer is type *T, and the resulting bounded pointer is type []T.

FWIW this is how it's declared/works both in Rust and in C++:
https://doc.rust-lang.org/std/slice/fn.from_raw_parts.html
https://en.cppreference.com/w/cpp/container/array/to_array

@rsc
Copy link
Contributor

@rsc rsc commented Oct 7, 2020

Based on Ian's response to my objection as well as the reactions, I retract the objection above.

Based on the discussion above, this is now a likely accept (again). Thanks for bearing with me. I just want to make sure we get this right.

@rsc
Copy link
Contributor

@rsc rsc commented Oct 14, 2020

No change in consensus, so accepted.

@rsc rsc moved this from Likely Accept to Accepted in Proposals Oct 14, 2020
@rsc
Copy link
Contributor

@rsc rsc commented Oct 14, 2020

A bit late for Go 1.16 so milestoning to Go 1.17.

@rsc rsc modified the milestones: Proposal, Go1.17 Oct 14, 2020
@rsc rsc changed the title proposal: unsafe: add Slice(ptr *T, len anyIntegerType) []T unsafe: add Slice(ptr *T, len anyIntegerType) []T Oct 17, 2020
@gopherbot
Copy link

@gopherbot gopherbot commented Oct 20, 2020

Change https://golang.org/cl/263800 mentions this issue: unsafe: add Slice and String headers

copybara-service bot pushed a commit to google/gvisor that referenced this issue Jan 28, 2021
See golang/go#19367 for rationale. Note that the
upstream decision arrived at in that thread, while useful for some of our use
cases, doesn't account for all of our SliceHeader use cases (we often use
SliceHeader to extract pointers from slices in a way that avoids bounds
checking and/or handles nil slices correctly) and also doesn't exist yet.

PiperOrigin-RevId: 350259679
copybara-service bot pushed a commit to google/gvisor that referenced this issue Feb 16, 2021
See golang/go#19367 for rationale. Note that the
upstream decision arrived at in that thread, while useful for some of our use
cases, doesn't account for all of our SliceHeader use cases (we often use
SliceHeader to extract pointers from slices in a way that avoids bounds
checking and/or handles nil slices correctly) and also doesn't exist yet.

PiperOrigin-RevId: 350259679
copybara-service bot pushed a commit to google/gvisor that referenced this issue Feb 17, 2021
See golang/go#19367 for rationale. Note that the
upstream decision arrived at in that thread, while useful for some of our use
cases, doesn't account for all of our SliceHeader use cases (we often use
SliceHeader to extract pointers from slices in a way that avoids bounds
checking and/or handles nil slices correctly) and also doesn't exist yet.

PiperOrigin-RevId: 350259679
copybara-service bot pushed a commit to google/gvisor that referenced this issue Feb 17, 2021
See golang/go#19367 for rationale. Note that the
upstream decision arrived at in that thread, while useful for some of our use
cases, doesn't account for all of our SliceHeader use cases (we often use
SliceHeader to extract pointers from slices in a way that avoids bounds
checking and/or handles nil slices correctly) and also doesn't exist yet.

PiperOrigin-RevId: 350259679
copybara-service bot pushed a commit to google/gvisor that referenced this issue Feb 18, 2021
See golang/go#19367 for rationale. Note that the
upstream decision arrived at in that thread, while useful for some of our use
cases, doesn't account for all of our SliceHeader use cases (we often use
SliceHeader to extract pointers from slices in a way that avoids bounds
checking and/or handles nil slices correctly) and also doesn't exist yet.

PiperOrigin-RevId: 350259679
copybara-service bot pushed a commit to google/gvisor that referenced this issue Feb 18, 2021
See golang/go#19367 for rationale. Note that the
upstream decision arrived at in that thread, while useful for some of our use
cases, doesn't account for all of our SliceHeader use cases (we often use
SliceHeader to extract pointers from slices in a way that avoids bounds
checking and/or handles nil slices correctly) and also doesn't exist yet.

PiperOrigin-RevId: 358071574
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Proposals
Accepted
Linked pull requests

Successfully merging a pull request may close this issue.

None yet