RFC: Custom DSTs

[ubsan]

This has been a long time coming - similar to #2580, but more general.

On the unresolved question about ?Sized -> DynamicallySized, and the need for ?DynamicallySized. I think this is actually a reasonable change which makes extern types act much nicer around generic functions that already exist, and fixes all the Rust code that has been written already. No longer is, for example, Box<extern-type> allowed. I've chosen to make this change.

Note that DynamicallySized is a new language trait, on the level of Sized and Copy.

Rendered

[ubsan]

It's likely that this RFC should be merged with some amount of #2580, specifically the stuff for mentioning VTables.

[Ixrec]

#2255 also seems like a relevant link, since I'm not entirely sure whether those issues are settled. I think the current status of that discussion is:

• introducing any new ?Trait bounds like ?Move or ?DynamicallySized, such that Trait is a new implicit bound on all existing generics, is backwards incompatible and therefore can never be done (not even with editions)
• the only loophole would be traits that are "not really new" because they're already implied by Sized, such as DynamicallySized
• the contention that ?Move was a brute force solution seems to have proven accurate now that we've discovered the alternative of Pin, so DynamicallySized appears to be the only plausible new implicit trait bound
• the only alternative to DynamicallySized that we've been able to find is panicking at runtime

This seems like a crucial (albeit not obviously crucial) piece of the argument in favor of this RFC, though I've never seen it spelled out like this before, so it seems worth asking whether I got all of that right.

---

My only actual concern with the RFC is the same one that niko articulated last time: this doesn't appear to be a hard blocker or a game changer for anything, so do we really want to prioritize work on it in the near future? But we can always accept the RFC and not implement it for a while; this is clearly the best proposal so far and the design space seems pretty thoroughly explored at this point.

To double-check: The CStr and FAM use cases are not "hard blockers" because you can do all that stuff with unsafe shenanigans today, right? i.e. there isn't any C code that's impossible to soundly integrate with Rust because of this, and this is "just" (emphasis on the scare quotes) a really huge ergonomics win for those use cases?

[mark-i-m]

Rendered: https://github.com/ubsan/rust-rfcs/blob/custom-dst/text/0000-custom-dst.md

[glaebhoerl]

Instead of declaring a type which would otherwise be Sized and having it become !Sized if you implement DynamicallySized for it, could we go the other way, and have the way to do things be to declare a type which is not-even-DynamicallySized and then implement DynamicallySized for it?

In the current language the only such types are extern types, so you'd start off with an extern type or a struct which has an extern type as the last field, and then implement DynamicallySized.

The problem there of course is not knowing what the alignment of that field should be, so maybe instead of relying entirely on extern types we ought to introduce something like a Flexible<T> (name subject to bikeshedding!) which has the alignment of T but is neither Sized nor DynamicallySizeds, so then you'd have Flexible<T> as the last field instead of [T; 0] as in the RFC.

Just in general it feels slightly nicer and less magical if the act of implementing a trait only causes a trait to become implemented, rather than another one to become deimplemented. (There would be some precedent for the latter w.r.t. Drop causing various things like pattern matching to become disabled if you implement it, though.)

Another question: I'm guessing the language implementation would implicitly call the provided size_of_val and align_of_val in various places? What are those places? Depending on what the places are, a possible drawback might be the ability to incur unexpected costs, e.g. if instead of just reading the size from a vtable it ends up traversing a 1MB string to find the end.

[ubsan]

@glaebhoerl I would argue that an extern type solution would make using FAMs far more annoying. Also, I think this is the most ergonomic way of doing this, even if it's slightly odd. Sized, Move, and DynamicallySized are odd traits.

Places the language calls size_of_val/align_of_val - afaik, just in the intrinsics (which could probably be removed). Otherwise, only in the library, when dealing with allocators.

[ubsan]

@Ixrec technically you're right, but it's such an incredible ergonomics win, and it's been suggested so many times. It also is an incredible ergonomics win for Matrix libraries, imo, since they'll suddenly be able to use the Deref/Index traits.

I'd also argue it should be a blocker for extern type, since as it currently stands, they're fairly broken.

[Ixrec]

@Ixrec technically you're right, but it's such an incredible ergonomics win, and it's been suggested so many times. It also is an incredible ergonomics win for Matrix libraries, imo, since they'll suddenly be able to use the Deref/Index traits.

Yeah, I'm totally on board with actually doing this. Just wanted to be sure I understood it all correctly.

I'd also argue it should be a blocker for extern type, since as it currently stands, they're fairly broken.

Interestingly, the extern types RFC states "before this is stabilized there should be some trait bound or similar on them that prevents [size_of_val's and align_of_val's] use statically", but does not list this as an unresolved question. Fortunately, the tracking issue comments also seem to imply that nobody wants to stabilize it before we figure this out.

[ubsan]

@Ixrec ah, good, I literally started writing this because someone was suggesting we panic on size_of_val.

[Ixrec]

Sorry, by "figure this out" I meant come to an explicit consensus on whether or not panicking was the least bad solution to that problem.

Now that I look closer, it seems like that discussion petered out after rust-lang/rust#43467 (comment) which explicitly suggests we make align_of_val work and let size_of_val panic. So it does seem like size_of_val panics are still on the table. I guess we should ping @RalfJung and @eddyb to comment further on that.

[ubsan]

Since we have a reasonable type system solution in this RFC, I don't think panicking is a reasonable choice.

[Centril]

DynamicallySized -> DynSized;
Feels distinctly more rust-y to me since we have dyn.

[eddyb]

And DynSized has been used in previous (non-RFC?) proposals and discussion, so more people are already familiar with it. Not the best argument, but I was surprised to see DynamicallySized.

[ubsan]

I like longer names, probably the C++ programmer in me. This name's not all that important tho :P

[Ixrec]

I am very mildly in favor of the longer name if this is expected to be a trait bound that relatively few users will ever need to write directly (it is, right?)

[ubsan]

Yeah - it gets implied by ?Sized. The only time you'd need to write it is for T: ?DynamicallySized, in case you want to be generic wrt extern types.

[gnzlbg]

I prefer longer names, but DynSized feels more consistent with the rest of Rust to me and that trumps my "longer name" preference.

[Centril]

N.B. Type: Trait is not a bound (because Trait + 'static is a bound...), it is a constraint.

[ubsan]

What's the distinction?

[Centril]

So a trait Foo { ... } has an implicit first parameter we call Self which is not present in Haskell; This means that when you say something like MyTrait, it has the kind bound = k1 -> constraint where k1 : type | lifetime ;. The operator : in this context can then be understood as something of kind type -> bound -> constraint. Notice that when you say Show Int => in Haskell, Show :: * -> Constraint (a "bound") but Show Int :: Constraint.

[ubsan]

ah, I see!

[jturner314]

Something like this RFC would be really nice for the ndarray crate. It would greatly simplify handling of array views (ArrayView/ArrayViewMut) and would make it possible to do things like return n-D array views from Index and use the standard library's Cow with arrays.

However, ndarray supports arrays with a dynamic number of dimensions (axes), so we would need the Metadata (the shape and strides of a view) to support being dynamically sized (or sized with heap-allocated data). Would it be possible to extend this RFC to handle this use-case? Maybe we could change the Copy bound on Metadata to Clone, and then provide a special implementation of Copy for &T that actually clones the metadata?

[eddyb]

@jturner314 That's unworkable because it would break existing code generic over references and pointers. I think what you'd want to be able to use this feature is const generics and actually parametrize those types by the number of dimensions, e.g. ArrayView<A, 4>.

[kennytm]

What happens if these methods are implemented to return invalid values (UB?)

struct CWStr([u16; 0]);

unsafe impl DynamicallySized {
    type Metadata = ();
    fn size_of_val(&self) -> usize { 3 }
    fn align_of_val(&self) -> usize { 3 }
}

[ubsan]

UB, that's why it's an unsafe impl.

[gnzlbg]

The text should spell out exactly why DynamicallySized is unsafe to implement and which guarantees implementations have to uphold.

[cramertj]

@ubsan Can you add these additional docs? I'd find it really helpful.

[jturner314]

I think what you'd want to be able to use this feature is const generics and actually parametrize those types by the number of dimensions, e.g. ArrayView<A, 4>.

ndarray supports arrays with the number of dimensions (axes) known at compile time (e.g. arrays with "const dimensionality" Ix3) and arrays with the number of dimensions known only at runtime (arrays with "dynamic dimensionality" IxDyn). For const-dimensionality arrays, the shape and strides are represented as [T; NDIM], while for dynamic-dimensionality arrays, the shape and strides are represented Vec<T>¹. The "known only at runtime"/"dynamic dimensionality" case is what I'm asking about, since const generic [T; N] doesn't work for this case.

¹We actually use a small-vector optimization for four axes or less, but that's irrelevant for this discussion.

That's unworkable because it would break existing code generic over references and pointers.

Yeah, that's what I was afraid of. Another possible approach that would enable this use-case is providing an owned equivalent of &[T] that could be placed on the stack, used in Metadata, etc. (This would be similar to C99's variable length arrays.) That feature is probably outside the scope of this RFC, though. [Edit: After further thought, I believe this wouldn't work well anyway because it would make &T itself potentially dynamically-sized.]

Out of curiosity, what's a case where the "use Clone bound on Metadata, and implement Copy for &T/*const T/etc. in terms of cloning the metadata" proposal would break things?

[Ixrec]

Out of curiosity, what's a case where the "use Clone bound on Metadata, and implement Copy for &T/*const T/etc. in terms of cloning the metadata" proposal would break things?

I guess the simplest case would be any code relying on the fact that copying a Copy type cannot panic.

[jturner314]

I guess the simplest case would be any code relying on the fact that copying a Copy type cannot panic.

Good point. We could treats panics as aborts in this case, though. (We already do this kind of thing for panic inside drop when unwinding from another panic.)

I thought of another potentially breaking case – code could rely on the original and copy from Copy being equal, but .clone() doesn't necessarily guarantee that. We could say that ensuring clones of Metadata are equal is a safety requirement for implementing DynamicallySized, but if code relies on the copies being bitwise identical, that wouldn't allow for Vec in Metadata.

Another potentially breaking case is dealing with uninitialized memory. If the type of the uninitialized memory is Copy, it's usually considered safe to assign to it normally (without using ptr::write) because Copy requires that the type not implement Drop. With only a Clone bound that's not the case. Unlike the other issues, I don't see a backwards compatible workaround for this that would enable using Vec in Metadata. [Is normal assignment in the Copy case really safe, though? In order to perform the assignment without ptr::write, you have to create an &mut T. Is creating an &mut T always defined behavior for uninitialized T where T: Copy?]

Yeah, I'm coming to the conclusion that a Clone bound instead of Copy bound on Metadata is unworkable without breaking changes.

[eddyb]

@jturner314 Rust is designed with "moves and copies are literally bitwise copies always" as a core promise, as contrasted with C++. Unsafe code is free to rely on this as much as it wants to.
So yeah, it's a serious breaking change, and one that I don't see happening.

If it's opt-in, maybe it could happen. But I also feel like the dynamic number of dimensions is less common (or will become less common with const generics).

[kennytm]

@gnzlbg Let me clarify, suppose there are m dimensions and N elements (the length), if place the dimensions separately from the elements, then for slicing we only need to allocate for those m dimensions and borrow the existing N elements.

However, if these are placed together next to each other, we'll need to allocate for all m + N items.

[gnzlbg]

If place the dimensions separately from the elements, then for slicing we only need to allocate for those m dimensions and borrow the existing N elements.

I think I see what you mean. What I showed is that one can, in some cases (e.g. if the number and values of dimensions don't need to change), do this without allocating at all as long as the dimensions and values are layed out appropriately.

If one wants to create a view with a different run-time number and values of dimensions over an already existing slice of elements, then one has to allocate those dimensions somewhere. I do not see any major problems with the Metadata owning those dimensions by containing a Box<[usize]> beyond having to perform a memory allocation, but for Metadata to borrow them one might need something a bit messier like:

struct View<'a, T> { data: ([&'a T; 0]) }

struct ViewMetadata<'a> {
    dims: &'a [usize],
    len: usize,
}

unsafe impl<'a, T> DynamicallySized for View<'a T> {
    type Metadata = ViewMetadata<'a>;
    ...
}

impl<'a, T> View<'a, T> {
    fn new<'b, 'c: 'a + 'b>(ts: &'b [T], dims: &'a [usize]) -> &'c Self { 
        let ptr = t.as_ptr();
        let meta = ViewMetadata { dims, len: ts.len() };
        unsafe {
            std::raw::from_raw_parts::<Self>(ptr, meta)
        }
    }
}

let ts: &[T] = ...;
let dims: &'a [usize] = &[...];

let v: &View<'a, T> = View::new(ts, dims);

[Ericson2314]

I think you want DynamicallySized: Sized. Otherwise the T: Sized bound doesn't imply T: DynamicallySized. You'd then do a impl !Sized for Foo or however those opt-out traits work to implement your own DynamicallySized.

[KrishnaSannasi]

@Ericson2314 I think you got it backwards, it should be Sized: DynamicallySized. This would be a breaking change. DynamicallySized: Sized means that you must implement Sized to implement DynamicallySized.

[Ericson2314]

Yes sorry that is what I meant.

[alexreg]

I like this proposal, in essence and most of the details. A few thoughts:

• DynamicallySized is too long. I'd prefer DynSized myself.
• If you have a type which you would like to be unsized, you can implement this trait for your type! -- I know "unsized" has traditionally been synonymous with "dynamically sized" in Rust, but it would be clearer to write "dynamically sized" here. Also, I personally like the distinction, especially since the advent of extern type, which are truly unsized from the perspective of the Rust side of FFI (right now).
• I'm a "yes" for allowing implementations of DynSized for extern types and types containing them. Sometimes there is no Rust data structure capable of representing a weird foreign type (e.g., with dynamic layout), so this is handy in such cases. It would mean that extern type can be unsized or dynamically sized, but never (statically) sized, of course. This seems reasonable to me.
• More bikeshedding: header_size rather than size_of_header, etc., would appeal more to me.

[cramertj]

What are the tradeoffs between this and requiring that every type implement DynamicallySized? Are there examples aside from extern type that shouldn't implement DynamicallySized?

[gnzlbg]

To double-check: The CStr and FAM use cases are not "hard blockers" because you can do all that stuff with unsafe shenanigans today, right? i.e. there isn't any C code that's impossible to soundly integrate with Rust because of this, and this is "just" (emphasis on the scare quotes) a really huge ergonomics win for those use cases?

While I believe that the claim that this can be done is technically correct, if the amount of work or expertise required to do this is too high. In practice, for many Rust users, doing this is not possible because they can't do it.

One example is the standard library itself, which doesn't do this for CStr, even though it technically could (CStr is not pointer sized). That suggest that only Rust programmers whose expertise surpases that of the standard library maintainers are able to do it.

[thomcc]

One example is the standard library itself, which doesn't do this for CStr, even though it technically could (CStr is not pointer sized)....

I'm not sure "it technically could" is accurate. How could that be made to work with Rust's current feature set?

[gnzlbg]

How could that be made to work with Rust's current feature set?

We'd make CStr a #[lang_item].

[velvia]

Hi folks,

I love this proposal, especially for parsing and access to some externally defined structs which have variable size.

Does the associated type Metadata have to point to some user defined structure? What if the info for deriving the size of the structure is in the struct itself (for example, many structures have length headers)? Would we create some unnecessary Metadata = SizeMetadata struct where SizeMetadata contains what is already in the original (dynamically sized) struct?

[KrishnaSannasi]

@velvia The idea would be that you split up your data from your metadata. Because your type is a DST, and you can't have !Sized values without indirection. This means that your metadata will live on the pointer, along side your value, but not inside your value.

for example you could define a slice like so

use std::mem::PhantomData;
struct Slice<T>(PhantomData<T>);

unsafe trait DynamicallySized {
    type Metadata = usize; // len

    fn size_of_val(&self) -> usize {
        std::mem::size_of::<T>() * metadata(self)
    }
    
    fn align_of_val(meta: Self::Metadata) -> usize {
        std::mem::align_of::<T>()
    }
}

[Lokathor]

How would a person go about actually determining how much space it takes to store a DST value. Like, a slice is a pointer an a len, so the actual size of a slice is usize * 2. Would there be a formalized way to get that number so that you could set aside space to place a DST value?

[Ericson2314]

@Lokathor take a look at #2580 if you havne't seen it. Hopefully that lands soon, so this Custom DST RFC can not worry about litigating those issues too. Simple divide-and-conquer of the problem space!

[KrishnaSannasi]

@Lokathor One thing to note is that, currently, we can only create DSTs in three ways: string literals, coercion, unsafe code (with pointer casts and transmute). So you would have set aside the space already to create the underlying Sized type and now you are simply pointing to it, or use unsafe code, which is operating on values that are already initialized.

[thomcc]

@velvia I think the idea is that you'd use Metadata = () for that case, and only look at the data in the header.

[Lokathor]

@Ericson2314 cool, will do.

@KrishnaSannasi no perhaps you misunderstand me, I do not care about the size of the data pointed to.

[kennytm]

@Lokathor You just reserve size_of::<*const CustomDst>() bytes?

[Lokathor]

That's the pointer only. You'd then also need the metadata after it. Which you could do with something like size_of::<*const CustomDst>() + size_of::<<CustomDST as DynamicallySized>::Metadata>() but it should be obvious that such an expression is something long and fiddly that we should give a name to and make a method to avoid errors

[thomcc]

@Lokathor why would it only be the pointer? std::mem::size_of<*const str>() is 16 on 64 bit systems, for example, because it includes both the pointer and the length.

[gnzlbg]

That's the pointer only.

The metadata is part of the pointer, e.g., right now *const [T] is 2 words in size.

[Lokathor]

Oh! Well then ignore me.

[kennytm]

Err does this RFC now require #2255?

[ubsan]

I mean, maybe? It seems like a silly question. We've already got ?Sized, and these are the natural extension of that. There doesn't seem to be any other way of these existing without breaking a whole heck of a lot of generic code.

[cramertj]

Should be Pointee, right?

[cramertj]

What prevents the automatic implementation of Contiguous given that struct Plane<T> { buffer: [T; 0] } is Sized? Does implementing Pointee make the implementation of Sized go away?

[ubsan]

Yeah.