Tracking issue for `raw` stabilization (`raw::TraitObject`)

[aturon]

The std::raw module exports some representation details for the purposes of transmuting and taking things apart. Stabilizing these means pinning down the representation details forever.

One possibility is to instead provide various conversions, rather than asking people to use transmute, when we're sure the conversions will always be possible. Methods like from_raw_parts are already doing this.

The TraitObject case in particular is unsettled due to the potential for upcasting in the future.

cc @carllerche

CURRENT STATUS AS OF 2016-08-18: Discussed in @rust-lang/lang meeting. The situation is this. We expect that this will be insufficient in the future, particularly if/when we support trait objects like Trait1+Trait2 (which would require 2 vtables). We could stabilize it, since it would remain correct for all existing objects, but it'd be a shame. In contrast, the various "custom DST" proposals would allow for a much more flexible way of extracting and creating this data (sort of like from_raw_parts), so we think we'd prefer to explore that route. (from #27751 (comment))

[SimonSapin]

raw::Slice can be replaced with [T]::len, [T]::as{,_mut}_ptr, and slice::from_raw_parts{,_mut}. Kill it.

TraitObject is used in Servo:

• To implement downcasting from trait objects of a custom trait, like std::error::Error does. It’s unfortunate that std::any::Any can not be re-used. (I understood why at some point, but forgot.)
• Using obj.data as usize as an identifier. (LayoutId, OpaqueFlow)
• To do some kind of inheritance. Getting a reference from a &Flow trait object to the BaseFlow (concrete struct) that is must be the first field of every implementors of the Flow trait. Yes, relying on struct field order.
• To emulate Arc<Trait> before Arc supported DSTs. I’m trying to switch to Arc now.

[carllerche]

I have used TraitObject a number of times.

One case, in the bytes crate, I use TraitObject to avoid an allocation, which can add up to a significant performance win in some cases (for example, when using Rope).

The insight is that, pretty often, implementations of ByteStr can fit in a pointer size. So, Bytes, which is a wrapper around a ByteStr trait object does an optimization where the data component of the trait object gets "inlined".

Another case I had was building a somewhat involved tree structure, there would be many pointers to the same trait object. The goal was to get an entire node of the tree to fit in a cache line and duplicating the vtable ptr prevented this. So, in this case, I saved the vtable pointer only once and "reconstructed" the trait object on demand.

[kamalmarhubi]

I have a possible use case for stabilizing Slice: I would like to use byte slices directly as struct iovec for readv and writev in nix. There's an issue with more detail: nix-rust/nix#263

Mostly I'm looking to exploit the (pointer, length) representation matching up with iovec, and thereby allowing a nice ergonomics win for calls using iovecs.

[alexcrichton]

🔔 This issue is now entering its cycle-long final comment period 🔔

Specifically, the libs team is considering deprecating raw::Slice and raw::Repr as these are replaced by std::slice functions.

@kamalmarhubi we specifically don't want to stabilize the layout of slices to be used in places like that, hence the deprecation!

[SimonSapin]

Was TraitObject discussed?

[alexcrichton]

Yes, and we neither want to stabilize it nor deprecate it yet, this module will stick around for that reason.

[aturon]

A bit more detail from the discussion at the meeting: TraitObject is, amongst the types that have ever appeared in this module, the most likely to change in the future. That's in part due to considerations about upcasting, multiple trait inheritance, and so on. It's unclear whether we will ever want to commit firmly to a representation here.

That said, it'd be good to use this issue to track usecases for the raw representation, and see whether we can find other ways of accommodating them.

[kamalmarhubi]

@alexcrichton

@kamalmarhubi we specifically don't want to stabilize the layout of slices to be used in places like that, hence the deprecation!

I figured as much. Maybe specialization and conditional compilation will let me be bad in the way I want to be there... :-)

Thanks for the ping!

[alexcrichton]

The libs team discussed this during triage yesterday and the decision was to deprecate

[kamalmarhubi]

@alexcrichton is that the whole module, or just Slice and Repr as mentioned above:

Specifically, the libs team is considering deprecating raw::Slice and raw::Repr as these are replaced by std::slice functions.

[nrc]

reluctant +1 - I kind of liked the raw::Slice approach, but the std::slice solutions do the job just as well and we only need one.

[alexcrichton]

@kamalmarhubi just the Slice and Repr traits, as noted in #32804

[SimonSapin]

It looks like fat raw pointers can be cast to thin raw pointers to access the "data" part: (x: &Trait) as *const Trait as *const (). This leaves missing:

• Some way to extract a vtable pointer from &Trait or *const Trait. (I don’t know if it is possible or desirable to exclude other kinds of dynamically-sized types from this. Would a Trait trait make sense? For use like &T where T: Trait.)
• Some unsafe from_raw_parts function or method that returns *const Trait or *mut Trait (which can then be used with &mut *x or Box::from_raw(x)) from separate data and vtable pointers.

These together could replace std::raw::TraitObject without forcing users to make memory representation assumptions e.g. by transmuting *const Trait to and from (*const (), *const ()).

[durka]

If #34785 is merged, the raw feature will not track the std::raw module itself, but unstable items within it (currently just TraitObject).

[nrc]

@rust-lang/libs So was this feature deprecated? Should this issue be closed or a label changed or something?

[alexcrichton]

@nrc currently this covers raw::TraitObject, which is still unstable.

[nrc]

@alexcrichton is there a plan for raw::TraitObject (beyond speculation in this thread)?

[alexcrichton]

I believe not currently, no. I think we may have vaguely been waiting on the lang team to say whether it's cool or not to stabilize such a structure?

[SimonSapin]

To make my previous comment more explicit: I think a couple functions in the spirit of std::slice::from_raw_parts would be better than a struct intended to be used with transmute. transmute is dangerously easy to use incorrectly.

[nikomatsakis]

Discussed in @rust-lang/lang meeting. The situation is this. We expect that this will be insufficient in the future, particularly if/when we support trait objects like Trait1+Trait2 (which would require 2 vtables). We could stabilize it, since it would remain correct for all existing objects, but it'd be a shame. In contrast, the various "custom DST" proposals would allow for a much more flexible way of extracting and creating this data (sort of like from_raw_parts), so we think we'd prefer to explore that route.

[SimonSapin]

the various "custom DST" proposals

Are these written down? Could you link to them?

[ubsan]

@SimonSapin rust-lang/rfcs#1524

[SimonSapin]

I don’t know if this was always the case, but I’ve just found out that casting a raw pointer from a ?Sized type to a Sized one is allowed, and for trait objects extracts the data pointer. I’ve replaced with this all uses of std::raw::TraitObject in Servo.

fn data_pointer(object: &SomeTrait) -> *const () {
    let ptr: *const SomeTrait = object;
    ptr as *const ()
}

[eddyb]

@SimonSapin You're also able to make that function generic over a T: ?Sized pointee, right?

EDIT: I mean this:

fn data_pointer<T: ?Sized>(r: &T) -> *const u8 {
    r as *const T as *const u8
}

[SimonSapin]

@eddyb Yes, correct. In fact Servo does this, with a trait bound, to allow both trait objects and concrete types that implement that trait.

[durka]

I think that's always been the case. It's a footgun because you can lose the fat part of a fat pointer while casting. I'm not sure if it's specified which thin part you end up with.

…

On Fri, Oct 13, 2017 at 10:08 PM, Simon Sapin ***@***.***> wrote: @eddyb <https://github.com/eddyb> Yes, correct. In fact Servo does this, with a trait bound, to allow both trait objects and concrete types that implement that trait. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#27751 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAC3n2s4TUOqik2U8-LrlasX4P3hdthfks5sr8NZgaJpZM4FqbX-> .

[SimonSapin]

What do you mean, which thin part? I don’t think it would make sense for this cast to extract a vtable pointer or a slice length when casting to an arbitrary pointer to a statically-sized type.

[durka]

I agree and I don't see it changing. But as long as the actual repr is unstable I'm not sure the result of this cast would be officially set in stone.

…

On Fri, Oct 13, 2017 at 10:48 PM, Simon Sapin ***@***.***> wrote: What do you mean, which thin part? I don’t think it would make sense for this cast to extract a vtable pointer or a slice length when casting to an arbitrary pointer to a statically-sized type. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#27751 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAC3n5enzkTPDk-CPQCuJSHXjSl0tDmUks5sr8yLgaJpZM4FqbX-> .

[SimonSapin]

Even if the repr changes it has to include at least a data pointer, doesn’t it?

[durka]

The reference says it is a "pointer-to-pointer cast" but doesn't say what that means. Sounds OK though?

[eddyb]

@durka The casts are properly defined and there's even fat-to-fat casts which check the metadata and whatnot - they might be described in an RFC, or not at all, but that's orthogonal.

[SimonSapin]

https://doc.rust-lang.org/nightly/nomicon/casts.html says:

*T as *U where T, U: Sized
*T as *U TODO: explain unsized situation

and

Note that lengths are not adjusted when casting raw slices - *const [u16] as *const [u8] creates a slice that only includes half of the original memory.

Maybe it’s time to fix that TODO and define this more precisely?

[SimonSapin]

rust-lang/rfcs#2580 proposes deprecating this.