Tracking issue for arbitrary_self_types

[arielb1]

Tracking issue for #![feature(arbitrary_self_types)].

This needs an RFC before stabilization, and also requires the following issues to be handled:

• figure out the object safety situation
• figure out the handling of inference variables behind raw pointers
• decide whether we want safe virtual raw pointer methods

Object Safety

See #27941 (comment)

Handling of inference variables

Calling a method on *const _ could now pick impls of the form

impl RandomType {
    fn foo(*const Self) {}
}

Because method dispatch wants to be "limited", this won't really work, and as with the existing situation on &_ we should be emitting an "the type of this value must be known in this context" error.

This feels like fairly standard inference breakage, but we need to check the impact of this before proceeding.

Safe virtual raw pointer methods

e.g. this is UB, so we might want to force the call <dyn Foo as Foo>::bar to be unsafe somehow - e.g. by not allowing dyn Foo to be object safe unless bar was an unsafe fn

trait Foo {
    fn bar(self: *const Self);
}

fn main() {
    // creates a raw pointer with a garbage vtable
    let foo: *const dyn Foo = unsafe { mem::transmute([0usize, 0x1000usize]) };
    // and call it
    foo.bar(); // this is UB
}

However, even today you could UB in safe code with mem::size_of_val(foo) on the above code, so this might not be actually a problem.

More information

There's no reason the self syntax has to be restricted to &T, &mut T and Box<T>, we should allow for more types there, e.g.

trait MyStuff {
    fn do_async_task(self: Rc<Self>);
}

impl MyStuff for () {
    fn do_async_task(self: Rc<Self>) {
        // ...
    }
}

Rc::new(()).do_async_stuff();

This doesn't have an RFC, but we want to experiment on this without one.

See #27941.

[porky11]

Why would you need this?
Why wouldn't you write an impl like this:

impl MyStuff for Rc<()> {
    fn do_async_task(self) {
        // ...
    }
}

I'd rather define the trait different. Maybe like this:

trait MyStuff: Rc {
    fn do_async_task(self);
}

In this case, Rc would be a trait type. If every generic type implemented a specific trait (this could be implemented automatically for generic types) this seems more understandable to me.

[cuviper]

This could only be allowed for trait methods, right?

For inherent methods, I can't impl Rc<MyType>, but if impl MyType can add methods with self: Rc<Self>, it seems like that would enable weird method shadowing.

[arielb1]

@cuviper

This is still pending lang team decisions (I hope there will be at least 1 RFC) but I think it will only be allowed for trait method impls.

[arielb1]

@porky11

You can't implement anything for Rc<YourType> from a crate that does not own the trait.

[arielb1]

So changes needed:

• remove the current error message for trait methods only, but still have a feature gate.
• make sure fn(self: Rc<Self>) doesn't accidentally become object-safe
• make sure method dispatch woks for Rc<Self> methods
• add tests

[SimonSapin]

I’ll look into this.

[arielb1]

Note that this is only supported to work with trait methods (and trait impl methods), aka

trait Foo {
    fn foo(self: Rc<Self>);
}
impl Foo for () {
    fn foo(self: Rc<Self>) {}
}

and is NOT supposed to work for inherent impl methods:

struct Foo;
impl Foo {
    fn foo(self: Rc<Self>) {}
}

[SimonSapin]

I got caught in some more Stylo work that's gonna take a while, so if someone else wants to work on this in the meantime feel free.

[kennytm]

Is this supposed to allow any type as long as it involves Self? Or must it impl Deref<Target=Self>?

trait MyStuff {
    fn a(self: Option<Self>);
    fn b(self: Result<Self, Self>);
    fn c(self: (Self, Self, Self));
    fn d(self: Box<Box<Self>>);
}

impl MyStuff for i32 {
   ...
}

Some(1).a();  // ok?
Ok(2).b();  // ok?
(3, 4, 5).c(); // ok?
(box box 6).d(); // ok?

[mikeyhew]

I've started working on this issue. You can see my progress on this branch

[mikeyhew]

@arielb1 You seem adamant that this should only be allowed for traits and not structs. Aside from method shadowing, are there other concerns?

[arielb1]

@mikeyhew

inherent impl methods are loaded based on the type. You shouldn't be able to add a method to Rc<YourType> that is usable without any use statement.

[arielb1]

That's it, if you write something like

trait Foo {
    fn bar(self: Rc<Self>);
}

Then it can only be used if the trait Foo is in-scope. Even if you do something like fn baz(self: u32); that only works for modules that use the trait.

If you write an inherent impl, then it can be called without having the trait in-scope, which means we have to be more careful to not allow these sorts of things.

[mikeyhew]

@arielb1 Can you give an example of what we want to avoid? I'm afraid I don't really see what the issue is. A method you define to take &self will still be callable on Rc<Self>, the same as if you define it to take self: Rc<Self>. And the latter only affectsRc<MyStruct>, not Rc<T> in general.

[mikeyhew]

I've been trying to figure out how we can support dynamic dispatch with arbitrary self types. Basically we need a way to take a CustomPointer<Trait>, and do two things: (1) extract the vtable, so we can call the method, and (2) turn it into a CustomPointer<T> without knowing T.

(1) is pretty straightforward: call Deref::deref and extract the vtable from that. For (2), we'll effectively need to do the opposite of how unsized coercions are implemented for ADTs. We don't know T, but we can can coerce to CustomPointer<()>, assuming CustomPointer<()> has the same layout as CustomPointer<T> for all T: Sized. (Is that true?)

The tough question is, how do we get the type CustomPointer<()>? It looks simple in this case, but what if CustomPointer had multiple type parameters and we had a CustomPointer<Trait, Trait>? Which type parameter do we switch with ()? In the case of unsized coercions, it's easy, because the type to coerce to is given to us. Here, though, we're on our own.

@arielb1 @nikomatsakis any thoughts?

[nikomatsakis]

@arielb1

and is NOT supposed to work for inherent impl methods:

Wait, why do you not want it work for inherent impl methods? Because of scoping? I'm confused. =)

[nikomatsakis]

@mikeyhew

I've been trying to figure out how we can support dynamic dispatch with arbitrary self types.

I do want to support that, but I expected it to be out of scope for this first cut. That is, I expected that if a trait uses anything other than self, &self, &mut self, or self: Box<Self> it would be considered no longer object safe.

[mikeyhew]

@nikomatsakis

I do want to support that, but I expected it to be out of scope for this first cut.

I know, but I couldn't help looking into it, it's all very interesting to me :)

[arielb1]

Wait, why do you not want it work for inherent impl methods? Because of scoping? I'm confused. =)

We need some sort of "orphan rule" to at least prevent people from doing things like this:

struct Foo;
impl Foo {
    fn frobnicate<T>(self: Vec<T>, x: Self) { /* ... */ }
}

Because then every crate in the world can call my_vec.frobnicate(...); without importing anything, so if 2 crates do this there's a conflict when we link them together.

Maybe the best way to solve this would be to require self to be a "thin pointer to Self" in some way (we can't use Deref alone because it doesn't allow for raw pointers - but Deref + deref of raw pointers, or eventually an UnsafeDeref trait that reifies that - would be fine).

I think that if we have the deref-back requirement, there's no problem with allowing inherent methods - we just need to change inherent method search a bit to also look at defids of derefs. So that's probably a better idea than restricting to trait methods only.

Note that the CoerceSized restriction for object safety is orthogonal if we want allocators:

struct Foo;
impl Tr for Foo {
    fn frobnicate<A: Allocator+?Sized>(self: RcWithAllocator<Self, A>) { /* ... */ }
}

Where an RcWithAllocator<Self, A> can be converted to a doubly-fat RcWithAllocator<Tr, Allocator>.

[mikeyhew]

@arielb1

Because then every crate in the world can call my_vec.frobnicate(...); without importing anything, so if 2 crates do this there's a conflict when we link them together.

Are saying is that there would be a "conflicting symbols for architechture x86_64..." linker error?

Maybe the best way to solve this would be to require self to be a "thin pointer to Self" in some way (we can't use Deref alone because it doesn't allow for raw pointers - but Deref + deref of raw pointers, or eventually an UnsafeDeref trait that reifies that - would be fine).

I'm confused, are you still talking about frobnicate here, or have you moved on to the vtable stuff?

[arielb1]

I'm confused, are you still talking about frobnicate here, or have you moved on to the vtable stuff?

The deref-back requirement is supposed to be for everything, not only object-safety. It prevents the problem when one person does

struct MyType;
impl MyType {
    fn foo<T>(self: Vec<(MyType, T)>) { /* ... */ }
}   

While another person does

struct OurType;
impl OurType {
    fn foo<T>(self: Vec<(T, OurType)>) {/* ... */ }
}   

And now you have a conflict on Vec<(MyType, OurType)>. If you include the deref-back requirement, there is no problem with allowing inherent impls.

[KodrAus]

Also, is the expectation that this MethodReceiver trait will be publicly implementable?

If so, it seems like we'd probably end up recommending types that implementing Deref (where there's already an understanding that you don't add inherent methods to avoid shadowing) also implement this trait. The bounds in my example would then become MethodReceiver<Target = Self> + DerefMut<Target = Self>, which seems ok.

If not, I think we shouldn't implement it for any non-fundamental containers, like Mutex, otherwise we'd be making the std implementations of those types special compared to ecosystem ones.

[f32by]

Currently lifetimes cannot be elided when using arbitrary_self_types:

struct A(u32);

impl A {
	// compile error
    pub fn value(self: &Arc<Self>) -> &u32 { &self.0 }
//                                    ^ error: missing lifetime specifier. this function's return type contains a borrowed value, but there is no value for it to be borrowed from

    // ok
    pub fn value_lifetimed<'a>(self: &'a Arc<Self>) -> &'a u32 { &self.0 }
}

And The Rust Reference says:
https://doc.rust-lang.org/reference/lifetime-elision.html

If the receiver has type &Self or &mut Self, then the lifetime of that reference to Self is assigned to all elided output lifetime parameters.

Should this rule also be expanded to arbitary self types?

[TOETOE55]

Is that a feature like UFCS(Uniform Function Call Syntax)?

[oconnor663]

Has there been any discussion about self: &Cell<T>? That seems especially interesting in that &Cell<T> is (as of Rust 1.37) convertible from &mut T, so methods taking &Cell<T> could be useful in existing programs, without any change in data layout. I understand there's a Deref requirement that currently stands in the way, but from the thread above it sounds like people are considering relaxing that requirement somehow.

[dimpolo]

What could be the reason I cannot get a receiver with an extra trait bound (e.g. Send) to work?
If you remove the + Send bound on all impls, everything works fine.

struct Ptr<T: ?Sized + Send>(Box<T>);

// impls for Deref, CoerceUnsized and DispatchFromDyn

trait Trait: Send { fn ptr_wrapper(self: Ptr<Self>) -> i32; }

impl Trait for i32 { fn ptr_wrapper(self: Ptr<Self>) -> i32 { *self }}

fn main() {
    let p = Ptr(Box::new(5)) as Ptr<dyn Trait>;
    assert_eq!(p.ptr_wrapper(), 5);
}

error[E0038]: the trait "Trait" cannot be made into an object

Playground

[jackh726]

Going to retag this as wg-traits; kind of the best wg- label right now, even though we don't really discuss this

[pnkfelix]

Input from T-lang Backlog Bonanza meeting from 2022-02-09: Can anyone write down a summary of the status of this feature?

Should any of the checkboxes in the description be checked off now as done? Do new checkboxes need to be added?

[jackh726]

Changing WG-traits to T-types. We discussed this in today's types triage meeting and ultimately decided that there are types decisions to be made here, so the team label feels appropriate. We don't have any concrete plans for this right now though.

[adetaylor]

Here's a summary of this issue as requested in this comment @pnkfelix.

Disclaimer: I haven't been involved so this may not be exactly right. But I'm interested in taking it forward so this summary was useful to compile anyway.

What works now

• Nightly only, behind feature flag arbitrary_self_types:
  • Any type implementing Deref<Target=T> may accept method calls on behalf of T. (@mikeyhew's PR Implement arbitrary_self_types #45870)
  • *const T and *mut T may accept method calls (@mikeyhew's PR arbitrary_self_types: add support for raw pointer self types #46664)
  • Dynamic dispatch and object safety were implemented (@mikeyhew's PR Take 2: Implement object-safety and dynamic dispatch for arbitrary_self_types #54383) using DispatchFromDyn
• Stabilized already:
  • Rc<Self>, Arc<Self> and Pin<P> may be used as a receiver type (@withoutboats' issue [Stabilization] Stablize using some arbitrary self types defined in std #55786 amd @mikeyhew's PR Stabilize Rc, Arc and Pin as method receivers #56805)

Other work done

• @withoutboats raised an RFC (Custom self types rfcs#2362) which was marked as postponed in August 2019 (Custom self types rfcs#2362 (comment))

Concerns

This attempts to be a complete summary of all known concerns raised in the discussions (at least, those visible on github). Please shout if I've missed something!

• Major things we can defer till later (i.e. not stabilize yet):

  • Object safety. The algorithm for object safety is quite complex and relies on some object layout assumptions described in DispatchFromDyn. There are not currently any known problems here but it seems wise to be cautious in stabilizing this bit. For example, "before this gets merged, I think we should either remove the part about object safety, or say that the way object-safety is determined will remain unstable for a while, and will probably need a new RFC for a final decision to be made".
  • It only works on types implementing Deref. There have been many requests to call methods on types which can't implement Deref: for instance, RefCell, NonNull, Weak. A summary is here. It's proposed this be solved using a MethodReceiver trait (formerly known as ArbitrarySelfTypePtr). This also helps with some of the C++ interop use cases which is why I'm interested..

• Smaller issues that we probably need to address in the next bit of stabilization:

  • Calling methods on raw trait pointers with invalid vtables might be UB. Any use of a raw pointer in Rust typically requires unsafe. Calling trait methods (*const dyn Trait, for instance) via arbitrary self types might "use" the vtable, and therefore should be unsafe. (This might also apply to future extension via use of MethodReceiver.)
  • Handling of inference variables. As noted in the initial issue description *const _ may require some specific error messages just like &_ does.

• Smaller issues which we can defer till later:

  • Lifetime elision isn't yet done. Minor inconvenience? Example, example.
  • Can't call *const T methods on a *mut T. Minor inconvenience.

• Issues which are resolved or aren't relevant to stabilization:

  • Method shadowing. In the early days of this effort there were concerns about method shadowing and a proposal of an additional orphan rule. It was agreed that there are no problems here if we rely on Deref.
  • Name of the feature. Some say it should be "custom method receivers" rather than "arbitrary self types".

Next steps

This seems to decompose nicely into several sequential RFCs.

1. (already done in Stabilize Rc, Arc and Pin as method receivers #56805!) Stabilize support for Rc<Self>, Arc<Self> and Pin<P> using a hard-coded Receiver trait within core. (This initial split was proposed here: Tracking issue for arbitrary_self_types #44874 (comment))
2. Extend stable support to truly custom method receivers - any type which implements Deref<Target=T> - but do not yet attempt object safety beyond the hard-coded types which already implement the hidden Receiver trait. (This split was proposed back here Tracking issue for arbitrary_self_types #44874 (comment)). We should also ensure that this stabilization does not yet allow calls to *const dyn T or *mut dyn T, or that such calls require unsafe. We should also ensure that error messages relating to inference variables are great. As far as I can see, there are no other unresolved issues or complexities here.
3. Stabilize object safety. This would stabilize DispatchFromDyn or similar.
4. Extend to types which don't or can't support Deref by implementing a MethodReceiver trait
5. Minor future extensions in any order: (a) Allow calling *const T methods on a *mut T, (b) Solve lifetime elision, (c) Allow calls to *const dyn T and *mut dyn T.

The only blocker to doing step 2 is if we decide that we might not want to rely on Deref for this feature. But that seems pretty certain by this point...?

[RalfJung]

Calling methods on raw trait pointers with invalid vtables might be UB. Any use of a raw pointer in Rust typically requires unsafe. #44874 (comment). (This might #44874 (comment).)

This is closely related to rust-lang/rfcs#3324. There it was decided that upcasting on raw pointers is safe. This means, at the very least, that the safety invariant for raw dyn pointers says that the vtable is correct. Therefore it would only make sense to also make method calls safe.

This means unsafe code constructing invalid raw dyn pointers is currently on shaky grounds -- we likely want to permit some invalid values for intermediate local use, but it is not yet determined which ones.

[madsmtm]

It only works on types implementing Deref

One thing that I want to make sure we know that we're deciding now, is that this will work "by default" with Deref (whereas we could consider wanting to make it opt-in).

This effectively locks us in the future to the MethodReceiver having a blanket impl<T: Deref> MethodReceiver for T.