&move, DerefMove, DerefPure and box patterns

[arielb1]

I think I managed to get these features together into something I like.

Rendered.

[oli-obk]

why isn't the argument to deref_move simply a &move self?

[arielb1]

Because then it would drop self.

[RalfJung]

If that's not what the implementation of deref_move wants, it can always to mem::forget.

Permission wise, the deref_move type somehow does not seem to make sense to me: The &mut self argument means that when the (implicit) lifetime 'a ends, ownership of the referred data (and the memory) is transferred back to the lender. On the other hand, the return type indicates that full, direct ownership of the data is returned, just the memory remains borrowed. That just does not work together. It seems impossible to safely implement this function, say, for a newtype alias of Box.

Am I missing something?

EDIT: Notice that your example for why deref_move cannot be called directly is also a direct consequence of not using &move in the signature. Wrt. drop, actually it seems to me that &move T would be the better type to use for the argument of drop, since it would actually reflect its destructive nature. I guess that ship has long sailed, though...
Anyway, I think I am arguing that indeed, drop and deref_move are similar, and both should use &move.

[ubsan]

@RalfJung

ping @Ericson2314

[arielb1]

@RalfJung

You still won't be able to call deref_move directly. For example, if you call deref_move on a Box, the "exterior" of the Box is still present and needs to be dropped.

Well okay, it will only result in leaks, but that kind of random leaking feels bad.

DerefMove can't really be typed under Rust's type-system.

[RalfJung]

@arielb1 That's equivalent to calling foo(&move *x) where x: Box<T>, right? When foo returns, the exterior of the box (the heap allocation) is still around, but there is no ownership in there anymore.

I feel like &move exactly adds the possibility to type drop and deref_move, or at least that's what it should do. If it doesn't, I failed to understand its intended meaning.

[Ericson2314]

DerefMove needs typestate to be safe and expressive.

[oli-obk]

At what point is the memory freed? E.g. a &move [T] created from a Vec<T> needs to know the capacity to be able to deallocate properly.

[ticki]

@oli-obk [T] doesn't have a destructor itself.

[arielb1]

@oli-obk

&move does not drop anything but its contents, it's just a thin pointer. <RawVec as Drop>::drop does all the deallocations.

This is not Rooted<T, Destructor> but &move.

[pnkfelix]

A type is allowed to implement both Drop and DerefMove, correct?

I've copied some text below that I think does imply this is the case. If so, it would be good to have an example of that (even an artificial one that just calls println!), explicitly spelling out the event ordering.

When such a type is dropped, *x (aka x.deref_move()) is dropped first if it was not moved from already, similarly to Box today. Then the normal destructor and the destructors of the fields are called.

[arielb1]

Sure it is.

[Stebalien]

For the sake of discussion, what if I want to implement DerefMove on Cell?

[arielb1]

What do you mean by that?

[Stebalien]

Nevermind. I was trying to come up with an example where one might what to implement DerefMove but not Deref. Cell was the only case I could think of where this might make sense but it's not really a useful example.

[ubsan]

Some things I notice are missing:

1. *move, the raw pointer version of &move

2. lvalue semantics: a *(x: *move|&move) creates a moveable lvalue, just like *(x: *mut|&mut) creates a mutable lvalue.

3. Variance: &'a move T should be variant over 'a, variant over T

Why is DerefPure required for DerefMove? I'd much rather have the guarantee of only being called once, and I'll copy my reply to the other thread over:

This [T: DerefMove + !DerefPure] allows for things like dropping in DerefMove, since it's only called once. If, for example, you have a cache, and you don't need it once you're moved out of, it would be nice to get rid of it once you're no longer useful; and, I don't see the point of requiring DerefPure. It just makes it more annoying to implement DerefMove for users.

An example like:

struct Foo(Box<u32>, Box<u32>);
fn main() {
  let x = Box::new(Foo(Box::new(0), Box::new(1)));
  drop(x.1);
}

would be turned into

struct Foo(Box<u32>, Box<u32>);
fn main() {
  let x: Box<Foo>;
  let x_inner: Foo;
  let tmp0: Box<u32>;
  let tmp1: Box<u32>;
  let tmp2: Foo;
  tmp0 = Box::new(0);
  tmp1 = Box::new(1);
  tmp2 = Foo(tmp1, tmp2);
  x = Box::new(tmp2);  x_inner = x.deref_move();
  drop(x_inner.1);
  // drop glue
  Drop::drop(&mut x_inner.0);
  Drop::drop(&mut x);
}

[arielb1]

Why is DerefPure required for DerefMove? I'd much rather have the guarantee of only being called once, and I'll copy my reply to the other thread over:

I am not sure how we can guarantee the single-deref property in codegen.

[ubsan]

@arielb1 I mean, we guarantee single-drop; I imagine we could guarantee the same for deref_move.

[arielb1]

@ubsan

The way we guarantee single drop is by turning all duplicate drops to a no-op. If we want to go that way with DerefMove, we would need to do some sort of CSE, which looks non-trivial (e.g. we don't want to do CSE if you re-initialize the original).

[ubsan]

@arielb1 Could you explain as if I don't understand what CSE is?

[ubsan]

Okay, I looked up common subexpression elimination and... it doesn't really seem to have anything to do with what we're talking about? If DerefMove isn't pure, then calling it twice won't be the same as calling it once, therefore you can't eliminate a common subexpression.

[arielb1]

I think I figured out a way to implement impure DerefMove - and that is probably what you intended.

Could you try to write some interesting test cases for it?

[nrc]

It would be nice to have some examples of this, or at least something a bit more concrete

[arielb1]

I have the Vec example

[eefriedman]

Potentially interesting impure DerefMove example:

fn f(v: Vec<Foo>, b: bool) {
  if b {
    drop(v[1]); // Calls deref_move()
  }
  drop(v[0]); // Conditionally calls deref_move()
}

[arielb1]

@eefriedman

This does not compile. It is translated into:

fn f(v: Vec<Foo>, b: bool) {
  if b {
    let tmp0 = DerefMove::deref_move(&move v);
    //~^ NOTE value moved here
    drop(tmp0[1]);
    drop tmp0; // end of temporary scope
  }
  let tmp1 = DerefMove::deref_move(&move v);
  //~^ ERROR use of moved value
  drop(tmp1[0]);
  drop tmp1; // end of temporary scope
  drop v; // end of argument scope
}

Of course, if Vec is DerefPure, it is treated as an lvalue projection:

fn f(v: Vec<Foo>, b: bool) {
  if b {
    drop((*v)[1]);
  }
  drop((*v)[0]);
  drop v; // end of argument scope
  // ^ (*v)[0] and (*v)[1] are already dropped - drop the rest and the allocation
}

CHANGED: added drop for v.

BTW, the code as written would not compile because you can't move out of an array - you have to use a pattern on the &move [T] to get the equivalent effect.

[eefriedman]

@arielb1 So this RFC changes the semantics of Box?

struct Foo(i32);
impl Drop for Foo {
    fn drop(&mut self) {
        println!("{}", self.0);
    }
}

fn f(x: Box<(Foo, Foo)>, b: bool) {
    if b {
        drop(x.1);
        // RFC says x.0 gets dropped here
    }
    println!("X");
    // On nightly, x.0 gets dropped here
}

fn main() {
    f(Box::new((Foo(1),Foo(2))), true);
}

[ubsan]

@arielb1 When would v be dropped? I'd think, if b is true, after the drop of tmp0, and if false, at the end of the function.

Note: still thinking about an interesting test case.

[arielb1]

@eefriedman

No. Box is DerefPure, so its semantics does not change. OTOH, that probably means that DerefPure also needs the special destructor treatment - I think that's why I wanted to force DerefMove to be a supertrait of it.

@ubsan

You mean the allocation? It is not moved anywhere, so it will be dropped along with v at the end of the function.

[eefriedman]

A DerefPure Box and an impure Box-like type have a different destruction order? That's probably worth calling out explicitly in the RFC.

[Ericson2314]

I've spent a few days working on a MIR formalization to handle &move, &out, etc. Certainly &move is simpler to add than the others, and my proposal is quite complex. But I'd be wary adding these unsafe traits and whatnot, if we can sacrifice some simplicity for safety. I hope to finish it (or release it in installments) soon.

[eefriedman]

It might be possible to make impure DerefMove work like pure DerefMove by keeping track of a separate "have we called DerefMove yet" flag. Then we just call deref_move the first time we need the value, stash the returned value in a local variable, and reuse the previous value where appropriate. I'm not sure it's a good idea, but I don't see any obvious reason why it wouldn't work.

[durka]

Btw, is DerefPure the same as StableAddress in the sense that owning-ref uses that trait?

[Ericson2314]

Yes, I intend to dig into it shortly. I didn't have time while on vacation.

Ah, I didn't realize you were still on vacation, sorry. Indeed, It's no beach read :).

[ubsan]

This isn't a new mutability. (mutable) Local variables already have this mutability.

[arielb1]

Local variables are currently special-cased.

[ubsan]

That doesn't mean it doesn't exist. I believe prvalues (function returns, constants, etc.) also have this mutability.

[arielb1]

That's a matter of how you structure the specification - bikeshedding.

[ubsan]

Nit: exmaple

[NXTangl]

Mm.

So, what I'm getting here is two things:

1. There should be a way to indicate "borrow and consume, but leave in valid state for Drop::drop()"
2. There should be a way to indicate "We have entirely taken control of this type, and we can choose to mem::drop() or mem::forget() it." This is &move.

Hey, I've thought of something that nobody else seems to have brought up yet: what should PhantomData<&move T> do? Is it just a new way to write mem::forget()? And if so, does that tell us something about the relative usefulness of this feature, or is it just reflecting the fact that &move T is expressing the same thing as T?

If we do use type state for this eventually, will we also add type state for Drop? And if so, wouldn't that be a breaking change?

This also made me realize that type state is, in many cases, doing the same thing as initialization analysis--that is, dropck is arguably just checking a special case of a type-state transition (any sort of consumption is equivalent to &mut<T, Uninit>, where Uninit is a magic type that means 'this variable cannot be read'--which then makes me wonder if

This also made me realize that type state is, in many cases, doing the same thing as initialization analysis--that is, dropck is arguably just checking a special case of a type-state transition. Specifically, you can implement automatic drops with three rules:

1. Any sort of consumption is equivalent to &mut<T, Dropped>, where Dropped is a magic type that means 'this variable cannot be read'.
2. All affine variables must have the typestate Dropped at the time they go out of scope.
3. Calls to Drop::drop are automatically added at each place where it is necessary to change a type's state to Dropped. (However, IIRC, this would change the order in which objects are dropped in cases like {let a = foo(); let b = bar(); if false { drop(a); } baz(); }, though--I think drop is currently called after baz() in all cases, whereas it would be valid to drop it before under my rules. I'm really still not sure whether "Values are dropped at the end of their scope" is actually a better idea "values are dropped at the end of their compiler-checked accessable lifetime", though, since although the former is more accessible to newbies, the latter seems a bit more self-consistent.)

Question: What about BorrowMove?

There is a point to it--here's an example (assuming that &move [T] is added):

impl<T> BorrowMove<[T]> for Vec<T> {
    fn borrow_move<'a>(&'a mut self) -> &'a move [T] {
        let buf = from_raw_parts_move(self.raw.ptr(), self.len);
        self.len = 0;
        buf
    }
}

[ubsan]

ping @arielb1 @nikomatsakis

Status?

[burdges]

Just a few notes from the SafeDeref/SafeDerefMut thread #1873 :

@glaebhoerl suggested that all traits could have an unsafe variant that "raised the stakes" of disregarding the trait's rules from incorrect behavior to undefined behavior. Deref would specify stability/purity, so unsafe Deref would become the DerefPure specified here. Anything that requires <T as Deref>::deref to be stable for safety, like by calling it during drop, should use the bound T: unsafe Deref`.

@glaebhoerl found a crate using what you'd call "AsMutPure" here : https://docs.rs/atomic_ring_buffer/*/atomic_ring_buffer/trait.InvariantAsMut.html And BorrowPure might get assumed occasionally too.

Also @eddyb observed that if a Cow-like type satisfies DerefMut then that would necessarily be impure.

I asked if a DerefField trait could be a safe substitute for unsafe trait ideas like DerefPure, but Cow would not work unless trait fields support enums.

[eddyb]

I don't get why anyone mentioned DerefField - surely any smart pointer needs to dereference a raw pointer and then borrow either the resulting lvalue or a field thereof, how could DerefField work?

EDIT: Oh, you used *self.field on the RHS of an "associated field", which was never proposed to Rust, AFAIK. The problem there is that it becomes wholly unusable with any vtable-based solution, so it would only work through monomorphization and not in trait objects, but that would have to be opt-in.

[burdges]

I wrote it out with just _deref: self first, but wound up with a morass of lifetimes. I felt this would make writing impls for DerefField hard, so I posted something different, but the first version looked like :

pub trait DerefField : Deref {
    type 'target;  // must bound the lifetime somewhere
    _deref: &'target Self::Target
}

impl<'a, T: ?Sized> DerefField for &'a T {
    type 'target = 'a;
    _deref: self
}
impl<'a, T: ?Sized> DerefField for &'a mut T {
    type 'target = 'a;
    _deref: self
}

// Use DerefField for Deref it exists
impl<P> Deref for P where P: ?Sized + DerefField + P::'target {
    // We cannot define P::Target here, but it must exist since DerefField: Deref
    default(std) fn deref(&self) -> &P::Target { self._deref }  // Disallow overloading outside std
}
impl<P> DerefMut for P where P: ?Sized + DerefField + P::'target {
    default(std) fn deref_mut(&self) -> &mut P::Target { self._deref }
}

// Finalize Deref/DerefMut to use DerefField
impl<'a, T: ?Sized> Deref for &'a T {  type Target = T;  }
impl<'a, T: ?Sized> Deref for &'a mut T {  type Target = T;  }
impl<'a, T: ?Sized> DerefMut for &'a mut T {  type Target = T;  }

Initially I wrote this with a P::'target lifetime annotation for the return types of deref and deref_mut, which should be true, but maybe not ideal for the Deref trait definition.

I'd envision smart pointers pointing their own DerefField to the DerefField of the Share<T> or Unique<T> they contain, but maybe it'd break something bigger than Cow.

Edit: I suppose this version might encounter your vtable problem anytime you want to impl DerefField to point to owned data though?

[burdges]

I suppose one could maybe combine both approaches with some negative reasoning, so

pub trait DerefOwned : Deref + !DerefBorrowed {
    _deref: Self::Target
}
impl<P> Deref for P where P: ?Sized + DerefOwned {
    default(std) fn deref(&self) -> &P::Target { &self._deref }
}
impl<P> DerefMut for P where P: ?Sized + DerefOwned {
    default(std) fn deref_mut(&self) -> &mut P::Target { &self._deref }
}

pub trait DerefBorrowed : Deref + !DerefOwned {
    type 'target;  // must bound the lifetime somewhere
    _deref: &'target Self::Target
}
.. Just like DerefField above ..

pub trait DerefPure : Deref { }
impl<T> DerefPure for T where T: DerefOwned { }
impl<T> DerefPure for T where T: DerefBorrowed { }

although this still does not work for Cow since neither variant can satisfy both traits.

All this sounds needlessly complex just to avoid a few unsafe impl anyways. :)

[DemiMarie]

Will this create an ambiguity in the grammar? Can it be resolved by means of precedence?

[nikomatsakis]

@rfcbot fcp close

I would like to propose that we postpone this RFC. I think that the underlying ideas that it has are very good, as I would expect from @arielb1, but I don't feel that the "time is right" for this addition yet. Let me spell out my thoughts here.

To start, though, I think it's helpful to enumerate the motivations
for the RFC. The RFC primarily describes two motivations:

1. making box less special;
2. supporting "deref patterns" in matches.

To those I would add the following:

1. providing a uniform abstraction for thinking about local variable stack allocations in the unsafe code guidelines;
2. passing values of unsized type (such as a trait object or slice) as function parameters;
3. offering the capability to resolve some lingering dissatisfaction with the signature of the Drop trait.

Now I'll go through and spell out why I think we ought to close:

Learning curve. A primary goal of the 2017 roadmap is to decrease the learning curve. My feeling is that this RFC moves somewhat counter to that goal, by introducing new "base reference" types into the language. The main counterargument to this that I have heard is roughly that by making things like Box less special, we ultimately make things less confusing, and I am sympathetic to that, but I think that right now we are focused very early on the learning curve, and this is not yet a factor. "We have bigger problems", in short.

In some cases, simpler solutions exist. Specifically for the problem of passing ownership values of unsized type, I would prefer to see a solution based on allocas. I think @arielb1 agrees, and indeed we elaborated the outlines of such a solution a few days back. =)

Somewhat more controversially, I am not convinced that DerefMove should be specified using "safe types". It might well be acceptable to use raw pointers in the trait, much as we did with the placer traits (though those are not yet stable). That would permit library authors to extend the language with their own abstractions without growing the complexity of day-to-day life in safe code.

In some cases, more elaborate problems exist. If we were going to address extending the base reference types, I think we would want to "get more" than &move gets us. &move does not (for example) solve the problem of out-pointers, nor problems like having a struct with references into its own fields or references into an arena that it owns; it doesn't permit methods that only use disjoint fields to compose.

It might well be the case that we should address these problems as separate extensions to the language, but if we wind up with new types and syntax for all of them, that (in my mind) is a failure state. Rather, if we are going to extend lifetimes and references, I would want to strive for some more unifying concept that allows us to address more things with less mechanism ("eliminate the tradeoff"). I think that @RalfJung's work as well as @Ericson2314's "stateful mir" suggest some pointers as to what these mechanisms might be, not to mention the huge body of research that is out there around affine references, permissions, and the like.

We expect to be learning more in the near future. In the case of the unsafe code guidelines in particular, I feel like work is very much still ongoing there. It seems premature to "bless" &move as the way to think about locals until we've got a more elaborated model. And of course even if we feel like &move is the right way to think about locals, that does not immediately argue that it should be exposed in the type system. Rust has a tradition of maintaining a richer internal state (e.g., the borrow checker) than what can be captured in a function signature, and I think by and large this trade-off has been acceptable.

[rfcbot]

Team member @nikomatsakis has proposed to close this. The next step is review by the rest of the tagged teams:

• @aturon
• @eddyb
• @nikomatsakis
• @nrc
• @pnkfelix
• @withoutboats

No concerns currently listed.

Once these reviewers reach consensus, this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[Ericson2314]

suggest some pointers as to what these mechanisms might be

I hope that pun is intended :D

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[nrc]

I agree that this is good idea at a bad time. I am very excited to see these ideas develop in the long run though.

[gnzlbg]

@nikomatsakis Could you maybe elaborate your thoughts (when you get the time) about everything that you ideally would like an extension to the reference system to buy us if you could have it all?

I think having something like that written down somewhere would make it easier for everybody to frame ideas and proposals like this (and in particular future ones) into a "more global picture".

I don't know, maybe this would belong into a "vision RFC" where it can be discussed, but a blog post with an internal threads could be a start.

[Ericson2314]

https://internals.rust-lang.org/t/idea-extending-the-rfc-process-experimental-features/4832 + something over-the-top-but-not-actually-new-theoretically like my stateful MIR would be a killer masters thesis :D.

[rfcbot]

The final comment period is now complete.

[aturon]

The FCP has elapsed, and it looks like there's still agreement that postponing is the right move for now. We'll surely be revisiting this topic in the future! Thanks @arielb1 for the RFC.

[kennytm]

cc #997

(Just a cc. A DerefMove RFC was not referred anywhere in the DerefMove issue 🙄)