New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking issue for Pin APIs (RFC 2349) #49150

Open
withoutboats opened this Issue Mar 18, 2018 · 203 comments

Comments

Projects
None yet
@withoutboats
Contributor

withoutboats commented Mar 18, 2018

Tracking issue for rust-lang/rfcs#2349

Blocking stabilization:

  • Implementation (PR #49058)
  • Documentation

Unresolved questions:

  • Should we provide stronger guarantees around leaking !Unpin data?

Edit: Summary comment: #49150 (comment) (in the hidden-by-default part)

@Nemo157

This comment has been minimized.

Contributor

Nemo157 commented Mar 20, 2018

I now notice that stack pinning is not part of the RFC, @withoutboats are you planning on releasing a crate for this, or should I just copy the example code into my crate that needs it?

@withoutboats

This comment has been minimized.

Contributor

withoutboats commented Mar 21, 2018

@Nemo157 You should copy it and report your experience!

The unresolved question about leaking Unpin data relates to this. That API is unsound if we say you cannot overwrite Unpin data in a Pin unless the destructor runs, as @cramertj requested. There are other, less ergonomic, stack pinning APIs that do work for this. Its unclear what the right choice here is - is the ergonomic stack pinning API more useful or is the extra guarantee about leaking more useful?

One thing I'll note is that the stack pinning was not sufficient for things like Future::poll inside the await! macro, because it didn't allow us to poll in a loop. I'd be interested if you run into those issues, and how/if you solve them if you do.

@Nemo157

This comment has been minimized.

Contributor

Nemo157 commented Mar 21, 2018

My current usage is pretty trivial, a single-threaded executor running a single StableFuture without spawning support. Switching to an API like @cramertj suggests would work fine with this. I have wondered how to extend this to allow spawning multiple StableFutures, but at least with my current project that's not necessary.

@valff

This comment has been minimized.

Contributor

valff commented Mar 29, 2018

Just tried to experiment with the API. Looks like the following (suggested by RFC) definition of Future is no longer object-safe?

trait Future {
    type Item;
    type Error;

    fn poll(self: Pin<Self>, cx: &mut task::Context) -> Poll<Self::Item, Self::Error>;
}
@valff

This comment has been minimized.

Contributor

valff commented Mar 29, 2018

Nevermind. Found a note about a plan to make arbitrary_self_types object-safe.

@RalfJung

This comment has been minimized.

Member

RalfJung commented Mar 29, 2018

@withoutboats

One thing I'll note is that the stack pinning was not sufficient for things like Future::poll inside the await! macro, because it didn't allow us to poll in a loop.

Could you elaborate on that?

@cramertj

This comment has been minimized.

Member

cramertj commented Mar 29, 2018

@RalfJung You need Pin to support reborrows as Pin, which it currently does not.

@RalfJung

This comment has been minimized.

Member

RalfJung commented Mar 30, 2018

@cramertj That sounds like a restriction of the Pin API, not of the stack pinning API?

@cramertj

This comment has been minimized.

Member

cramertj commented Mar 30, 2018

@RalfJung Yes, that's correct. However, PinBox can be reborrowed as Pin, while the stack-pinned type cannot (one borrow as Pin creates a borrow for the entire lifetime of the stack type).

@RalfJung

This comment has been minimized.

Member

RalfJung commented Mar 30, 2018

Given a Pin, I can borrow it as &mut Pin and then use Pin::borrow -- that's a form of reborrowing. I take it that's not the kind of reborowing you are talking about?

@cramertj

This comment has been minimized.

Member

cramertj commented Mar 30, 2018

@RalfJung No-- methods like Future::poll were planned to take self: Pin<Self>, rather than self: &mut Pin<Self> (which isn't a valid self type since it isn't Deref<item = Self> -- it's Deref<Item = Pin<Self>>).

@withoutboats

This comment has been minimized.

Contributor

withoutboats commented Mar 30, 2018

It might be the case that we could get this to work with Pin::borrow actually. I'm not sure.

@RalfJung

This comment has been minimized.

Member

RalfJung commented Mar 30, 2018

@cramertj I did not suggest to call poll on x: &mut Pin<Self>; I thought of x.borrow().poll().

@cramertj

This comment has been minimized.

Member

cramertj commented Mar 30, 2018

@RalfJung Oh, I see. Yes, using a method to manually reborrow could work.

@cramertj

This comment has been minimized.

@Nemo157

This comment has been minimized.

Contributor

Nemo157 commented Mar 30, 2018

I'll try and remember to post an example of some of the stuff I'm doing with Pins next week, as far as I can tell reborrowing works perfectly. I have a pinned version of the futures::io::AsyncRead trait along with working adaptors like fn read_exact<'a, 'b, R: PinRead + 'a>(read: Pin<'a, R>, buf: &'b [u8]) -> impl StableFuture + 'a + 'b and I'm able to work this into a relatively complex StableFuture that's just stack pinned at the top level.

@Nemo157

This comment has been minimized.

Contributor

Nemo157 commented Apr 3, 2018

Here's the full example of what I'm using for reading:

pub trait Read {
    type Error;

    fn poll_read(
        self: Pin<Self>,
        cx: &mut task::Context,
        buf: &mut [u8],
    ) -> Poll<usize, Self::Error>;
}

pub fn read_exact<'a, 'b: 'a, R: Read + 'a>(
    mut this: Pin<'a, R>,
    buf: &'b mut [u8],
) -> impl StableFuture<Item = (), Error = Error<R::Error>>
         + Captures<'a>
         + Captures<'b> {
    async_block_pinned! {
        let mut position = 0;
        while position < buf.len() {
            let amount = await!(poll_fn(|cx| {
                Pin::borrow(&mut this).poll_read(cx, &mut buf[position..])
            }))?;
            position += amount;
            if amount == 0 {
                Err(Error::UnexpectedEof)?;
            }
        }
        Ok(())
    }
}

This is slightly annoying as you have to pass instances through everywhere as Pin and use Pin::borrow whenever you call functions on them.

#[async]
fn foo<'a, R>(source: Pin<'a, R>) -> Result<!, Error> where R: Read + 'a {
    loop {
        let mut buffer = [0; 8];
        await!(read_exact(Pin::borrow(&mut source), &mut buffer[..]));
        // do something with buffer
    }
}
@Nemo157

This comment has been minimized.

Contributor

Nemo157 commented Apr 3, 2018

I just had a thought that I could impl<'a, R> Read for Pin<'a R> where R: Read + 'a to workaround having to pass values as Pin<'a, R> everywhere, instead I could use fn foo<R>(source: R) where R: Read + Unpin. Unfortunately that fails because Pin<'a, R>: !Unpin, I think it's safe to add an unsafe impl<'a, T> Unpin for Pin<'a, T> {} since the pin itself is just a reference and the data behind it is still pinned.

@RalfJung

This comment has been minimized.

Member

RalfJung commented Apr 4, 2018

Concern: It seems likely that we want most types in libstd to implement Unpin unconditionally, even if their type parameters are not Pin. Examples are Vec, VecDeque, Box, Cell, RefCell, Mutex, RwLock, Rc, Arc. I expect most crates will not think about pinning at all, and hence only have their types be Unpin if all their fields are Unpin. That's a sound choice, but it leads to unnecessarily weak interfaces.

Will this solve itself if we make sure to implement Unpin for all libstd pointer types (maybe even including raw pointers) and UnsafeCell? Is that something we will want to do?

@withoutboats

This comment has been minimized.

Contributor

withoutboats commented Apr 4, 2018

Will this solve itself if we make sure to implement Unpin for all libstd pointer types (maybe even including raw pointers) and UnsafeCell? Is that something we will want to do?

Yes, it seems like the same situation as Send to me.

@RalfJung

This comment has been minimized.

Member

RalfJung commented Apr 6, 2018

A new question just occurred to me: When are Pin and PinBox Send? Right now, the auto trait mechanism makes them Send whenever T is Send. There is no a priori reason to do that; just like types in the shared typestate have their own marker trait for sendability (called Sync), we could make a marker trait saying when Pin<T> is Send, e.g. PinSend. In principle, it is possible to write types that are Send but not PinSend and vice versa.

@withoutboats

This comment has been minimized.

Contributor

withoutboats commented Apr 6, 2018

@RalfJung Pin is Send when &mut T is Send. PinBox is Send when Box<T> is Send. I see no reason for them to be different.

@RalfJung

This comment has been minimized.

Member

RalfJung commented Apr 6, 2018

Well, just like some types are Send but not Sync, you could have a type relying on "Once this method is called with Pin<Self>, I can rely on never being moved to another thread". For example, this could give rise to futures that can be sent around before being started for the first time, but then have to remain in one thread (much like they can be moved around before being started, but then have to remain pinned). I'm not sure if I can come up with a convincing example, maybe something about a future that uses thread-local storage?

kennytm added a commit to kennytm/rust that referenced this issue Sep 7, 2018

Rollup merge of rust-lang#53874 - withoutboats:pin-ptr-impls, r=RalfJung
Implement Unpin for Box, Rc, and Arc

Per the discussion in rust-lang#49150, these should implement `Unpin` even if what they point to does not.

kennytm added a commit to kennytm/rust that referenced this issue Sep 7, 2018

Rollup merge of rust-lang#53874 - withoutboats:pin-ptr-impls, r=RalfJung
Implement Unpin for Box, Rc, and Arc

Per the discussion in rust-lang#49150, these should implement `Unpin` even if what they point to does not.
@RalfJung

This comment has been minimized.

Member

RalfJung commented Sep 12, 2018

I just realized we forgot about one situation where rustc will copy stuff around--- it's probably not a big deal though, for now. I am talking about packed structs. If a packed struct has a field that needs dropping, rustc will emit code to copy that field's data to somewhere aligned, and then call drop on that. That is to make sure that the &mut passed to drop is actually aligned.

For us, it means that a repr(packed) struct must not be "structural" or "recursive" wrt. pinning -- its fields cannot be considered pinned even if the struct itself is. In particular, it is unsafe to use the pin-accessor-macro on such a struct. That should be added to its documentation.

@alercah

This comment has been minimized.

Contributor

alercah commented Sep 13, 2018

That seems like a giant footgun that should be plastered over all of the pinning docs.

@cramertj

This comment has been minimized.

Member

cramertj commented Sep 13, 2018

@alercah I don't think it's much more of a footgun than the existing ability to impl Unpin or Drop-- certainly both are far more commonly seen alongside pinned values than #[repr(packed)].

@alercah

This comment has been minimized.

Contributor

alercah commented Sep 13, 2018

That's fair. My concern is that someone might think that a projection is safe for a type that they didn't write, and not realize that packed makes it unsafe to do so, because it's decidedly non-obvious. It's true that whoever is doing the projection is responsible for being aware of this, but I think it needs to be additionally documented anywhere where such a projection might occur.

Hm, would this also mean that all packed structs could be Unpin regardless of field types since the pinning is not recursive?

@withoutboats

This comment has been minimized.

Contributor

withoutboats commented Sep 13, 2018

@alercah I'm not super familiar with how packed types are implemented, but I believe its safe to pin to a packed type, just not to pin through a packed type. So the only case where you don't control the packed type is if you project to a public field of someone else's type, which could be just as unsafe because of Drop or anything else. In general, it seems illadvised to pin project to someone else's fields unless they provide a pin projection indicating that its safe.

@withoutboats

This comment has been minimized.

Contributor

withoutboats commented Sep 13, 2018

Hm, would this also mean that all packed structs could be Unpin regardless of field types since the pinning is not recursive?

A use case I could imagine is an intrusive linked list, where you only care about the address of the whole struct, but its got a bunch of badly aligned data you want to squash together without caring about the address of that data.

bors added a commit that referenced this issue Sep 17, 2018

Auto merge of #53877 - withoutboats:compositional-pin, r=aturon
Update to a new pinning API.

~~Blocked on #53843 because of method resolution problems with new pin type.~~

@r? @cramertj

cc @RalfJung @pythonesque anyone interested in #49150

bors added a commit that referenced this issue Sep 18, 2018

Auto merge of #53877 - withoutboats:compositional-pin, r=aturon
Update to a new pinning API.

~~Blocked on #53843 because of method resolution problems with new pin type.~~

@r? @cramertj

cc @RalfJung @pythonesque anyone interested in #49150

bors added a commit that referenced this issue Sep 19, 2018

Auto merge of #53877 - withoutboats:compositional-pin, r=aturon
Update to a new pinning API.

~~Blocked on #53843 because of method resolution problems with new pin type.~~

@r? @cramertj

cc @RalfJung @pythonesque anyone interested in #49150
@Pauan

This comment has been minimized.

Member

Pauan commented Oct 22, 2018

I've had to learn the Pin API due to my work with Futures, and I have a question.

  • Box unconditionally implements Unpin.

  • Box unconditionally implements DerefMut.

  • The Pin::get_mut method always works on &mut Box<T> (because Box unconditionally implements Unpin).

Taken together, that allows for moving a pinned value with entirely safe code.

Is this intended? What is the rationale for why this is safe to do?

@tikue

This comment has been minimized.

Contributor

tikue commented Oct 22, 2018

It looks like you have Pin<&mut Box<...>>, which pins Box, not the stuff inside Box. It's always safe to move a Box around, because Box never stores references to its location on the stack.

What you likely want is Pin<Box<...>>. Playground link.

@tmandry

This comment has been minimized.

Contributor

tmandry commented Oct 22, 2018

EDIT: no longer accurate

@Pauan Just because a Box is pinned does not mean the thing inside it is pinned. In other words, you cannot get a Pin<Foo> from a Pin<Box<Foo>>, and the latter does not provide the same guarantees as the former. Any code depending on this would be incorrect.

The thing you are looking for is probably PinBox, which disallows the behavior you mentioned, and allows you to get a PinMut<Foo>.

@tikue It’s still possible to move out of a Pin<Box<...>>, which I think is what they were trying to avoid.

@tikue

This comment has been minimized.

Contributor

tikue commented Oct 22, 2018

@tmandry Correct me if I'm wrong, but Pin<Box<...>> pins the thing inside the Box, not the Box itself. In @Pauan's original example, they had a Pin<&mut Box<...>>, which only pinned the Box. See my playground link showing how Pin<Box<...>> prevents getting a mutable reference to the thing in the box.

Note that PinBox was recently removed, and Pin<Box<T>> now has the same semantics as PinBox.

@Pauan

This comment has been minimized.

Member

Pauan commented Oct 22, 2018

@tmandry PinBox<T> has been removed and replaced with Pin<Box<T>> on Nightly (the doc link you gave is for Stable). Here is the correct Nightly link.

@tmandry

This comment has been minimized.

Contributor

tmandry commented Oct 22, 2018

Oh, the rules must have changed since I last used these. Sorry for the confusion.

@Pauan

This comment has been minimized.

Member

Pauan commented Oct 22, 2018

@tmandry Yes, the changes were very recent. Since things are still in flux, it's hard to keep up with all the changes.

@withoutboats

This comment has been minimized.

Contributor

withoutboats commented Oct 22, 2018

The comment by @tikue is correct. You need to remember that pins only pin one level of indirection down.

@Pauan

This comment has been minimized.

Member

Pauan commented Oct 23, 2018

@tikue @tmandry @withoutboats Thanks for the answers! It was very helpful.

@crlf0710

This comment has been minimized.

Contributor

crlf0710 commented Oct 28, 2018

So what are the states of the two concerns now? (api-refactor & get_mut_unchecked_mut_mut) As someone who's eagerly waiting for the async/await series feature, I wonder which rustc version will the Pin APIs target? Is there an estimation?

@withoutboats withoutboats referenced this issue Nov 7, 2018

Open

[Stabilization] Pin APIs #55766

0 of 5 tasks complete
@aturon

This comment has been minimized.

Member

aturon commented Nov 9, 2018

@crlf0710 see the stabilization proposal.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment