Unboxed closure metabug #18101

Closed
aturon opened this Issue Oct 16, 2014 · 17 comments

Comments

Projects
None yet
@aturon
Member

aturon commented Oct 16, 2014

This is a metabug for tracking work remaining to be done on unboxed closures. It contains only the high-level remaining issues. For a comprehensive list of closure bugs, search using the A-closures label.

1.0 Issues

  • #20872 -- Make Fn traits be treated like everything in trait selection
  • #19982 -- ICE using parenthetical notation in impls
  • #11740 -- Unexplained lifetime error in closure type
  • #18835 -- Unable to manually implement FnOnce
  • #23015 -- Impl FnMut for &mut F where F: FnMut

Completed issues

  • #20871 -- Make return type of the Fn traits an associated type
  • #21603 -- Allow individual upvars to be moved
  • #16640 -- Implement unboxed closure upvar inference
  • #18799 -- Infer move using Send trait
  • #16749 -- |&mut:| wants (Copy) upvar to be mutable when it shouldn't
  • #17881 -- Bad type inference error message with unboxed closures
  • #18875 -- Adjust feature gates
  • #17908 -- clarify lifetimes in closure return type syntax
  • #12831 -- by-ref closure capture semantics are a backwards compatibility hazard
  • #17661 -- Implement higher-ranked trait bounds
  • #14798 -- Remove old (boxed) closures from the language
  • #18387 -- Fn trait hierarchy not respected
  • #18992 -- Parentheses notation should support lifetime elision
  • #19817 -- unboxed closures should not be implicitly copyable
  • #19836 -- should pretty-print with parenthesized notation
  • #19730 -- "perfect forwarding" traits are not accepted by HRTB
  • #17551 -- ICE on generic type inference failure for unboxed closures
  • #17545 -- ICE using explicit lifetime with unboxed closures
  • #17060 -- ICE fictitious type O in sizing_type_of()
  • #17021 -- ICE when using unboxed closures
  • #16939 -- ICE calling an unboxed closure with the wrong number of parameters inside another closure
  • #16808 -- Unboxed closures ICE when debug println!()d
  • #16791 -- ICE related to sizing of unboxes closures
  • #16790 -- ICE in metadata::tydecode::parse_def_id related to unboxes closures
  • #16774 -- Running into segmentation fault with new closures
  • #16739 -- ICE: When creating Box<FnOnce<(), ()>> from unsugared unboxed closure
  • #16672 -- ICE returning unboxed closure
  • #18057 -- internal compiler error
  • #17816 -- Unboxed closure ICE: cannot relate bound region: ReInfer(16) <= ReLateBound
  • #18378 -- Unboxed closures ICE: Expected(expected EBML doc with tag EsEnumVid but found tag 27)
  • #18453 -- Unboxed closures and traits ICE
  • #20582 -- normalization failure in return type in impl
  • #17403 -- Aliasing rules are broken for closures
  • #16560 -- Struct + Fn fulfills Send, but sugary version |&: ...| does not
  • #16929 -- Cannot call a unboxed-closure trait object with () syntax
  • #18385 -- Overloaded call notation doesn't work with trait bounds
  • #16814 -- Unboxed closures: ref-prefixed closures still capture by value
  • #16668 -- Unboxed closures: cannot borrow data mutably in aliasable location
  • #17961 -- Function pointers should implement the various Fn* types
  • #17907 -- Unboxed closures: Type inferencer failed to detect closure argument types
  • #17703 -- unboxed closure sugar not working in trait objects
  • #18772 -- Box<Fn() + 'static> not accepted

@aturon aturon added the metabug label Oct 16, 2014

@huonw huonw added the A-closures label Oct 16, 2014

@aturon

This comment has been minimized.

Show comment
Hide comment
@japaric

This comment has been minimized.

Show comment
Hide comment
@japaric

japaric Nov 2, 2014

Member

Re: #14798 -- Remove old (boxed) closures from the language

Are we going to replace all the uses of boxed closures in the standard library with unboxed closures? I think that in some cases it may not be doable (*) without abstract return types (ART). Let me explain why with a (rather long) story:

(*) Ok, it may be doable, but tedious. See alternatives section at the bottom:


I was navigating ancient issues today, and found #12677: "std::str::Bytes should implement Clone". So I decided to investigate why string.bytes() wasn't cloneable:

The bytes method returns an iter::Map which can't implement clone. This is the current definition of iter::Map:

#[deriving(Clone)]
struct Map<'a, A, B, I> {
    iter: I,
    f: |A|:'a -> B,  //~ error: type `|A| -> B` does not implement any method in scope named `clone`
}

So, my first attempt at a solution was: "Let's change Map to use unboxed closures":

#[deriving(Clone)]
struct Map<A, B, I: Iterator<A>, F: FnMut(A) -> B> {
    iter: I,
    f: F,
}

impl<A, B, I: Iterator<A>, F: FnMut(A) -> B> Iterator<B> for Map<A, B, I, F> {
    fn next(&mut self) -> Option<B> {
        self.iter.next().map(|elt| self.f.call_mut((elt,)))
    }
}

Which compiles fine, it's cloneable and should work in principle, but when I tried to replicate the bytes method, I found a more serious problem:

fn bytes<'a>(slice: &'a str) -> () {
    Map {
        iter: slice.as_bytes().iter(),
        f: |&mut: &b| b,
    }
}

I wrote () as the return type to make the compiler tell me what's the actual return type:

error: mismatched types: expected `()`, found `Map<_, _, core::slice::Items<'_, u8>, closure>` (expected (), found struct Map)

Note the 4th specialization argument of Map: closure. I realized that this function is impossible to define because the return type can't be "named", in particular the unboxed closure has an anonymous type.

If we had ART, then the return type could be Map<&'a u8, u8, slice::Items<'a, u8>, impl <'a> FnMut(&'a u8) -> u8, or more succinctly, just: impl Iterator<u8>.


Alternatives

Since we don't have ART, here are a few alternatives for the particular case of the bytes method:

  • Make bytes return a Bytes struct that implements Iterator<u8>. This basically bypasses the use of unboxed closures, buts add yet another struct to the library.
  • Create a DerefCopy struct that implements the general |:&mut &t| t operation, and change the return type of bytes to Map<_, _, _, DerefCopy>. The DerefCopy could be used in some other places of the stdlib.

@aturon This is relevant to the library stabilization process:

  • As part of the stabilization of std::iter, are we going to fully move to unboxed closures?
  • If yes, how do we dealt with these cases where the return type that can't be named?
  • Are we going to stabilize functions that return iterator adapters? Or perhaps should we mark them as [unstable = "return type may change"] to account for the introduction of ART in the future.
Member

japaric commented Nov 2, 2014

Re: #14798 -- Remove old (boxed) closures from the language

Are we going to replace all the uses of boxed closures in the standard library with unboxed closures? I think that in some cases it may not be doable (*) without abstract return types (ART). Let me explain why with a (rather long) story:

(*) Ok, it may be doable, but tedious. See alternatives section at the bottom:


I was navigating ancient issues today, and found #12677: "std::str::Bytes should implement Clone". So I decided to investigate why string.bytes() wasn't cloneable:

The bytes method returns an iter::Map which can't implement clone. This is the current definition of iter::Map:

#[deriving(Clone)]
struct Map<'a, A, B, I> {
    iter: I,
    f: |A|:'a -> B,  //~ error: type `|A| -> B` does not implement any method in scope named `clone`
}

So, my first attempt at a solution was: "Let's change Map to use unboxed closures":

#[deriving(Clone)]
struct Map<A, B, I: Iterator<A>, F: FnMut(A) -> B> {
    iter: I,
    f: F,
}

impl<A, B, I: Iterator<A>, F: FnMut(A) -> B> Iterator<B> for Map<A, B, I, F> {
    fn next(&mut self) -> Option<B> {
        self.iter.next().map(|elt| self.f.call_mut((elt,)))
    }
}

Which compiles fine, it's cloneable and should work in principle, but when I tried to replicate the bytes method, I found a more serious problem:

fn bytes<'a>(slice: &'a str) -> () {
    Map {
        iter: slice.as_bytes().iter(),
        f: |&mut: &b| b,
    }
}

I wrote () as the return type to make the compiler tell me what's the actual return type:

error: mismatched types: expected `()`, found `Map<_, _, core::slice::Items<'_, u8>, closure>` (expected (), found struct Map)

Note the 4th specialization argument of Map: closure. I realized that this function is impossible to define because the return type can't be "named", in particular the unboxed closure has an anonymous type.

If we had ART, then the return type could be Map<&'a u8, u8, slice::Items<'a, u8>, impl <'a> FnMut(&'a u8) -> u8, or more succinctly, just: impl Iterator<u8>.


Alternatives

Since we don't have ART, here are a few alternatives for the particular case of the bytes method:

  • Make bytes return a Bytes struct that implements Iterator<u8>. This basically bypasses the use of unboxed closures, buts add yet another struct to the library.
  • Create a DerefCopy struct that implements the general |:&mut &t| t operation, and change the return type of bytes to Map<_, _, _, DerefCopy>. The DerefCopy could be used in some other places of the stdlib.

@aturon This is relevant to the library stabilization process:

  • As part of the stabilization of std::iter, are we going to fully move to unboxed closures?
  • If yes, how do we dealt with these cases where the return type that can't be named?
  • Are we going to stabilize functions that return iterator adapters? Or perhaps should we mark them as [unstable = "return type may change"] to account for the introduction of ART in the future.
@aturon

This comment has been minimized.

Show comment
Hide comment
@aturon

aturon Nov 2, 2014

Member

@japaric Note that you will be able to use "unboxed closures" in boxed form, something like: Box<Fn(uint) -> uint> for example. For library stabilization, we will be switching all APIs currently using closures to use the new closure system, but they will not necessarily all be boxed.

This is just the usual tradeoff between generics (with specialization) and trait objects (with dynamic dispatch). Right now, since we don't have abstract "unboxed" return types, anything returning a closure will have to use the trait object (boxed) form.

Put another way, "unboxed closures" is really an unfortunate misnomer. What's really changing is that closures are going away as a "special" thing, and instead being replaced by use of the existing trait system. That means you can choose to use them in both boxed and unboxed forms.

As we've been stabilizing APIs, we've been leaving anything that uses closures as #[unstable] for this reason. We'll be making a sweep later and determining which kinds of closures to use once the feature is ready.

Member

aturon commented Nov 2, 2014

@japaric Note that you will be able to use "unboxed closures" in boxed form, something like: Box<Fn(uint) -> uint> for example. For library stabilization, we will be switching all APIs currently using closures to use the new closure system, but they will not necessarily all be boxed.

This is just the usual tradeoff between generics (with specialization) and trait objects (with dynamic dispatch). Right now, since we don't have abstract "unboxed" return types, anything returning a closure will have to use the trait object (boxed) form.

Put another way, "unboxed closures" is really an unfortunate misnomer. What's really changing is that closures are going away as a "special" thing, and instead being replaced by use of the existing trait system. That means you can choose to use them in both boxed and unboxed forms.

As we've been stabilizing APIs, we've been leaving anything that uses closures as #[unstable] for this reason. We'll be making a sweep later and determining which kinds of closures to use once the feature is ready.

@japaric

This comment has been minimized.

Show comment
Hide comment
@japaric

japaric Nov 2, 2014

Member

Note that you will be able to use "unboxed closures" in boxed form, something like: Box<Fn(uint) -> uint> for example.

I'm aware that we can still use boxed closures, but it seemed a rather bad idea to use a boxed closure for the bytes method, hence I didn't listed it under the alternatives.

As we've been stabilizing APIs, we've been leaving anything that uses closures as #[unstable] for this reason.

Since we have to stabilize (as in mark as #[stable]) most of stdlib so it can be used on the stable/beta channels, does this mean well have to make tradeoffs like this one:

  • Make bytes return Map<_, _, _, Box<Fn(_) -> _>, which uses boxed closures and its bad for performance, or
  • Make bytes return Bytes (which would also need to be marked stable), where Bytes implements Iterator<u8>, which means we'll fill stdlib with more one use structs

right? (FWIW, I prefer the latter option wherever possible)

Member

japaric commented Nov 2, 2014

Note that you will be able to use "unboxed closures" in boxed form, something like: Box<Fn(uint) -> uint> for example.

I'm aware that we can still use boxed closures, but it seemed a rather bad idea to use a boxed closure for the bytes method, hence I didn't listed it under the alternatives.

As we've been stabilizing APIs, we've been leaving anything that uses closures as #[unstable] for this reason.

Since we have to stabilize (as in mark as #[stable]) most of stdlib so it can be used on the stable/beta channels, does this mean well have to make tradeoffs like this one:

  • Make bytes return Map<_, _, _, Box<Fn(_) -> _>, which uses boxed closures and its bad for performance, or
  • Make bytes return Bytes (which would also need to be marked stable), where Bytes implements Iterator<u8>, which means we'll fill stdlib with more one use structs

right? (FWIW, I prefer the latter option wherever possible)

@alexcrichton

This comment has been minimized.

Show comment
Hide comment
@alexcrichton

alexcrichton Nov 2, 2014

Member

In this case Box<Fn(..)> may not be that bad because a 0-sized allocation (there's no environment for most of these functions) costs nothing. We may also be able to get away with &'static Fn(..) depending on how much trans wants to cooperate (in theory).

Member

alexcrichton commented Nov 2, 2014

In this case Box<Fn(..)> may not be that bad because a 0-sized allocation (there's no environment for most of these functions) costs nothing. We may also be able to get away with &'static Fn(..) depending on how much trans wants to cooperate (in theory).

@japaric

This comment has been minimized.

Show comment
Hide comment
@japaric

japaric Nov 2, 2014

Member

In this case Box<Fn(..)> may not be that bad because a 0-sized allocation (there's no environment for most of these functions) costs nothing.

Doesn't each call still has to go through the trait object tough? Or is LLVM able to optimize it to a function call (or maybe even better: inline it away)?

Member

japaric commented Nov 2, 2014

In this case Box<Fn(..)> may not be that bad because a 0-sized allocation (there's no environment for most of these functions) costs nothing.

Doesn't each call still has to go through the trait object tough? Or is LLVM able to optimize it to a function call (or maybe even better: inline it away)?

@alexcrichton

This comment has been minimized.

Show comment
Hide comment
@alexcrichton

alexcrichton Nov 3, 2014

Member

Yes, but that's precisely what happens today (relying on LLVM to inline), as in all closures are boxed today.

Member

alexcrichton commented Nov 3, 2014

Yes, but that's precisely what happens today (relying on LLVM to inline), as in all closures are boxed today.

@TeXitoi

This comment has been minimized.

Show comment
Hide comment
@TeXitoi

TeXitoi Nov 3, 2014

Contributor

At worst, if you want to return an unboxed closure, you can write the struct by hand and not rely on the syntaxic sugar. It correspond to your DerefCopy alternative. And you can write polymorphic closure! http://is.gd/iqdHfp

Contributor

TeXitoi commented Nov 3, 2014

At worst, if you want to return an unboxed closure, you can write the struct by hand and not rely on the syntaxic sugar. It correspond to your DerefCopy alternative. And you can write polymorphic closure! http://is.gd/iqdHfp

@cristicbz

This comment has been minimized.

Show comment
Hide comment
@cristicbz

cristicbz Nov 5, 2014

Contributor

Found two more ICE-s today: #18652 (to do with move capture) and #18661 (to do with passing Fn-s as args).

Contributor

cristicbz commented Nov 5, 2014

Found two more ICE-s today: #18652 (to do with move capture) and #18661 (to do with passing Fn-s as args).

@cristicbz

This comment has been minimized.

Show comment
Hide comment
@cristicbz

cristicbz Nov 8, 2014

Contributor

A whole bunch of these bugs have been fixed:

ICE-s
#16790
#16774
#18378
#18453

Other bugs
#17703

Contributor

cristicbz commented Nov 8, 2014

A whole bunch of these bugs have been fixed:

ICE-s
#16790
#16774
#18378
#18453

Other bugs
#17703

@carllerche

This comment has been minimized.

Show comment
Hide comment
@carllerche

carllerche Nov 9, 2014

Member

Infer move using Send trait: #18799

Member

carllerche commented Nov 9, 2014

Infer move using Send trait: #18799

@abonander

This comment has been minimized.

Show comment
Hide comment
@abonander

abonander Nov 19, 2014

Contributor

Can check off #17661. HRTB landed today.

Contributor

abonander commented Nov 19, 2014

Can check off #17661. HRTB landed today.

@abonander

This comment has been minimized.

Show comment
Hide comment
@abonander

abonander Nov 20, 2014

Contributor

Just opened #19135, related to HRTBs and lifetimes on unboxed closures.

Contributor

abonander commented Nov 20, 2014

Just opened #19135, related to HRTBs and lifetimes on unboxed closures.

@alexchandel

This comment has been minimized.

Show comment
Hide comment
@alexchandel

alexchandel Nov 30, 2014

An ICE related to rust-call: #16039

An ICE related to rust-call: #16039

@tomaka

This comment has been minimized.

Show comment
Hide comment
@tomaka

tomaka Nov 30, 2014

Contributor

Also #19374

(this is very selfish, but the glium library will become unusable if #19338 lands before #19374 is fixed)

Contributor

tomaka commented Nov 30, 2014

Also #19374

(this is very selfish, but the glium library will become unusable if #19338 lands before #19374 is fixed)

@bstrie

This comment has been minimized.

Show comment
Hide comment
@bstrie

bstrie Dec 5, 2014

Contributor
Contributor

bstrie commented Dec 5, 2014

@steveklabnik

This comment has been minimized.

Show comment
Hide comment
@steveklabnik

steveklabnik May 7, 2015

Member

Given that only two issues remain, I don't think this metabug is particularly valuable anymore, so I'm gonna close it.

Member

steveklabnik commented May 7, 2015

Given that only two issues remain, I don't think this metabug is particularly valuable anymore, so I'm gonna close it.

@Eh2406 Eh2406 referenced this issue Jan 2, 2017

Closed

Document all features in the reference #38643

0 of 17 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment