Introduce the `IntoIterator` trait and reimplement for-loops to use it

[japaric]

As per RFC #235, you can now do:

let mut v = vec![1];

// iterate over immutable references
for x in &v {
    assert_eq!(x, &1);
}

// iterate over mutable references
for x in &mut v {
    assert_eq!(x, &mut 1);
}

// iterate over values, this consumes `v`
for x in v {
    assert_eq!(x, 1);
}

[breaking-change]s

For loops now "consume" (move) the iterator, this breaks iterating over mutable references to iterators, and also breaks multiple iterations over the same iterator:

fn foo(mut it: &mut Iter) {  // `Iter` implements `Iterator`
    for x in it { .. }  //~ error: `&mut Iter` doesn't implement Iterator
}

fn bar() {
    for x in it { .. }  //~ note: `it` moved here
    for x in it { .. }  //~ error: `it` has been moved
}

Both cases can be fixed using the by_ref() adapter to create an iterator from the mutable reference:

fn foo(mut it: &mut Iter) {
    for x in it.by_ref() { .. }
}

fn bar() {
    for x in it.by_ref() { .. }
    for x in it { .. }
}

This PR also makes iterator non-implicitly copyable, as this was source of subtle bugs in the libraries. You can still use clone() to explictly copy the iterator.

Finally, since the for loops are implemented in the frontend and use global paths to IntoIterator, Iterator and Option variants, users of the core crate will have to use add an std module to the root of their crate to be able to use for loops:

#![no_std]

extern crate core;

fn main() {
    for x in 0..10 {}
}

#[doc(hidden)]
mod std {
    // these imports are needed to use for-loops
    pub use core::iter;
    pub use core::option;
}

---

r? @nikomatsakis @aturon
cc #18424
closes #18045

[rust-highfive]

r? @alexcrichton

(rust_highfive has picked a reviewer for you, use r? to override)

[huonw]

Can we do this without expanding in the front end?

[japaric]

@huonw @nikomatsakis was interested in an implementation that used the frontend (hence this prototype)

[huonw]

Also, expanding to include _ => break will mean that refutable patterns like for 1 in iter will "work", which seems like something that we don't want to allow.

[japaric]

(I think @aturon and @nikomatsakis mainly want to measure the size of the fallout and the change in ergonomics of these new for-loop semantics before commiting to it. This implementation is minimal work to measure that. We can come up with a better implementation if we decide to actually do this.)

[alexcrichton]

Could some loops like this be altered to:

for ch in self.iter.by_ref() { /* ... */ }

That may avoid consuming the entire iterator.

[alexcrichton]

Actually, if it helps in basically all cases, I wonder if we could spec it like this...

[japaric]

I had forgotten about by_ref()! I think we can use that instead of most (all?) the while lets.

Actually, if it helps in basically all cases, I wonder if we could spec it like this...

You mean using by_ref() in the for loop expansion?

[japaric]

Updated to use the by_ref() adapter instead of while let.

[nikomatsakis]

Some thoughts:

1. As I mentioned to @japaric on IRC, I think we can expand to match <iterator> { mut iter => ... } instead of { let mut iter = <iterator>; .. } to resolve the matter of temporary lifetimes.
2. Personally, I prefer to do this expansion than to have for loops in the AST that typeck/borrowck/everybody-else has to deal with. Basically desugaring makes the rest of the compiler simpler. However, the catch would be if it negatively affects error messages and so forth. So I'm curious @japaric as to the impact there. There is also the problem of our inability to really name things in std reliably, though this is ameliorated by the fact that #![no_std] is not stable yet.

[jroesch]

@japaric and @nikomatsakis As a fan of desugaring as much as possible I think this seems like a good idea. If error messages become a problem you could use a similar scheme to what they do with Pyret where you provide a tag (either in some table or directly on the AST) for which rewrite rule you used, and then utilize that information when reporting errors by re-sugaring it then pretty printing. I would be happy to help explore such a scheme if the error reporting suffers.

[japaric]

Updated PR with @nikomatsakis suggestions (match IntoIterator + UFCS with absolute paths, see expansion in the top comment), which got rid of the temporary lifetime issue, reduced the size of the import fallout and removed the need to have IntoIterator in the prelude. I'm now blocked on a type inference bug, but @nikomatsakis is aware of it.

I've collected some error messages from 3 cfail tests related to for-loops:

• for-loop-bogosity.rs

struct MyStruct {
    x: isize,
    y: isize,
}

impl MyStruct {
    fn next(&mut self) -> Option<isize> {
        Option::Some(self.x)
    }
}

fn main() {
    let mut bogus = MyStruct {
        x: 1,
        y: 2,
    };
    for x in bogus {
//~^  old-ERROR has type `MyStruct` which does not implement the `Iterator` trait
//~^^ new-ERROR the trait `iter::Iterator` is not implemented for the type `MyStruct`
    }
}

The error message changed its wording but the meaning is the same.

• for-loop-refutable-pattern-error-message.rs

fn main() {
    for
        &1i8
//~^  old-ERROR refutable pattern in `for` loop binding
//~^^ new-ERROR non-exhaustive patterns: `Some(&_)` not covered
        in [1i8].iter() {}
}

The message is more obscure now.

• for-loop-type-error.rs

fn main() {
    let x = () + (); //~ ERROR binary operation

    // this shouldn't have a flow-on error:
    // japaric: and it doesn't with the either version of for loops
    for _ in x {}
}

Behavior remains the same.

[alexcrichton]

Now that we have cfg_attr, could this be #![cfg_attr(stage0, allow(unused_mut))]? Anything with the string "stage0" in it is a pretty easy flag for something to clean out when dealing with snapshots.

[nikomatsakis]

@japaric and I were talking on IRC. My feeling is that the error in this example:

fn main() {
    for
        &1i8
//~^  old-ERROR refutable pattern in `for` loop binding
//~^^ new-ERROR non-exhaustive patterns: `Some(&_)` not covered
        in [1i8].iter() {}
}

is actually better. I doubt most "lay people" know what a "refutable pattern" is. (I personally have to work out from first principles which is "irrefutable" and which is "refutable".) But having a concrete counter example seems immediately understandable. It might be better still to combine the two messages:

"refutable pattern in `for` loop binding: `Some(&_)` not covered"

[huonw]

@nikomatsakis I think introducing the Some out of nowhere is confusing (and is not very useful since it happens for all invalid iterators, and is not related to how one might fix it, in the common case).

[japaric]

This PR is ready! (passed check-stage1, running full make check now)

re: error message on refutable patterns, I decide to reuse the old message/diagnostic code, we can customize it later if desired.

[japaric]

rebased and added tests for issues that new for loops fix. Also fixed a recursive call that the new unconditional_recursion lint pointed out. (last 3 commits are new)

[japaric]

Just a heads up, until #21637 gets fixed, we won't be able to use the IntoIterator trait to build iterator-centric APIs. Code like this one ICEs right now:

fn concat<I: IntoIterator>(it: I) -> String where <I::Iter as Iterator>::Item: Str {
    unimplemented!();
}

[alexcrichton]

oops!

[japaric]

ugh..
/me snipes file

[bors]

⌛️ Testing commit b9a9030 with merge e47284b...

[alexcrichton]

@bors: retry

[bors]

⌛️ Testing commit b9a9030 with merge e513946...

[bors]

💔 Test failed - auto-linux-64-opt

[alexcrichton]

@bors: retry

[bors]

⌛️ Testing commit b9a9030 with merge 581a5c8...

[bors]

💔 Test failed - auto-linux-64-x-android-t

[alexcrichton]

@bors: retry

[bors]

⌛️ Testing commit b9a9030 with merge 692c620...

[bors]

💔 Test failed - auto-linux-32-opt

[alexcrichton]

@bors: retry

[bors]

⌛️ Testing commit b9a9030 with merge 72eab3e...

[taralx]

The RFC didn't mention this "mod std" business, and I'm a bit uncomfortable with it. Why not directly access the core crate?

[japaric]

Why not directly access the core crate?

If you mean expanding to ::core::iter::IntoIterator, that won't work in most cases, since most crates are linked to std but not to core (they have access to core stuff because of re-exports).

The RFC didn't mention this "mod std" business, and I'm a bit uncomfortable with it.

You already have to use the mod std to use the #[derive] syntax extension and/or slicing syntax in #![no_std] crates. At the moment this required for stuff that's implemented in the frontend, but we'd definitely like to improve this situation in the future.

[japaric]

@taralx see #21912 ;-)

[alexchandel]

The relevant section of the Rust Reference ought to be updated to reflect this, cc @steveklabnik