RFC: movable array iterators

[ishitatsuyuki]

Rendered

[SimonSapin]

I very much want this, but what is return type of [T; N]::into_iter? I think it should be IntoIter<T, N>, with const generics. So this needs to wait for that to be implemented.

Or maybe IntoIter<A> where A: Unsize<[T]> might work, but stabilizing with Unsize itself unstable doesn’t feel right.

---

Transmute the array so it has ManuallyDrop signature. This should be memory safe.

I believe that in a method taking self, the array can be safely moved into ManuallyDrop::new(…) without transmute. Hopefully this move optimizes to a no-op?

[ishitatsuyuki]

Please prefer line comments so I can respond without copying :)

I believe that in a method taking self, the array can be safely moved into ManuallyDrop::new(…) without transmute. Hopefully this move optimizes to a no-op?

I think there's a need to be able to drop each elements individually (in other words, only in the range that is not moved out). So we need ManuallyDrop on each element, and trasmuting should be the easiest so far.

[SimonSapin]

You can call drop_in_place(…) selectively on individual items or a sub-slice inside of ManuallyDrop<[T; N]>.

[ishitatsuyuki]

Good. I will update the text later.

[bluss]

This is pretty much ensured to break actual code for many users, both in crater (that we can test, find and maybe fix), and lots of code we can't find or test. Maybe we can begin warning for this long before the new iterator is implemented?

[ishitatsuyuki]

Personally speaking, I don't think people would use into_iter despite the fact it yields references, and they would just iter instead. Thus I think fixing after a crater should be sufficient.

[bluss]

We don't reach all rust code in the world with crater. If the break is wide spread, we have to be more careful.

[main--]

And what about for &x in [1,2,3], isn't that just into_iter?

[bluss]

That does not compile today I think so it's not a backwards compat concern.

[main--]

Oh I see. [1].into_iter() currently resolves to the impl for &[_; _] which the new [_; _] impl would override, right?

[bluss]

Yes!

[cristicbz]

@SimonSapin

I very much want this, but what is return type of [T; N]::into_iter? I think it should be IntoIter<T, N>, with const generics. So this needs to wait for that to be implemented.

This could be done without stabilising Unsize, by doing:

impl<T> Iterator for IntoIter<[T; 1]> { ... }
impl<T> Iterator for IntoIter<[T; 2]> { ... }
impl<T> Iterator for IntoIter<[T; 3]> { ... }
// ...
impl<T> Iterator for IntoIter<[T; 32]> { ... }

with all the impls calling to some shared unsafe method to avoid duplication. Has a nice migration story for when const generics land, just replacing with

impl<T, const N> Iterator for IntoIter<[T; N]> {}

[ishitatsuyuki]

Please take a look, I think I have addressed the concerns.

[SimonSapin]

This looks great to me, thanks for writing it up!

[sinkuu]

Drop impls cannot be specialized, so we may have to use UnsafeUnsize or FixedSizeArray if we implement this RFC before const generics.

struct IntoIter<T> { ... }
impl<T> Drop for IntoIter<[T; 0]> { ... }
impl<T> Drop for IntoIter<[T; 1]> { ... } // error: Implementations of Drop cannot be specialized
...

// I expect this will work
struct IntoIter<T, const N: usize> { ... }
impl<T, const N: usize> Drop for IntoIter<T, N>

// This works today
struct IntoIter<T, A> where A: FixedSizeArray<T> { ... }
impl<T, A> Drop for IntoIter<T, A> where A: FixedSizeArray<T> { ... }

EDIT: typo

[eddyb]

@sinkuu What do you mean by Unsafe?

[ishitatsuyuki]

Presumably Unsize.

[ishitatsuyuki]

Should we make multiple IntoIter structs until const generic lands? Like this: IntoIter32<T>.

I think this should resolve the specialization problem, alongside reducing the template complexity, so we can store pointers (or unsafe slices) inside the struct.

[SimonSapin]

IntoIter32<T>

I want this feature, but I don’t think I want to stabilize such an API.

It’d be harder to generalize once const generics land, too.

so we can store pointers (or unsafe slices) inside the struct

I don’t understand what you mean here. The point of into_iter is that the items move when the iterator moves. Pointers should not be involved.

[ishitatsuyuki]

but I don’t think I want to stabilize such an API.

I'm afraid that even IntoIter<T> has to change into IntoIter<T, const N> so signature change is unavoidable.

I don’t understand what you mean here.

We need to save the range we have consumed with the iterator. It can be either usize-indexed or just start-end pointer.

[SimonSapin]

Changing the signature of stable APIs is not acceptable, so I’m afraid that this implies we should not stabilize anything here without const generics.

Self-referential pointers would become invalid when the iterator is moved. It has to be indices. (Two of them, or a Range, to support DoubleEndedIterator.)

[ishitatsuyuki]

I've come up with a partial stabilization plan, please let me know if this is possible.

As the signature of the iterator struct is subject to change, we will be keeping that struct(s) unstable until const generics land. I will call this struct UnstableIntoIter below.

IntoIterator is implemented like below, using the length of 32 as an example:

impl<T> IntoIterator for [T; 32] {
    type Item = T;
    type IntoIter = UnstableIntoIter32<T>;

    fn into_iter(self) -> T { ... }
}

I have no idea if this implementation will be accepted as stable in the compiler, so we can use it without the ability to annotate the type.

[kennytm]

Instead of a specific IntoIter or UnstableIntoIter method, could it just return an existential type impl Iterator<Item=T>?

[scottmcm]

Would it be compatible to replace

struct IntoIter0<T>{ .. };
..
struct IntoIter32<T>{ .. };

with

type IntoIter0<T> = IntoIter<T, const 0>;
..
type IntoIter32<T> = IntoIter<T, const 32>;

once const generics land?

Then the eventual type would be the nice one, and the residual cruft is just a bunch of deprecated type aliases, which should be easy to maintain and ignore in the docs.

[bluss]

We can do one thing before we even start the implementation: Work on deprecating & future warning the existing array_value.into_iter() calls (those that resolve to the slice's iterator).

[ishitatsuyuki]

I experimented with an implementation of deprecation warning, but it seems that I can't avoid one particular false positive: https://play.rust-lang.org/?gist=8b69b43bc947bf112c0f20d0e58b3add&version=stable

Basically, the differences between the below cannot be detected without changing the signature:

a.into_iter();
(&a).into_iter();

[sfackler]

We discussed this at the @rust-lang/libs meeting today, and feel like this won't really be actionable until integer generics are implemented.

@rfcbot fcp postpone

[rfcbot]

Team member @sfackler has proposed to postpone this. The next step is review by the rest of the tagged teams:

• @BurntSushi
• @Kimundi
• @KodrAus
• @alexcrichton
• @aturon
• @dtolnay
• @sfackler
• @withoutboats

No concerns currently listed.

Once these reviewers reach consensus, this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[bluss]

@sfackler What do you think about starting to warn / soft deprecate [1, 2, 3].into_iter() in favour of [1, 2, 3].iter() ?

[ishitatsuyuki]

I have mentioned that if we don't stabilize the iterator types, we can implement it with macro generated iterators and partially stabilize the into_iter() function only.

[sfackler]

@bluss

Not sure - we haven't really done anything like that before. @aturon thoughts?

@ishitatsuyuki

I am personally not a fan of the current "up to size 32" approach, and don't really want to expand that further.

[ishitatsuyuki]

@sfackler - what we only can do currently is that "up to size 32" approach; it should not be a blocker to improve array usability.

I originally thought that the main concern is backwards compatibility, which needs a lint to be developed. As I emphasized above, const generics is not a blocker for this, just a wishlist item. As const generics is an approved RFC, we can expect the stabilization at the same time.

[aturon]

@bluss I'd be in favor.

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[rfcbot]

The final comment period is now complete.

[Centril]

Closing since FCP with the proposal to postpone is now complete.