RFC: Collections reform

[aturon]

This is a combined conventions and library stabilization RFC. The goal is to establish a set of naming and signature conventions for std::collections.

The major components of the RFC include:

• Removing the traits in collections.
• A general proposal for solving the "equiv" problem, as well as improving MaybeOwned.
• Patterns for overloading on by-need values and predicates.
• Initial, forwards-compatible steps toward Iterable.
• A coherent set of API conventions across the full variety of collections.

A big thank-you to @Gankro, who helped collect API information and worked through an initial pass of some of the proposals here.

Rendered

[aturon]

cc #17

[Gankro]

Pretty sure these need to be dereffed, unless the semantics of operators have changed from underneath me.

[Gankro]

Signatures still missing names for the uints.

[andrew-d]

typo: s --> self (and also in borrow_mut, below)

[Gankro]

It's a static function, not an instance method.

Edit: This is because many structures can implement Borrow for the same type. E.G. &String and &str are both Borrow for String as far as I can tell.

[andrew-d]

Ah, fair enough then. That being said: the selfs below should probably be changed to s then 😄

[aturon]

@andrew-d Thanks, fixed!

[P1start]

Now that borrow is an instance method, s should probably be changed to self.

[Gankro]

//

[Gankro]

mut_values is certainly possible, but probably pushing it.

[alexcrichton]

This would also yield patterns like:

for x in v { ... } // iterate over Foo
for x in v.iter() { ... } // iterate over &Foo

These both seem quite ergonomic and quite tempting, and look fairly similar, yet yield different results. I'd be ok with it, does it sound ok to you?

[Gankro]

It's probably unlikely that you wouldn't notice very quickly if you used the wrong one. Although I could see this burning newbies.

[aturon]

@alexcrichton Yes, the fact that the "default" interpretation is a moving iterator is perhaps a bit weird. On the other hand, moving is the default almost everywhere in Rust, and with this approach you get to use & and &mut to easily select other forms of iteration.

Unfortunately, it's a bit tricky to make for use by-ref iterators instead. The problem is that an iterator is IntoIterator, but it is not Iterable (or whatever we call the by-reference trait). Why? Because IntoIterator gives you an iterator that can be used only once, while Iterable allows you to ask for iterators repeatedly.

If for demanded an Iterable, then for x in v.iter() and for x in v.iter_mut() would cease to work -- we'd have to find some other approach. It might be doable, but offhand I don't see how.

[aturon]

(I've added this text to the RFC)

[zkamsler]

Making for use IntoIterator exclusively would prevent one from using an Iterator after partially iterating over it with a for. This is possible now, since it is mutably borrowed. Ideally, for would treat an Iterator differently from a non-iterator, but then it would have to be more than a parser desugaring.

[thestinger]

I don't think moving out of a container should be the default iterator. I'm strongly against adding an Iterable trait until there's a proposal without it. Containers shouldn't be treated as cheap temporaries. Moving from a container via an iterator is far from being the common case. It's such an edge case that more sugar for it is totally unnecessary and will lead people down the wrong path.

[pcwalton]

I don't see how moving out encourages people to treat containers to be cheap temporaries. You still have to create the container in the first place, via .collect() or Vec::new() or whatnot.

[huonw]

It renders the container unusable later, so people will apply the clone hammer; e.g. with a scenario like

let v: Vec<int> = ...;
let mut result = 0;
for x in v {
    result += f(x);
}

return (result, v);

I'm 100% sure that there will be people who fix the compiler error with for x in v.clone() (and I would guess that it won't be a trivial number). I also have a feeling that consuming iteration is likely to be the least efficient in general:

• large values will be (in general) unoptimisably memcpy'd around
• when iterating over something complicated like treemap, consuming it will probably have to be quite careful to avoid crashes; possibly at the expense of speed

[aturon]

A couple of points here:

1. I don't see any simple way to make for use an Iterable-style API that defaults to by-reference, for reasons that I outlined here. Basically, an iterator can only implement a by-value Iterable API.

   Please let me know if you see a way around this, though.

2. In terms of shaping what people do, note that the ergonomics of for x in v and for x in &v are almost identical. So I think the main thing is documentation. When we first introduce iteration in the docs, we would surely show for x in &v and explain what the & is doing. If we make sure that programmers are made aware of that from their first encounter with iteration, I think it's unlikely that they'd reach for .clone.

3. Moving is the default almost everywhere in Rust. To me, writing for x in &v serves as a nice signal to the reader that references are being taken. It also plays quite nicely with the deref coercions that I've recently proposed.

[huonw]

I do agree that it's hard to do a by ref iterator with a naive Iterable API.

There are some points in the other direction:

• I believe unconditionally calling IntoIterator (i.e. not dectecting and handling Iterators specially) will stop rust-lang/rust#8372 being possible, since the iterator for ... in iter { ... } wlll be moved (or duplicated) and thus either iter will not be usable in the body, or it will not refer to the same thing as the for loop head.

• Some containers can only provide &, i.e. & is the lowest common denominator.

• On that note, it is weird that for x in v is by value for Vec but by reference for &[], assuming this is the case, anyway (and there's a similar difference for for x in map e.g. if map is a function argument that is passed as &HashMap<...> vs. if it is a local).

• If we make sure that programmers are made aware of that from their first encounter with iteration, I think it's unlikely that they'd reach for .clone.

  Well, people do a lot of passing Vec and String by value now, even when they could just pass & if not &str or &[]; maybe that's just a problem with our documentation.

[alexcrichton]

Could this actually take T: Borrow to convert Option<&str> to Option<String>?

[aturon]

It could indeed! (Well, it'd take T: ToOwned).

And perhaps, more generally, we may find that a ToOwned bound is more useful than a Clone bound in many cases, for exactly this reason.

[alexcrichton]

For strings, if T is char, would it be possible to leverage this method to test if a string contains a substring?

[aturon]

No; we'll need a separate contains_str variant.

[aturon]

Alternatively, we could use a double-dispatch trick to have the String contains method be more general than the interface given here. That's probably fine, and we can make the final decision as part of String stabilization.

[Gankro]

I skimmed/skipped the first 1000 lines assuming it hasn't changed much since last I saw it. Overall I'm a big 👍 to this proposal's contents. Some misc notes that I don't think you ever discussed/addressed:

• Predicate as written might be better served as a special case of Comparators (on hold pending full unboxed closures). As it stands, it strikes me as odd that equality is "the" element predicate. In particular a function like lower_bound wouldn't be able to use this kind of predicate. This could be more flexible if what we requested was effectively a "partially applied comparator". Or perhaps we can just have a family of traits EqPred, LePred, GePred, etc?
• Maps love only yielding V from queries in our current design. Would it be possible to yield (K, V) for more map methods? To my knowledge, when you insert a key-value pair, the key always also gets swapped. Some people might want that Key back? With tuple indexing this should be a trivial ergonomic loss. It also means Sets (which are trivial wrappers around maps) can start telling you about their contents beyond "yeah, I got that" or "Okay, I removed something".

[aturon]

@Gankro I addressed the various typos you pointed out.

Predicate as written might be better served as a special case of Comparators (on hold pending full unboxed closures). As it stands, it strikes me as odd that equality is "the" element predicate. In particular a function like lower_bound wouldn't be able to use this kind of predicate. This could be more flexible if what we requested was effectively a "partially applied comparator". Or perhaps we can just have a family of traits EqPred, LePred, GePred, etc?

I don't think methods like lower_bound make sense to take a more general closure, since they rely intrinsically on ordering. I think that's true more generally: if you have a method that can take an arbitrary predicate on a type, it's unlikely that the natural way to interpret an element of that type is via some ordering.

That said, we could consider using PredEq or some such name, to leave room for adding other predicates with different defaults in the future if they turn out to be wanted.

Maps love only yielding V from queries in our current design. Would it be possible to yield (K, V) for more map methods? To my knowledge, when you insert a key-value pair, the key always also gets swapped. Some people might want that Key back? With tuple indexing this should be a trivial ergonomic loss. It also means Sets (which are trivial wrappers around maps) can start telling you about their contents beyond "yeah, I got that" or "Okay, I removed something".

That's certainly doable. It's a tradeoff, since it makes the API somewhat more complex for a pretty rare use case.

My plan for key recovery was to go through your collection view's RFC, which offers a bit more of a swiss army knife for working with maps. We'll have to think about sets, though.

[epdtry]

It seems to me that it would be more flexible to define Borrow with both type parameters being "inputs", instead of one input and one output. For example:

trait Borrow<Sized? B> {
    fn borrow(&self) -> &B;
}

impl<T: Sized> Borrow<T> for T { ... }
impl Borrow<str> for String { ... }
impl<T> Borrow<[T]> for Vec<T> { ... }

(I'm not 100% clear about the rules for blanket impls post-trait reform, but the one above seems like it ought to work, based on the blanket Borrow impl that is currently in the RFC.)

Now if you define a custom string type, you can still get the benefits of Borrow without needing to define your own wrapper around str:

struct MyString(String);
impl Borrow<str> for MyString { ... }

It also would let you have String borrow to more than one type:

impl Borrow<[u8]> for String { ... }

Then HashMap would look like this:

struct HashMap<K, V>;
impl<K, V> HashMap<K, V> {
    fn insert(&mut self, k: K, v: V) { ... }
    fn find<B>(&self, k: &B) -> Option<&V> where K: Borrow<B> { ... }
}

And you can still define Cow, though it's not as pretty since you need to specify both the owned and borrowed types:

trait FromBorrow<Sized? B>: Borrow<B> {
    fn from_borrow(b: &B) -> Self;
}

enum Cow<'a, T, B: 'a> where T: FromBorrow<B> {
    Owned(T),
    Shared(&'a B),
}

type MaybeOwned<'a> = Cow<'a, String, str>;

Did you consider implementing Borrow this way? I see the "borrowed -> owned" version in the main body of the RFC, and a sort of "owned -> borrowed" version in the alternatives, but nothing with both "owned" and "borrowed" as inputs.

[zkamsler]

I want to like the idea of Borrow, but it can be limiting. It precludes the use of equiv-like keys for HashMap<MyCoolString, int> and the like. String and Vec<T> are not the only things that could be 'borrowed' to str and [T]. They are good defaults, but it feels strange to bless them to the exclusion of everything else.

[Gankro]

Hmm, this just occurred to me. The proposal states we would "deprecate" the traits. Is this intended literally (as in #[deprecated]), or will they just be removed overnight? Due to naming conflicts I'm pretty sure that the traits can't coexist in a deprecated form with the proposed concrete impls.

Or will we have a brief period where the traits are deprecated as a warning, with no (merged) replacement?

[sfackler]

@Gankro:

This RFC proposes a somewhat radical step for the collections traits: rather than reform them, we should eliminate them altogether -- for now.

[Gankro]

@sfackler I assumed that wording was simply referring to the standard deprecation -> removal path. I was (am?) uncertain about the nature of the transition.

[reem]

I might just be being dense, but is there a reason for the name Cow?

[sfackler]

Copy on write I'd assume.

[Gankro]

One thing I keep remembering and forgetting, and don't really know where to put it, so I'm just going to put here since it's relevant to this work:

Instead of all this lower_bound and upper_bound nonsense, we could probably just provide one method on sorted collections:

/// Creates an iterator over a sub-range of the collection's items, 
/// starting at min, and ending at max. If min is `None`, then it will
/// be treated as "negative infinity", and if max is `None`, then it will
/// be treated as "positive infinity". Thus range(None, None) will yield
/// the whole collection.
fn range(&self, min: Option<&T>, max: Option<&T>) -> SubItems<'a, T>;

This iterator should be DoubleEnded and therefore reversible. This of course doesn't address the inclusive/exclusive scenario, for which we can either do the same thing as range and include an inclusive/exclusive variant, or we can change the Option to a Tristate enum:

Bound<T> {
    Included(T),
    Excluded(T),
    Unbounded,
}

which is the solution I prefer.

[thestinger]

It shouldn't produce a vector. Creating containers is very expensive and lazy iterators should be encouraged. An API producing temporary container will result in countless accidentally temporaries, as demonstrated by the code conventions in the compiler.

[thestinger]

I don't think this would do a good job conveying that these are the basic set operations to most people. Unless looking up union in the documentation will find iter_or... I don't think it will today.

[aturon]

That's a fair point. But one problem is that there are really several versions of the "basic set operations": those returning iterators, those returning new sets, and those mutating self. I'd love to hear any proposal for a naming convention that covers all of these cases more clearly.

[alexcrichton]

Tracking issue now made!