Make Number.range() return re-usable value

[sffc]

I would consider separating the iterator from the object returned from Number.range(), such that Number.range() returns an immutable object (with getter properties like .from, .to, etc), and then calling [Symbol.iterator] returns a "fresh" iterator with a .next() method. You can re-use the ranges. In other words, we should consider making the following code work:

const range = Number.range(0, 5);
for (let i of range) {
  // 0, 1, 2, 3, 4
}
for (let i of range) {
  // 0, 1, 2, 3, 4
}

This is what we are doing in the Intl.Segmenter proposal.

https://github.com/tc39/proposal-intl-segmenter/issues

CC @gibson042

[littledan]

This is sort of what I assumed the interface would be as well: Number.range() would return an iterable, so for-of loops work over it just fine. However, for the example in the README using iterator helpers, it'd be a little more verbose: you'd need Iterator.from(Number.range(...)).take(...). I could see the tradeoff either way, but mentally, for me, the "default" design would be an iterable.

[Jack-Works]

So reusable and easy to use iterator helpers are conflict...
I prefer the latter one but the reusable is also important.
Maybe add a .clone() to return a new clear RangeIterator with the same from, to and step?

[sffc]

We decided against returning an iterator from Intl.Segmenter.prototype.segment() after long discussions about ergonomics (see, e.g., tc39/proposal-intl-segmenter#93, and other issues in that proposal). Basically, if the return value of Number.range(...) should be considered a "thing", then you should make it have its own prototype that's an iterable but not an iterator.

I think we should explore other options here. For example, if you write Number.range(...)[Symbol.iterator].take(...), does that feel more ergonomic than Iterator.from(Number.range(...)).take(...)?

[Jack-Works]

Iterator.from(...) and [Symbol.iterator] is basically the same in this case, they are both not ergonomics for chaining iterator helpers on it.

[littledan]

So reusable and easy to use iterator helpers are conflict...

Yes, this is a thing that's been under discussion for iterator helpers in general. It's also important to be able to use them for Arrays, which are iterable but not iterators.

I liked this story: You can use Iterator.from for these situations, and we expect this to be ergonomic enough. (If we don't, I wonder if we could make some change to the iterator helpers proposal for a more ergonomic path...)

[devsnek]

Iterators are already iterable, so the only discussion point seems to be re-usability. I'd argue if you want to reuse it you should call it again, it's more explicit and its weird to me that you'd need to make two calls (range()[Symbol.iterator]()) to begin consuming an iterator you directly asked for.

[littledan]

My intuition around the iterator protocol was that we're sort of generalizing part of the functionality Arrays. Arrays support [Symbol.iterator] to iterate through them, and other objects do too. I was picturing that Number.range would produce something which was also this same kind of "generalization of an Array", and that's why my intuition was that it wouldn't be an iterator, but rather a reusable iterable.

@Jack-Works Patching it up with .clone() would restore the functionality kinda, but it wouldn't meet the goal of making Number.range follow a common protocol to other iterable objects. Maybe we could make .clone() a more general thing, but I'd like to see if we could continue following the split of the initial iterable/iterator protocol, which already handles these two things.

@devsnek What in particular is weird about this? If you have a range function that returns an Array, you'll also need to call the Symbol.iterator method to iterate over it. Accordingly, since they should also work for Arrays, lots of usage sites will be calling Symbol.iterator anyway.

[ljharb]

It's the way generators already work - the generator is called (perhaps with arguments) to produce the (iterable) iterator. The generator itself is not iterable, you have to call it every time you want to iterate.

[littledan]

@ljharb I agree with this description of what generators are, but I don't see how that applies here and why Number.range shouldn't return something analogous to Arrays.

[devsnek]

The correct pattern here is definitely to return an iterator. If we want iterables for some reason the pattern you're looking for is a Number.Range class.

If it helps:
Class -> Instance (Iterable or callable) -> Iterator
Array -> array -> ArrayIterator
Map -> map -> MapIterator
Set -> set -> SetIterator
String -> string -> StringIterator
Number.Range -> numberRange -> NumberRangeIterator
Generator -> generator -> GeneratorIterator (Number.range ~= generator)
Array -> array.entries -> ArrayIterator
Array -> array.values -> ArrayIterator
Map -> map.entries -> MapIterator
Map -> map.values -> MapIterator
Map -> map.keys -> MapIterator
...

[littledan]

@devsnek OK, interesting analysis. I'll have to think about this some more. I'm wondering how/whether it'd apply to @sffc 's example of Intl.Segmenter (which is definitely an instance method, but also a slightly awkward case due to the random access methods).

[devsnek]

Another way to think of it: JS moved reusability out of the "Iterator" protocol itself (f() -> iterator is very convenient). Iterable is just the reification of "object with Symbol.iterator".

@littledan I'm admittedly not very familiar with the Intl.Segmenter proposal but a cursory glance seems to be this, which holds to the pattern but is kind of weird:
Intl.Segmenter -> segmenter -> segmenter.SegmentsForString (via segmenter.segment() helper) -> segmentsForString -> SegmentsForStringIterator

It's worth noting that Intl has always had different patterns compared to the rest of the stdlib, most notably in their constructor patterns (Intl.NumberFormat(x).format(n) vs hypothetical Intl.formatNumber(x, n))

[littledan]

Well, I think Intl constructors are a good design that could make sense among anything TC39 creates, but we're getting a bit off-topic. For new APIs, I'd like to see if we can use common design patterns that make sense in general, and having Intl.Segmenter follow the iteration protocol (unlike Intl.v8BreakIterator) was intended to be part of that.

[devsnek]

my point was basically that both Number.range() and Intl.Segmenter().segement follow the pattern as far as i can tell, but i wouldn't really equate one directly to the other, because they have to (and do) represent different things.

On topic again, Number.range is itself the reusable bit (that is the generator pattern). You can just do const whateverRange = () => Number.range(from, to); and then you get the explicit bonus of having to do for (const item of whateverRange()) instead of for (const item of whateverRange) where its not immediately apparent whether or not you're doing something reusable (because as i said above, Iterable in js only means there is a Symbol.iterator method, not that it will create fresh state)

[Jack-Works]

Another thought: (working but it's a crazy design)

Add a @@iterator on the iterator itself to make it auto-cloneable.

const range = Number.range(0, 5);
for (let i of range) { } // use range[@@iterator] which return a cloned range
for (let i of range) { }  // use range[@@iterator] which return a cloned range
// range is not consumed yet.
range.take(...).toArray() // now it's consumed

[devsnek]

iterators already have @@iterator on them. But aside from that, doesn't

for (let i of range()) { }
for (let i of range()) { }

seem like a much more obvious way of saying you're using a separate range for each one? I don't understand why you want to make that implicit.

[gibson042]

An Intl.Segmenter Segments instance, like an array/set/map/string/etc., is Iterable but not itself an Iterator—it is primarily a factory for constructing iterators, and also supports a containing method of its own for returning a single random-access iteration result without actually constructing an iterator. In our opinion, it would be negatively surprising for the same object to support both next() and containing(index), especially if index can refer to a position that has already been described by a previous iteration result, so we instead separated the two roles by defining the Symbol.iterator method on the iterable to always return a new iterator (again, just like array/set/map/string/etc.).

So the question here is whether Number.range is an iterator factory or returns an iterator factory. I personally can't imagine much use for the latter, and to me Number.range(0, 10) is much more like Array(10).keys() (a trivial iterator that basically supports only next()) than it is like Array.from(Array(10), (_, i) => i) (an object with its own state and methods that are independent of any constructed iterator(s)).

[Jack-Works]

If there are no objections, I'll change it to return a Range object with [Symbol.iterator] on it later.
Wait for more discussions

Return an iterator

Current design.
Problem:

• cannot be reused.
• it may be strange to add to many items on an iterator prototype

Number.range(...): RangeIterator<number>

Behave like Symbol constructor

Problem:

• maybe not subclassable
• not match the mental modal?

Number.range(...): Range
class Range {
    [Symbol.iterator]
    get from()
    get to()
    get step()
}

A normal class

Problem:

• Usage become unacceptable long (Iterator.from(new Number.Range(...)).take(...).toArray(...)))

class Range {
    constructor(from, to, step)
    [Symbol.iterator]()
    get from()
    get to()
    get step()
}
Number.Range = Range
BigInt.Range = Range

[devsnek]

I think I've objected to that idea a few times over at this point...

[sffc]

I think it's a question we should discuss at plenary, since not everyone is in agreement on this thread.

[littledan]

We discussed the issue in plenary, but I don't think we reached a particular conclusion. What are the next steps?

[Jack-Works]

I'm going to study the ranges in other languages to make a comparison then decide which pattern is better 👀👀

[Jack-Works]

I'm going to study the ranges in other languages to make a comparison then decide which pattern is better 👀👀

Interestingly, I found this re-usable problem in Rust. Rust is using the Iterator semantics. (Two for in doesn't pass the borrow checker so I manually call the next() to consume it.)

fn main() {
    let mut a = std::ops::Range { start: 3, end: 5 };
    print!("Used: {:?}\t", a.next());
    for i in a {
        print!("In loop: {:?}\t", i)
    };
}

Used: Some(3) In loop: 4

But I think manually call the next method is too explicit, and I think others try to consume the iterator twice will not pass the borrow checker. So I have no idea is this really a problem in Rust.

[Jack-Works]

In my recent research of range in other languages (not completed, you can see it at https://github.com/tc39/proposal-Number.range/blob/master/compare.md). I changed my mind and now preferring to the Iterable semantics now (or other ways to keep the iterator semantics but can be re-use safely).

But before switching to the Iterable semantics, there're a few problems that we need to resolve.

Use with Iterator Helper

// Now
Number.range(0, 10).take(5).toArray()
// After
Iterator.from(Number.range(0, 10)).take(5).toArray()

I have an idea at tc39/proposal-iterator-helpers#78 (comment) but I'm not sure about it.

The naming problem

range for Iterator and Range for iterable (the idea of @devsnek)

Let it to be a class

// before
Number.range(0, 1)
// after
new Number.Range(0, 1)
// or Callable class
Number.Range(0, 1)

I have no idea if adding a new callable class like Array is acceptable today.

Let it to be a helper to create the Range class

Number.range(...) // implicitly calls new Range(...)

Another possible route: merge into Iterator namespace, becomes Iterator.range

Therefore the developers can clearly know, the return value is an iterator. And all iterators are not re-usable.

[ljharb]

Can you elaborate on why you think it would be better to add an entire class for something that seems only likely to be used to generate a single iterator?

[ljharb]

I'm not sure why that would be necessary; range() would of course return something that was already a proper Iterator. Either way, using Iterator.from is like Array.from or Promise.resolve, which in practice have demonstrated themselves to be quite usable - there's no ergonomics issue I'm aware of.

To be clear, I am still convinced that unless this returns an iterator directly, it should not be added to the language.

[Jack-Works]

Add two new slides https://docs.google.com/presentation/d/116FDDK2klJoEL8s2Q7UXiDApC681N-Q9SwpC0toAzTU/edit#slide=id.g8c83f0abc4_2_5 and https://docs.google.com/presentation/d/116FDDK2klJoEL8s2Q7UXiDApC681N-Q9SwpC0toAzTU/edit#slide=id.g8c83f0abc4_2_27. If I missed something please tell me before the presentation so I can add them to provide a more complete view

[hax]

@ljharb As the https://github.com/tc39/proposal-Number.range/blob/master/compare.md#return-type , most other programming languages (we can investigate more languages if needed) do not returns an one-time consumed iterator directly.

Either way, using Iterator.from is like Array.from or Promise.resolve, which in practice have demonstrated themselves to be quite usable - there's no ergonomics issue I"m aware of.

I think @Jack-Works means writing Iterator.from(Number.range(1, 10)) is much painful than Number.range(1, 10).values(). Personally I am neutral of it. This argument is only focus on whether range is ergonomics enough to use iterator helpers.

[ljharb]

While JS should always be informed by other languages, it is absolutely not constrained by them - new JS features should follow JS idioms, and values/keys/entries/matchAll very much define that idiom.

Again, I don't understand why Iterator.from would be required at all - I'd expect to be able to write Number.range(x, y).map(…).take(…) etc.

[Jack-Works]

Again, I don't understand why Iterator.from would be required at all - I'd expect to be able to write Number.range(x, y).map(…).take(…) etc.

Yes. I'd like to do it so. Iterator.from is used to convert iterators that do not have %IteratorPrototype% to use with the helpers (or get an iterator from the iterable).

If we decided the re-usable problem is important, either we need range(...).values().map(...), Iterator.from(range(...)) or range(...)[Symbol.iterator](), either the Iterator Helpers needs to support Iterables as the "this" value.

IteratorPrototype.map = function* (f) {
    const iter = Iterator.from(this)
    // ...
}

Then, set %RangeIteratorPrototype%.[[Prototype]] to %IteratorPrototype%, we can have both re-usable semantics and range(...).map(...) conversions.

[ljharb]

I would assume range() would return an iterator that already inherited from %IteratorPrototype%, making Iterator.from a no-op for it.

Like every builtin iterator, it would also have a Symbol.iterator method that does return this, so as to allow direct for..of usage.

[hax]

new JS features should follow JS idioms, and values/keys/entries/matchAll very much define that idiom.

@ljharb

Iterators/Generators and related Ranges are very common feature in many languages, especially JS iterators/generators is inspired by python design. The core concept are very same: Iterables are objects which can be iterated, iterators are special object which delegate the iteration behavior for iterables, generators are syntax which help implement iterators in user land.

So the common cases are a method returns an object, and it happened to have implemented iterable protocol. Actually return iterators directly are special cases.

If we see values/keys/entries as "idiom" we should defined it accordingly :

1. They are all instance methods on iterable object. (Number/BigInt.range() is static method which not match)
2. They are all methods with no argument. (Number/BigInt.range() have three arguments)
3. They just return plain iterators (only next() or protocol methods like @@toStringTag), no other addition (range already have additional accessors/methods)

In the context of this issue, these important natures make values/keys/entries do not have reusable issue, because u can just reuse the iterable object and nothing u can use on plain iterators except iteration. Number/BigInt.range are just the opposite.

matchAll is very new, and I don't think it stand for any "idiom" up to now, but we still could check the natures:

1. matchAll is an instance method on iterable (string).
2. matchAll have one argument
3. matchAll return plain iterators

So it's more close to values/keys/entries not range.

We should also notice Intl.Segmenter intentionally choose iterable even it much close to matchAll on the second point, the main difference is it need some addition like containing method.

I think @sffc have given a very reasonable comment:

Basically, if the return value of Number.range(...) should be considered a "thing", then you should make it have its own prototype that's an iterable but not an iterator.

And I believe range is a "thing" which beyond solely iteration behavior, for example, python range is used to returns an array, but change to iterable in python3, so it's a "thing" which array like but don't need waste memory.

[devsnek]

If you think range should be an iterable please specify it as a constructor to stay consistent with our language patterns.

[hax]

@devsnek Do u mean it should be new Number.Range ? Personally I don't think it's necessary , Number.range is a static method which could be a factory, but i'm ok with new Number.Range. This should be a separate issue.

[devsnek]

As per the chart I posted above it needs to be an iterable class or a function that returns an iterator. I would not be comfortable with anything else.

[hax]

@devsnek

I'm not sure, is this only apply to iterable? Why only iterables need to be constructor and can't created by a static factory method? What if we have both NumberRange and Number.range()? Do u think we also need to change Intl.Segmenter?

And again, let's first decide whether it should be iterable (semantic issue) and then consider surface api problems.

[Andrew-Cottrell]

And again, let's first decide whether it should be iterable (semantic issue) and then consider surface api problems.

I think the reason I feel uncomfortable with Number.range returning a reusable iterable entity is that I view it as a lightweight operation over an implicit iterable entity. The start and end arguments define an interval and, although I currently see no particular need, it would be reasonable to have a first-class Number.Interval type that implements the iterable protocol. Then Number.range would — ignoring side issues — be equivalent to

Number.range = function ( start, end, step ) {
    return new Number.Interval( start, end ).values( step ); // an iterator over the interval
};

Even without a first-class Number.Interval type, I still deem Number.range an operation in which the step argument enables me to specify how I would like to iterate over the interval.

Perhaps Number.range could return an iterator, and another proposal would consider a first-class type — Number.Interval, Number.Sequence, or Number.Stream — that implements the iterable protocol.

---

Note (2022-07-07): #57 makes a similar suggestion to split the proposal into two proposals.

[devsnek]

So. When we started working on iterator helpers, it was not clear whether they would be iterator helpers or iterable helpers, and the main disagreement was around how reusability is represented in iterators and iterables. In particular people argued, like you do, that using iterators instead of iterables meant you lost reusability. The conclusion we came to however, was that in JS the primitive of reuse was not actually the iterables, but function calls. This is mainly due to the fact that "iterable" does not describe a non-scalar type that can produce an iterator, but rather any object that happens to have a Symbol.iterator property, including iterators themselves. The signs of reusability of iterators were actually one of the following: The base being an instance of some non-scalar type (like Array.prototype[Symbol.iterator]), or getting the iterator directly from an explicit function call. I want to stress that this is not just my sole opinion of how you should use iterables, but the result of research over many months by many people to understand what the best way to deal with iterators in JS should be, and it allowed us to gain consensus and move forward with that proposal. Number is a scalar type, and Number.range is a function. These two points strongly suggest that the return value should be an iterator, not an iterable.

Finally, my opinion on this specific choice: I think you should choose Number.range over NumberRange because I think function calls generally provide a better way of being explicit about reusability than class instances, and the pattern also protects you in codebase-wide reuse. For example, export let x = () => Number.range() vs export let x = new NumberRange(). The latter is just some mutable object hanging around, not very clear about what patterns it follows.

Do u think we also need to change Intl.Segmenter?

Intl is somewhat different because it uses the factory pattern for all its top level apis, unlike the rest of our stdlib. I don't agree with the design but I don't think it would be constructive to discuss changing it at this point.

[hax]

@devsnek Is there any link of details about "research over many months by many people to understand what the best way to deal with iterators in JS should be" so I can learn it??

And I want to point out, we are not (only) talking about reuse iterator, but by itself, range (or interval), could be a "thing" beyond iteration.

[devsnek]

@hax

Is there any link of details about "research over many months by many people to understand what the best way to deal with iterators in JS should be" so I can learn it??

old issues about iterables in the proposal repo i guess? might be some stuff in the tc39 irc logs too.

And I want to point out, we are not (only) talking about reuse iterator, but by itself, range (or interval), could be a "thing" beyond iteration.

That's totally valid, it would just be a NumberRange constructor instead of Number.range.

[Jack-Works]

Another possible real-world usage that requires reusable value: facebook/react#20707

[tabatkins]

And again, that's a general iterator issue, and wouldn't be affected by anything we do with Range in any case.

[Jack-Works]

I believe this issue has been resolved by renaming it to Iterator.range (advanced to stage 2 on today's meeting). Now it's impossible to accidentally use it as a reusable value.

[jpolo]

The problems with polymorphic functions are :

• Not extensible => the function has to change its signature to support future types (SOLID open/close principle) Also imagine the scenario if BigDecimal is released after your proposal.
• Not easy to polyfill (test if Number.range != null, BigInt.range != null VS "Check if Iterator.range throw an exception if my type is supported")

[ljharb]

Polyfillability doesn’t impact proposals.

[hax]

@jpolo

The motivation and the main use cases of the proposal is about simple iteration of integers (like 1 to 100), change name to Iterator.range not only solve the dilemmatic arguments of "Iterable vs Iterator" but also make the API usage and intention clearer. Actually I am not sure there are strong use cases of BigInt.range (and Decimal.range), and there are footguns of iteration floats (see #64 ), so I would ok if the proposal would eventually only support safe integers (and renaming to Iterator.integers), and never extensible to future types.

On the other side, when we went to Iterator.range, Jack and I also discussed possibility of extend to any type which implement "range" protocol (like Stride protocol in Swift). If we really want this API be "open" to any future types, I think protocol is the right way (not adding range method to every new type), and protocol would also easy to polyfill.

[ogbotemi-2000]

Closed and all but had to drop this
A way to decide whether Number.range should return an iterable or an iterator is to examine after it is called:

• Iterable
  Returning an iterable such as new Map(...).entries(), [].values() offers the gain of dynamically generating an array of arbitrary length and the loss of having to wait until the iterable is ready before trying to .map, .filter etc, them

An allternative may be to provide room for a callback for each yielded value

• Iterator
  Offers a lot of room for creativity and improvement while allowing yielded values to be used with no delay. However, you have to ask, do range in Python, which may be an inspiration for this proposal, work this way? If not, what are the gains of this approach as regards the expected ergonomics of Number.range

Renaming it to Iterator.range only subtly hides this conundrum

[ljharb]

.entries() and .values() return iterators (which happen to be iterable). There's no such thing as a generic "iterable", so you'd have to define a "thing" that happens to be iterable to return.

[hax]

@ogbotemi-2000 Yes, python (and most programming languages) range api return re-useable iterables not one-shot iterators. Personally I would like the API follow python. Unfortunately it's very controversial in the committee. Renaming is the best and tradeoff I can tell, or we will never advance.