Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Why just on iterators and not on arrays? #1

Closed
ljharb opened this issue Aug 15, 2023 · 53 comments
Closed

Why just on iterators and not on arrays? #1

ljharb opened this issue Aug 15, 2023 · 53 comments

Comments

@ljharb
Copy link
Member

ljharb commented Aug 15, 2023

I have this need on Arrays just as often as on non-array iterators; any reason not to add it to both?

@michaelficarra
Copy link
Member

See the README:

do we support iterators and iterables like Iterator.from and flatMap?

Arrays are iterable, so they would "just work" if we answer "yes" to this question. Also, even if we answer "no", it's pretty easy to turn arrays into iterators with .values() or Iterator.from().

@bakkot
Copy link
Collaborator

bakkot commented Aug 16, 2023

so they would "just work" if we answer "yes" to this question

Well, you'd get an iterator out rather than an array, presumably? So you'd need to .toArray at the end. Iterator.zip(x, y).toArray() is a bit more awkward than Array.zip(x, y).

@ljharb
Copy link
Member Author

ljharb commented Aug 16, 2023

What I mean is, why not have these methods also on Array/Array.prototype?

@michaelficarra
Copy link
Member

We could do that, but I think the Array helper is a lot less motivated if we have a good Iterator helper. It's hard and complicated to write your own zip. It's not hard or complicated to write Iterator.zip(a, b).toArray(). And I imagine a good portion of invocations on arrays won't even need to realise an array as the output anyway. new Map(Iterator.zip(keys, values)) or for (let [k, v] of Iterator.zip(keys, values)) ... will work fine for array inputs. For the remainder of cases, calling .toArray() doesn't seem like much of a burden to me.

@ljharb
Copy link
Member Author

ljharb commented Aug 16, 2023

Most of my need for zip starts with two arrays and ends with one array, and it seems useful to me to avoid the iterator protocol entirely when it’s not needed.

@tophf
Copy link

tophf commented Aug 25, 2023

Having it on Array prototype would probably allow various optimizations in the JS engine, which might be impossible with the generic iterator version.

@michaelficarra
Copy link
Member

I wouldn't believe such claims unless they came directly from the implementors. The engines already keep taint bits on things like Symbol.iterator on built-ins and then short-circuit these kinds of operations to optimise when passed arrays or other built-ins. I don't see why they wouldn't do the same here.

@tophf
Copy link

tophf commented Aug 25, 2023

I don't see why they wouldn't do the same here.

I thought it's impossible to know beforehand that the developer wants to produce an array in the end so the engine would have to speculate, which is wasteful.

@michaelficarra
Copy link
Member

Closing this as we've taken the Iterator.from approach of accepting iterators/iterables, which includes arrays.

@ljharb
Copy link
Member Author

ljharb commented Dec 19, 2023

@michaelficarra i don't think that addresses the ergonomics issue nor precludes also adding the method on Arrays.

@michaelficarra
Copy link
Member

@ljharb You're right, this proposal does not preclude another proposal that would add a similar static method on Array. I don't think such a method should be included in this proposal, though.

@ljharb
Copy link
Member Author

ljharb commented Jan 6, 2024

Why not? This issue shouldn’t be closed until that original question is answered.

@zloirock
Copy link

zloirock commented Jan 6, 2024

I agree that it should be added to Array too. Extra .toArray() is too ugly for simple cases.

@michaelficarra michaelficarra reopened this Jan 6, 2024
@jridgewell
Copy link
Member

Mozilla has already said they'll reject anything adding to Array.prototype, so a method isn't happening. And Iterator.zip(array1, array2) already works. What's the end goal of this discussion?

@ljharb
Copy link
Member Author

ljharb commented Jan 6, 2024

@jridgewell that was not my interpretation of what they said; it was that new methods on Array.prototype would require a stronger motivation.

Array.zip would produce an array, which is the whole point - may programs don't want to work with the iterator protocol at all.

@bakkot
Copy link
Collaborator

bakkot commented Jan 6, 2024

Can you give an example of a time when you'd want an array as the result? In my experience, zip is almost exclusively used as a step in a chain: for (let [a, b] of zip(x, y)), or Object.fromEntries(zip(keys, values)), or whatever. Wanting an array seems like an unusual case which would need quite strong motivation, given that Iterator.zip(a, b).toArray() is not exactly hard to write.

@ljharb
Copy link
Member Author

ljharb commented Jan 6, 2024

I don't have a concrete example off hand - i'll certainly try to find one - but i definitely use "zip" at times as not a step in a chain (where "a chain" means "within the same function/scope") - and i want to pass around arrays, not iterators, basically 100% of the time. I only use iterators as part of an inline transformation.

@towerofnix
Copy link

I don't know how much there is to lose by adding Array.zip (or such) alongside Iterator.zip, except that perhaps it would be nicer to have syntax like Array.prototype.zip, which is obviously a no-go. I agree with @ljharb about the case for transferring between functions or contexts.

IMO Iterator.zip expresses something that is more complicated than Array.zip: "I would like to start an altogether new context of iterating over multiple things by zipping together these iterables." If you're creating that context (i.e. an iterator) out of an array, only to turn it right back into an array, then Iterator.zip is behavioral noise and Array.zip is much more to the point.

Of course, Iterator.zip is appropriate to use in for..of or alongside any others of the iterator helpers, where you might break early and not need all the results, or otherwise want to express a complex operation without immediately computing it. That's an obvious and widely applicable use for iterators in general, which Iterator.zip lends perfectly to, but it's not the only use for zipping stuff together.

@michaelficarra
Copy link
Member

If you're creating that context (i.e. an iterator) out of an array, only to turn it right back into an array, then Iterator.zip is behavioral noise and Array.zip is much more to the point.

This is a big "if". I agree with @bakkot that the zip result is almost always used as an intermediate result. If that's true, we should not make it as easy to use the version that realises an array. It's easy enough to do Iterator.zip(...).toArray() and better (in my opinion) that it calls out the eagerness. And I'm not concerned about the efficiency of iterating over an array as input, as engines can certainly optimise this case.

@tophf
Copy link

tophf commented Jan 15, 2024

we should not make it as easy to use the version that realises an array

Looks like a very heavy-handed way of helping the programmers. According to this logic there should be no Array.prototype.map, filter, and other established methods because they are often chained and produce an unnecessary array. And although I sometimes avoid using these methods for this exact reason, it doesn't mean I agree that this functionality shouldn't be present in the language as it's useful in case where performance/memory/GC is not a concern.

@towerofnix
Copy link

I don't know if that's what @michaelficarra is getting at. It sort of feels like the opposite, i.e. optimization isn't a concern here and we are talking much more about expression rather than performance (I interpret "engines can certainly optimise this case" as saying "Iterator.zip(...).toArray() can be trivially detected and flattened into a single operation that doesn't actually perform a second pass of iteration in toArray" — but I could be misinterpreting of course).

I don't understand what "better that it calls out the eagerness" means. Are you indicating that being forced to use toArray() is a way of explicitly saying, "OK, I really do need all the stuff from this zip operation as an array now, I know that's a bit weird so I'd better have some darn good justification"?

If that's the case, then I kind of agree with @tophf that it's heavy-handed. I don't think it's a good idea to consider iterator operations as the absolute replacement for array operations - I don't want to accuse you of that, mind! but if it's in line with your thinking, then it feels like the wrong angle to me. We have array methods and yes, those are a legacy inclusion because they've been around forever, and yes, there are a ton of cases where iterators are more appropriate and better.

But array methods still exist, and I feel as though introducing a behavior for iterators without also bringing along the direct analogue in arrays would be a missed opportunity. I feel the onus should be on the developer to learn and identify whether arrays or iterators are more appropriate for their use case, including the skill to effectively identify not just "if iterators are better than arrays", but in what specific instances their code would benefit from using each.

I don't think the spec should be arbitrarily lending preference to one choice or the other if there is not an extremely clear reason that zipping is fundamentally an operation that does not make sense for arrays.

@bakkot
Copy link
Collaborator

bakkot commented Jan 15, 2024

The criteria for adding something is not "there is no identifiable reason that this operation does not make sense here". It's "this operation makes sense and is a common enough need to warrant adding something to the language, rather than expecting programmers to use the next best option".

And I'm just not convinced that it is actually particularly common to need zip on arrays specifically, rather than on iterators. I went to check and every place I could find that I've used zip in the last couple of years has been one-shot. I'm not strongly opposed, and I'm open to being convinced, but it would help to have concrete examples of cases where you need this.

@tophf
Copy link

tophf commented Jan 15, 2024

There's historical parityintegration1 between Array and Iterator methods, so a much stronger reason is necessary to break it and not to introduce the new method on both.

[1]: iteration protocol is already seamlessly integrated in array syntax like [...arr1, ...arr2], we don't have to explicitly convert arrays to iterators in for-of, and there's even a Stage-3 proposal to add explicit array-like methods on iterators.

@towerofnix
Copy link

towerofnix commented Jan 15, 2024

(edit: Just to clarify, all my comments are made in the context of the Stage 3 Iterator Helpers proposal, which I was taking as common knowledge since this is also an iterator helper, just a new kind of operation rather than an equivalent to an existing array helper.)

I agree with @tophf's point above but I think it would help to have a lot more examples shared anyway. We should challenge and try to justify the historical parity going forward, rather than take it as a (functionally weaker) given, too!

Real-world use cases:

  • I don't benefit from using an iterator but do benefit from using an array, i.e. I want to check the length of all the results, or perform sorting on all the results, or select arbitrary indices from all the results. It's OK to atomically operate on fully resolved arrays in-between because I know I'm going to be using everything in the array at once. A zip would often be the first statement in a chain of such operations, eventually resolving into an array that I operate on with array-specific behavior (sorting, length, etc).
  • I am writing a function which must return an array and not an iterator because that's what the consumer is expecting — most likely this is a getter function for an object/class property. Zipping is the natural final step of whatever process is happening here, so Iterator.zip(whatever).toArray() becomes boilerplate and noise.
  • if an API I'm consuming or infrastructure I'm working within does have genuine reason to only accept an array, and I know this from the start, and don't otherwise benefit from using an intermediary iterator, then I may as well avoid it from the get-go and just work starting from an array here. I can't personally provide a real-world example of this so I hope someone else can, especially since @ljharb laid out clearly, "i want to pass around arrays, not iterators, basically 100% of the time." But yes, any such case of arrays (rather than iterators) having genuine use is automatically a good case for Array.zip.

Use-cases that I don't think are that strong:

  • Me consuming an old API that assumes it's being provided an array and doesn't run Array.from on its own input. It's on the API to take responsibility for this, i.e. the library / function should be updated, I shouldn't be dealing with tacking on a .toArray() or using specifically Array.zip. If I have to then this is "bad code". It's a real-world use, sure, but it's jank and expecting a .toArray() here is fine.
  • Similarly, me passing around arrays when there isn't a good justification for it. Working within big infrastructure that assumes arrays, without good reason, means the problem is in the infrastructure, not the code working within it. Using .toArray() until the infra adapts to apply Array.from where appropriate or else respect iterators is fine. (Updating infrastructure can be a tall order, but there has to be some push for it.)

I admit that in our own code the vast majority of the uses of our stitchArrays function (equivalent to Array.zipToObjects) is in an intermediary position, e.g. for-of, or in a finalizing position where the outside consumer has no justification to assume arrays rather than iterables (usually "here is a list of stuff I want embedded in this HTML context please"). The infrastructure needs updating to check for [Symbol.iterator] instead of Array.isArray (and so on), but that's not a complicated fix to bring about.

I'd love to have these use cases expanded upon. I strongly disagree with the premise that we need to justify retaining parity with concrete, real cases... (Wasn't much the point of iterator helpers to improve parity? Is there not an inherent value in making two API surfaces overlap with plainly analogous behavior in both? Don't we want to make room for programmer expression, rather than say, no, this is the way this works, iterators are arrays 2.0 and we won't help you use arrays anymore?)

...But I still think we should try to make that case. Losing sight of why parity is a good thing only hurts the case for parity overall, and keeps the conversation on new iterator features in general from having nearly as much perspective as is deserved.

@michaelficarra
Copy link
Member

Feedback from committee was to not include Array-specific handling in this proposal. Closing.

@michaelficarra michaelficarra closed this as not planned Won't fix, can't repro, duplicate, stale Mar 6, 2024
@ljharb
Copy link
Member Author

ljharb commented Mar 6, 2024

@michaelficarra um, that's not my recollection of the outcome? There's nothing in the notes that states that as a foregone conclusion, and I consider this something to resolve within stage 2, that will block stage 3.

@tophf

This comment was marked as resolved.

@michaelficarra
Copy link
Member

@ljharb Nobody spoke up for it when prompted, and one person explicitly spoke against it. But I'll re-open for now.

@tophf No, we're talking about having Array-specific accommodations. The proposal already works fine for Arrays because they are iterable.

@michaelficarra michaelficarra reopened this Mar 6, 2024
@ljharb
Copy link
Member Author

ljharb commented Mar 6, 2024

@devsnek can you elaborate on your comment? did you mean, you think it should be a separate proposal, or that it shouldn't happen at all?

@michaelficarra
Copy link
Member

@ljharb To be clear, I don't think this was like a "1 vs 0" thing. Since the default state was to not do anything, I took the complete lack of expressed support as a strong negative signal.

@ljharb
Copy link
Member Author

ljharb commented Mar 6, 2024

I was pretty clear that I'd be doing a followup if it wasn't part of this proposal, and that the cross-cutting concerns would mean that their fates were linked anyways, would would effectively mean that this proposal couldn't advance to stage 2.7 until that one hit stage 2. At that point I'm not sure why a separate proposal adds value.

@devsnek
Copy link
Member

devsnek commented Mar 6, 2024

@ljharb Ideally I'd prefer that we aren't duplicating iterator methods on arrays just for convenience. If there's an issue expressiveness or performance, I think that's worth discussing, but I'm not sure either point has been raised yet.

@ljharb
Copy link
Member Author

ljharb commented Mar 6, 2024

Gotcha - why not? Most all of the iteration helpers are for convenience. (it's certainly also about expressiveness and performance, and a number of other things, but i think "convenience" is pretty compelling on its own)

@towerofnix
Copy link

I agree with @ljharb's argument above and especially wtih @tophf in #1 (comment).

Parity is a big deal for iterator helpers. In my mind, these proposals bring the same functionality to iterators as you're used to working with in arrays. If iterators are not intended to canonically serve as the replacement for arrays, then we shouldn't give preference to adding a useful feature for iterators but not for arrays — namely because that goes against the parity that we're establishing through the rest of the iterator helpers.


Here are the helpers in Iterator Helpers:

Method Parity Notes
.map yes
.filter yes
.take no This is iterator-specific functionality, loosely analogous to .slice(0, n)
.drop no This is iterator-specific functionality, loosely analogous to .slice(n)
.flatMap yes
.reduce yes
.toArray N/A However, Iterator.from is analogous
.forEach yes
.some yes
.every yes
.find yes

Here are the other methods on Array.prototype that aren't paired in Iterator Helpers:

Method Index? From end? Mutating? Note
.at yes sometimes N/A
.concat no no? no
.copyWithin yes sometimes yes
.entries yes no N/A
.fill no yes sure
.findIndex yes no N/A
.findLast no yes N/A
.findLastIndex yes yes N/A
.flat no no no
.includes no no N/A
.indexOf yes no N/A
.join no no no
.keys yes no N/A
.lastIndexOf yes yes N/A
.pop no yes yes
.push no yes? yes
.reduceRight no yes no
.reverse no yes no
.shift no no yes somewhat analogous to .drop
.slice yes sometimes no somewhat analogous to .drop or .limit
.sort no no yes result wouldn't have meaning as an iterator
.splice yes sometimes yes
.toReversed no yes no
.toSorted no no no result wouldn't have meanign as an iterator
.toSpliced yes sometimes no
.unshift no no yes
.values no no no iterators are already iterators
.with yes sometimes no

I put this table together to substantiate or counter my argument — to actually assess the overall parity. And yeah, basically everything here fairly obviously doesn't work as an iterator method, so isn't part of Iterator Helpers. I can only really make cases for .flat(Infinity), .includes(), and .concat() maybe having analogies in iterators — but my point isn't that we should split hairs, only that as far as I can tell, parity and completeness matters in Iterator Helpers.

Going counter to that parity because we don't have effective circumstantial justification / precedent for Array.zip (etc) feels like a mistake to me — I don't think it needs "real-world use" precedent, and that parity is good enough in its own right. (Although I'd still argue there are real-world use cases, anyway!)

@devsnek
Copy link
Member

devsnek commented Mar 7, 2024

@towerofnix The intention of the iterator helpers proposal was not to "bring parity with array methods". It was to take a common interface shared among a great deal of the ecosystem and make it useful by default for js developers. Whether or not array has a certain method, while certainly not irrelevant to the discussion, should not be unto itself a reason to include or deny an iterator method. And this argument applies the other way too, just because iterators have a certain method doesn't mean that arrays (or any other collection type) must include it.

@ljharb I just don't think its particularly burdensome to jump between arrays and iterators. Maybe rust has desensitized me from typing array.iter().map(f).collect() a lot but I like this pattern because it gives you a consistent interface to work with across all the different collection types in the language. This is just like, my opinion, though. I'm not going to block joint array iteration if people want to spend time on it, but I will continue to hold my opinion that I think spending time on it is unnecessary.

@tophf
Copy link

tophf commented Mar 7, 2024

The danger of not maintaining the historical parity is that it will confuse developers as there's no logical substantiation for why some methods are missing in Array/Iterator. After developers needlessly suffer for a while a new proposal would appear to bring these new methods to arrays, similarly to the ongoing Iterator Helpers proposal.

@towerofnix
Copy link

TBF there isn't really "historical" parity, because Iterator Helpers and this proposal are both new additions to the language. I think parity is a good idea in general, and I feel that even if it wasn't part of the designed intent of Iterator Helpers there does appear to be a lot of parity where it makes sense. But those are also just my opinion.

If someone sees Iterator.zip exists and Array.zip does not, if they wonder anything, it's going to be along the lines of "huh, I think this has an obvious analogy in arrays, wonder why it's missing here even though most of the array functions with obvious analogies in iterators are there?" — not "huh, I thought we wanted parity from the start, very weird that it isn't pivotal now?"

I do think developers are going to have personal experiences that lead them to expect parity between obviously-analogous array and iterator functions, and IMO Iterator.zip / Array.zip are pretty obviously analogous. But I don't know if others feel that parity as important or if we're just a vocal minority LOL.

I'd like to say it should always matter as something that's fundamental to how people learn programming languages (and interact with them in general), but that's a somewhat high-and-mighty perspective if I haven't done research or at least got anecdotal experience beyond the way I've learned programming languages.

@ljharb
Copy link
Member Author

ljharb commented Mar 7, 2024

@devsnek i think it's totally fine if users want to only use the iterator interface. I definitely don't think that preference should have any bearing on the separate existence of an array interface, for those that prefer that.

In general, "it's not personally interesting to me" isn't a persuasive reason not to progress on any functionality.

@ljharb ljharb mentioned this issue Mar 22, 2024
9 tasks
@syg
Copy link

syg commented Apr 8, 2024

As a general principle, I also prefer to work with arrays where cache locality matters. But that argues that the use cases ought to be evaluated case-by-case. I am not sure zipping in particular is an operation I care about from a cache locality perspective. Should I?

@ljharb
Copy link
Member Author

ljharb commented Apr 8, 2024

I'm not sure what "cache locality" means here, and I don't think I have that use case - i definitely zip arrays together on a small number of projects, frequently.

@michaelficarra
Copy link
Member

Reminder to @ljharb and @syg to please discuss this and advise me on which direction to go.

@syg
Copy link

syg commented Apr 25, 2024

I'm happy with either direction and am leaning towards omitting given the ease of calling toArray(), just want to hear use cases.

By "cache locality" I mean if I care about the performance of accessing adjacent elements (both adjacent in their index and in time of access), I'll pay the up-front memory of the array. So @michaelficarra I was really asking you if I should care about zipping in that performance context. I still see mostly abstract discussion of use cases. @bakkot has a comment up-thread about zipping mostly being an intermediate step, which would make me think the per-item processing takes enough time that the performance of accessing adjacent, zipped elements isn't usually important.

@michaelficarra
Copy link
Member

@syg In that kind of performance context, you're probably going to try to avoid creating any kind of intermediate structure. Zip is useful when describing your program as a series of data transforms. In pure FP languages, those transforms get fused and the intermediate structures are never actually realised. But this is JavaScript, so if you want to achieve performance characteristics similar to what you'd get if you were manually jointly iterating (assuming you are doing further processing after your zip), you'd have to actually do just that.

which would make me think the per-item processing takes enough time that the performance of accessing adjacent, zipped elements isn't usually important.

This is probably usually true, which is another reason that Array zipping is not motivated by performance IMO. However, while you seem to want to use performance to justify Array zipping to yourself, I don't believe @ljharb was making a (solely) performance-based argument for it. From above,

it's certainly also about expressiveness and performance, and a number of other things, but i think "convenience" is pretty compelling on its own

If you find that unconvincing, we can omit it for now and pursue it as a follow-up. My goal here is to not put the Iterator-based zipping portion of this proposal at risk of not advancing.

@syg
Copy link

syg commented Apr 25, 2024

If you find that unconvincing, we can omit it for now and pursue it as a follow-up. My goal here is to not put the Iterator-based zipping portion of this proposal at risk of not advancing.

I don't find the convenience argument convincing by itself because I think toArray is convenient enough.

@ljharb
Copy link
Member Author

ljharb commented Apr 25, 2024

It's definitely not solely or even primarily performance-based; it's about the mental model.

Iterator.zip(a, b).toArray()

Array.zip(a, b)

Both are more or less identically convenient, but why should i have to reach for Iterator when i only want to work with arrays?

@towerofnix
Copy link

Yeah, and providing a standard language for that mental model is exactly why Array.zip is a good idea iMO.

Like, if you only work with arrays and are not interested in interacting with iterators, then there's no reason your own project can't export a utility for that in some convenient-to-access place:

// util.js
export function arrayZip(...args) {
  return Iterator.zip(...args).toArray()
}

But then it's up to each program to decide what name it wants to use for "array zip", and there aren't any good choices:

  • arrayZip is the "nicest" one analogous to Array.zip or Iterator.zip...
  • ...but surely some people will pick zipArray ("zip for arrays!"), and that's liable for confusion with zipToArrays, for example...
  • ...and some will say, well, gee, if I only care about zipping to arrays, why don't I just name it "zip"? And now, if they decide to integrate iterator-style code down the line, some of their code will use zip and other code will use Iterator.zip, and it's all a mess!

Of course, each program that decides to make up a shorthand name will likely use a different one, and some just won't and will keep using Iterator.zip(a, b).toArray(), and it'll be a no-good mishmash—of lots of programs representing the exact same, extremely simple and rather common logic with a wide variety of different forms/names.

It's impossible to get ahead of the curve on every possible "simple function", but because Array.zip is such a direct and obvious analogy for Iterator.zip, it would be reasonable to give it a standard (and predictable) name.

You can more or less address supporting your particular mental model using a shorthand utility function, but you certainly won't be using the same language as everyone else. Or you can hope to use the same language as people "should" use (longhand Iterator.zip().toArray()), but sacrifice the legibility of your code and the benefit of sticking to a clear mental model.

@towerofnix
Copy link

Hey, just caught up on the TC39 meeting minutes from April 8 and April 11—didn't see those til now, despite our more recent reply! (I guess these weren't prepared and online til a week ago, so no surprise there in the end. tc39/notes@5282761)

I'm not too worried about Array.zip not existing initially, i.e. to be added in a separate proposal, though the concerns over that being a waste of committee time make sense. Still in favor of Array.zip being a thing from the start because it aligns with differing mental models. I think the only comment I take an issue with is (April 11):

SYG: Like I think it behooves people to think about the difference. Like if you want to zip giant things you probably don’t want an array, right

I agree that it's of course a good idea for developers to be attentive about this. I just don't think it makes sense for the language to be, like, forcing a developer to make that choice, just because of a difference in API surface. That doesn't seem like a very useful precedent and I think it would be an un-fun time arguing against that when an Array.zip proposal comes around, i.e. "OK but wouldn't that negate one of the reasons we didn't include Iterator.zip? Like there's no way to add Array.zip without losing out of the teaching benefit of not having Array.zip". Like yes, of course this is true, but maybe it shouldn't be considered a meaningful teaching benefit in the first place.

Other than that I feel totally OK with this going either way for now. I hope if Array.zip is not included that a proposal is able to progress without too much trouble or "wasting" committee time, but it's also cool if more nuances are figured out in the space of a dedicated proposal. If we end up in a world where we only have Iterator.zip().toArray() then that's hardly the worst world to be coding in, too.

@syg
Copy link

syg commented May 29, 2024

@towerofnix I don't understand your reply to my comment. I wasn't talking about teachability, but the performance cost of zipping giant things. That is, zipping giant arrays means holding the entire zipped result in an array, which uses a lot of memory. If most of the time, the point of zipping giant lists of things is to process the pairs one at a time, you do not want an array.

@towerofnix
Copy link

towerofnix commented May 30, 2024

Yeah sorry, I can see that my point wasn't clear. What I meant is: That's a meaningful observation about performance, but it's something developers have to learn for themselves, so they can decide whether an iterator-based operation or an array-based operation is more suitable. Like it was discussed, for some kinds of data/operations zipping as an iterator is better, other times zipping into an array is. I could be misreading the minute here but that's what I took from your comment:

Like the performance characteristics are just different. I think we’ll be doing a disservice with performance if we have a catch all. We should have a version of zip on array because it’s more performance. That’s not true. It’s more performance in different cases, you’re making certain trade-offs and I want to be explicit that whatever proposal comes out of this, it’s called out, it’s not a general thing.

edit, also from @ljharb's reply to the above, emphasis mine:

JHD: Yea, I mean there are some use cases where you might want one over the other, of course, but that doesn’t change the conceptual operations you might want to perform.

Those trade-offs are things a dev should be aware of so they have to learn what they are, but they don't need to learn by being "forced" to consider the trade-off by the language, if that makes sense.

Like yeah, totally agreed with "If most of the time, the point of zipping giant lists of things is to process the pairs one at a time, you do not want an array." But zip isn't going to be used just for giant lists of things, it will also be used for comparatively very small lists of things, and the memory overhead of having all those things exist at once is not an issue. (Could be preferable compared to CPU overhead of initializing and processing an iterator, esp. if you're just toArray()ing it immediately, depending on how the JS engine is optimized.)

So 1) there may be a case for Array.zip being better in certain scenarios (in terms of perf), but my point is moreso 2) it's up to the JS dev to understand and learn how to differentiate, and the language doesn't need to move them along with that (by only having Iterator.zip). IMO if you have a basic understanding of iterators then it's pretty clear that zipping several super-long lists all at once may be a cause for memory concern, that's something people hopefully acknowledge when they use array methods instead of iterators (prior to a refactor or bc arrays are just more suitable for their context).

IMO the symmetry (for like operations, not in general) between arrays and iterators already does a good job of encouraging you to consider if iterators are better for you, i.e. if your process could be represented similarly with iterators, then JavaScript is going to make it easy for you (by giving map, filter, etc). Iterator.zip w/o Array.zip could cause you to just scratch your head and be annoyed and then .toArray(). Which TBF could cause you to rethink your decision (using arrays in the first place), but I think it's mostly a bad developer experience and they'd learn the trade-off better if they were presented with the choice, not forced / strongly pushed one way or the other.

@syg
Copy link

syg commented May 30, 2024

Thanks for the explanation. I see where you're coming from. The general principle here is "performance footgun", where browsers especially (I represent V8/Chrome) don't want the slow thing to be too easy to reach for. The thinking goes as you've laid out: that the lack of availability would be met with some surprise and the developer would give extra thought to why there isn't symmetry.

It is reasonable to disagree with the above principle. At V8 we've drawn the opposite conclusion because performance, AFAICT, is by and large deprioritized by most web developers. The MO is to ship and to ship fast. Performance matters for the tail of very mature products, certainly, but what we've observed is that performance engineering is mostly a luxury -- features, correctness, etc, usually take priority. If there's a choice in APIs, the more convenient one usually wins out. As a browser, we have a goal in making the web browsing experience, and therefore websites, performant. Taken together, that's why we prefer to not spec potential performance footguns, especially if it's a matter of convenience and not expressibility.

I disagree with the "forced/strongly pushed" framing. toArray() is not that inconvenient.

@tophf
Copy link

tophf commented May 30, 2024

The general principle here is "performance footgun", where browsers especially (I represent V8/Chrome) don't want the slow thing to be too easy to reach for.

Sounds like this approach may turn JS into Java with its super verbosity that's still not preventing the apps from being very slow. Anything can be a performance footgun in any language, JS included, even the best practices can be applied incorrectly.

@towerofnix
Copy link

I think ^ what @tophf says is true in a strict sense... but I also totally understand where the reasoning comes from. I wouldn't want to make the assumption that this sort of API surface (let's say as a decision by ECMAScript, affecting a basic interface in the language, not a more specialized web API) truly has an impact on the observed performance of apps or the broad experience of devs writing those apps wrt performance.

But, the reasoning is sound: reach for the easiest API, and Iterator.zip().toArray() is indeed the easiest; write that enough times and you may start asking yourself just what the deal is with this fancy little Iterator namespace. It's as simple as a teeny-tiny Tinker Bell voice saying "think about me! think about me!", which does add up eventually, though it's no major bother to swat away in the meanwhile!

I think it's very likely there is a general precedence for, like, making one operation easier than another, by making the latter operation composed out of two smaller parts (e.g. Iterator and toArray), because you want people to be thinking about the parts individually & considering, maybe reconsidering, why they are composing them like this. There's always an inherent benefit to that (as discussed!).

I think it'd be worth finding/sharing cases of that in general (for calls ECMAScript has made), but also to do with performance in specific. IMO performance being one of the weakest points of products made for the web (which feels like "basically all products", to us people who are always surrounded by the web, LOL)—the impact that ECMAScript decisions has on that should be assessed more closely, more rigorously. Best not to assume that your reasoning, despite being sound, actually has the impact you are going for, especially in a crucial area.

I don't say any of this to suggest anyone is making these assumptions, just sharing my angle! I don't want to come across with an aggressive tone, "examples or it didn't happen", but I do think identifying the impact ECMAScript decisions have practically had on performance would be pretty crucial to strengthening or weakening this argument, and a worthwhile effort.

I'm also OK with giving this more weight than I initially did, like, I do think aiding developers to write more performant code for the web - a context where no one writes performant code in part because it's so easy not to - is a valid concern. (The question is "how much 'in part', and how much can ECMAScript have an impact on this?")

I think this is a good angle to figure out in more detail and that, uh, it may require a significant amount of effort and discussion both outside and inside committee, that would be best suited on a proposal which is maybe pivotally affected by this angle. Not treated as something that blocks progress on Iterator.zip w/o Array.zip, because Array.zip can always be added later once it's decided if the performance-through-language-surface concern is best to drop or approach differently.

Oh right, and obviously you can't go back on adding Array.zip. If you don't add it now and you do watch how developers start interacting with Iterator.zip, then of course that's setting our own precedence on exactly this subject. It's a study worth paying attention to just as much as looking at examples in previous additions to the language.

@michaelficarra
Copy link
Member

The committee decided to move forward with this proposal without the Array variants at the June plenary. Closing.

@michaelficarra michaelficarra closed this as not planned Won't fix, can't repro, duplicate, stale Jun 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

9 participants