Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Should design a new standard library? #19

Open
littledan opened this issue Dec 17, 2018 · 61 comments
Open

Should design a new standard library? #19

littledan opened this issue Dec 17, 2018 · 61 comments

Comments

@littledan
Copy link
Member

@littledan littledan commented Dec 17, 2018

These are some fundamental questions I've heard from folks when I discuss built-in modules with JavaScript developers. Have you heard these concerns, or any other concerns that aren't tracked in other issues? Do you agree with my suggested answers here? Is there more we should consider digging into here?

Q: Rather than make new libraries that are built-in, should we standardize existing libraries and put them in the platform? These meet real needs, have adoption in existing code (so speedups will come sooner) and are more ergonomic.

A: Let's figure this out on a case-by-case basis. Typical JavaScript libraries do things that we might not want to bake into the language, but we in the standards world have a lot to learn from them in terms of attractive ergonomics and practicality for direct use.

Q: Should the platform focus exclusively on low-level capabilities, rather than building a standard library? The ecosystem does best at higher-level things, being more agile and not having to worry about rollout across browsers.

A: This project has a longer timescale, so it is complementary to many ecosystem efforts. Built-in modules will be designed to be accurately polyfillable when possible, shrinking the time to become usable, and will have strong tests to ensure consistency across browsers. There's value in standardizing certain core components for reuse, for various motivations that are specific to each library being added.

Q: How will built-in modules avoid the pitfalls of previous standardization efforts, which result in poor ergonomics, not really solving developers' needs, etc.?

A: The standards processes have been improving, incorporating more developer feedback and prototyping, though we still have a way to go. Hopefully, being able to develop standard libraries in JavaScript will help get more people involved. Your ideas here are appreciated.

@littledan littledan changed the title Basic concerns with built-in modules in general, and "answers" Should design a new standard library? Dec 17, 2018
@JonForest
Copy link

@JonForest JonForest commented Dec 17, 2018

Personally I'm very keen on extending the standard library. I expect a large part of the node_modules bloat I think comes from little modules patching common gaps.
A common, approved, safe set of functions helps protect us from the leftpad fiasco of a couple of years ago.

@ljharb
Copy link
Member

@ljharb ljharb commented Dec 17, 2018

@JonForest No, it doesn't; the issue with left-pad had zero to do with a hole in the standard library (one that's now filled by padStart), and was entirely to do with an author's ability to unpublish at will (which is now no longer possible after a few days, and thus no longer a problem in the future)

@littledan
Copy link
Member Author

@littledan littledan commented Dec 17, 2018

Let's work together to solve the parts of the problems where they can be solved. We can add new standard library features with work in this repository, and give feedback on npm in https://npm.community/ .

@evanplaice
Copy link

@evanplaice evanplaice commented Dec 17, 2018

The best part about JS is that there is no standard lib.

Take jquery as an example; a decade ago the entire JS community probably would've agreed that JS and JQuery are practically synonomous, I even developed a relatively popular lib that became popular in large part because it masqueraded as a jquery extension.

Nowadays, with better module support in ES6. The ability to tree shake out unused code from the output. The upcoming functional/pipeline syntax extensions. Etc... If jquery were rewritten today it would be designed fundamentally different and much better. IIFE's and prototype chaining used to be ubiquitous whereas now they're fundamentally bad practises.

Take lodash as an example. The new pipeline syntax for functional-like async processing is cleaner, more intuitive than the current rendition and will likely spark a new generation of utility libraries like lodash that will be more ES6/Tree Shaking friendly.

At best, a standard lib will inevitably become baggage over time as newer and better solutions are developed/discovered.

@yogevyuval
Copy link

@yogevyuval yogevyuval commented Dec 17, 2018

@evanplaice While I agree with what you are saying, it seems that the state of JS standard library is a bit behind other full featured languages like Python, Java and Rust. Mostly because of really surprising and basic features that you expect a language so powerful to have, and end up using lodash instead or writing a utils.js file to your project.

I think a lot of people agree that JS is really great especially with ES6, but it can be even better with a bit more work to the std, while not making it a bloated baggage.

@OlivierJM
Copy link

@OlivierJM OlivierJM commented Dec 17, 2018

For the case of lodash and possibly date-fns.
As mentioned they might be improved sooner or later as more better solutions are developed like the pipe operator and others.
Why isn’t the concept adopted rather than the current existing code of these libraries.

it would really make it easy to have such as part of the core language

@martinheidegger
Copy link

@martinheidegger martinheidegger commented Dec 17, 2018

A: Let's figure this out on a case-by-case basis....

This answer is avoiding the conversation. Do you want to the community to open a "case" for every library that should make it's way upstream? Then this would answer better:

"We'd be happy to consider elevating existing libraries. Please add an issue in this repo so we can discuss whether that library is good to consider for standards."

If not, maybe something like this might suit you better:

"We are looking into the solutions brought up by the ecosystem but ultimately attempt to build with the community something new that fulfills our quality standards."

A: This project has a longer times...

This is not answering the question: "Should the platform focus exclusively on low-level capabilities?". From what I have read this may rather be:

"The standard library is focusing on low-level capabilities! The issue is that by adding more and more features to the JavaScript core we risk startup-performance issues of JavaScript engines. We want to provide a way to developers to choose which parts of our work they wish to use."

@littledan
Copy link
Member Author

@littledan littledan commented Dec 17, 2018

Those answers look great! Maybe you would be a better person than me to make the PR to put an FAQ in this repo.

@martinheidegger
Copy link

@martinheidegger martinheidegger commented Dec 18, 2018

I can help reviewing any PR that you open 😉

@calebdwilliams
Copy link

@calebdwilliams calebdwilliams commented Dec 18, 2018

I want to throw my hat in with @evanplaice. This seems like a slippery slope and ultimately a solution looking for a problem. With the new standards processes that are in place, I don't ultimately see what this proposal would solve well (and the things it would solve introduce myriad complexities).

While I will agree with the fact that we need utilities like map and len and time utilities, I don't think importing them from a standard library that is somehow separate from JS core really does anything for developers that the current process doesn't allow. map and len could be added to Reflect, a new Time object could be created or time utilities added to Intl.DateTimeFormat.

The one real benefit might be versioning, but any utility that needs to be versioned might not belong in a standard library any way.

I believe implementing this proposal would result in a fractured experience for developers.

@martinheidegger
Copy link

@martinheidegger martinheidegger commented Dec 18, 2018

@calebdwilliams I am trying to understand your comment and put it into question form. Is it correct that your question would be:

Q: What advantages does the standard library have over simply adding functionality through the new standards process?

@calebdwilliams
Copy link

@calebdwilliams calebdwilliams commented Dec 18, 2018

@martinheidegger More or less.

@serapath
Copy link

@serapath serapath commented Dec 19, 2018

PLEASE STOP IT!!!!

  1. Any developer who moves into TC39 or any other standards body is basically avoiding competition and wants an easy way to PUSH their ideas to EVERYONE in the entire world.
    This is unbelievably rude! If you have an awesome idea, make a module, market it to other devs like everyone else does and if they like it they will adopt it.

  2. Userland exists so that the core can stay lean. Different people have different needs.
    The core needs to solve what cant be solved in userland.
    The only reason to add to the core is to enable useful features that can't be achieved without.
    Already today that rule is violated way too often and TC39 is abused.

  3. Maybe you can Lobby somewhere else to add some kind of cross domain module cache to javascript environments, so that a downloaded "module" used in one context can be re-used in another and doesn't need to be downloaded again.

  4. If such a mechanism exists, certain feature - whether new Map or new Set or new Array or in browsers window[XXX] could be a blocking lookup to fetch immutable modules dynamically from a standards body like w3.org or otherwise from configured vendors like firefox/chrome/safari/etc... or alternative endpoints (maybe npm?) AND then cache them and make them available to other programs too (e.g. a different browser tab loading a different domain or different node process should be able to re-use it)

Would't that solve all the problems?

@Nimelrian
Copy link

@Nimelrian Nimelrian commented Dec 19, 2018

PLEASE STOP IT!!!!

  1. Any developer who moves into TC39 or any other standards body is basically avoiding competition and wants an easy way to PUSH their ideas to EVERYONE in the entire world.
    This is unbelievably rude! If you have an awesome idea, make a module, market it to other devs like everyone else does and if they like it they will adopt it.

The "make a module for every single idea" approach has led us to the horrible NPM ecosystem, which has and is causing issues like the left-pad fiasco, a bloated node_modules folder and incredibly deep dependency trees.

  1. Userland exists so that the core can stay lean. Different people have different needs.
    The core needs to solve what cant be solved in userland.
    The only reason to add to the core is to enable useful features that can't be achieved without.
    Already today that rule is violated way too often and TC39 is abused.

Userland exists to achieve certain use cases using the means provided by the language (including its standard library). The goal of this proposal isn't to implement specific solutions for specific problems, but common solutions to common problems.

The standard library is also separate from the language. Language evolution e.g. by adding syntactic sugar isn't a bad thing, but often enhances maintainability and usability of the language itself.

  1. Maybe you can Lobby somewhere else to add some kind of cross domain module cache to javascript environments, so that a downloaded "module" used in one context can be re-used in another and doesn't need to be downloaded again.

If you're using yarn, you can already make use of that. This is a topic which is far outside the scope of the language committee.

  1. If such a mechanism exists, certain feature - whether new Map or new Set or new Array or in browsers window[XXX] could be a blocking lookup to fetch immutable modules dynamically from a standards body like w3.org or otherwise from configured vendors like firefox/chrome/safari/etc... or alternative endpoints (maybe npm?) AND then cache them and make them available to other programs too (e.g. a different browser tab loading a different domain or different node process should be able to re-use it)

See my answer above.

Would't that solve all the problems?

No, it would create a multitude of new ones.

@ljharb
Copy link
Member

@ljharb ljharb commented Dec 19, 2018

@Nimelrian since everyone seems to misunderstand the left-pad incident, please see #19 (comment).

Note that npm has the same machine-level cache as yarn for down loaded archives.

@Nimelrian
Copy link

@Nimelrian Nimelrian commented Dec 19, 2018

@ljharb No, I understand it exactly the way you described it.

The chain of events which lead to the incident has a simple root: A (at that point in time) lacking standard library. The horrible (non-existent) security model of NPM is obviously a different issue, but it served mainly as an enabler, not as the root.

The small standard lib is exactly what led to the huge amount of packages and the huge dependency trees we see nowadays. Missing functionality is "fixed" by writing a package for it and people import it, because they don't want to write "things which should be standard" themselves.

Just compare dependency trees of Java or C++ projects, which have huge standard libs, with those of JS projects. Most of the dependency bloat in JS comes from packages which implement small, common solutions to common problems, which is exactly the thing a standard lib is meant to do.

@martinheidegger
Copy link

@martinheidegger martinheidegger commented Dec 19, 2018

@serapath Attempting to put your concerns into question form for a Q&A. I don't want to attempt giving answers!

Any developer who moves into TC39 or any other standards body is basically avoiding competition and wants an easy way to PUSH their ideas to EVERYONE in the e....

"A lot of effort has been put by individuals to promote new ideas. Many of those innovators get nothing out of their effort but the fame of their library. The TC39 would gain the power to render an existing effort mute and thereby ruining a life's work. What do you intend to do to prevent private agendas resulting into half-baked and/or derivative solutions that might do more harm to the ecosystem than good?"

Userland exists so th... way too often and TC39 is abused.

"The bigger a core library grows, the harder it becomes to maintain, secure and update. Why do you think it is a good idea to increase that scope?"

(I felt this was slightly redundant to the former Q, reduced to non-redundant question)

Maybe you can Lobby somewhere else to add some kind of cross domain module cache to javascript environments, so that a downloaded "module" used in one context can be re-used in another and doesn't need to be downloaded again.

"Why is it not possible to distributed a good standard library separately from the vm? What drives you from finding solutions that utilize a flexible distribution mode in favor of stiff solution?"

If such a mechanism e...

"A good flexible and secure loading mechanism could allow the greater eco-system to piggy-back on your efforts, wouldn't that be a worthy goal?"

(the last one was really hard to put into question form)

@mcollina
Copy link

@mcollina mcollina commented Dec 19, 2018

I think JS not having a standard library so far has been one of the greatest things to make the broad JS community so innovative. I think extreme care needs to be taken to add things to the standard library, mostly working to standardize/add the most common patterns that are available in the ecosystem.

From my purely-Node.js point of view, I think the goal of the standard should be to enable the evolution of the language/platform without polluting the global namespace. I would love to see a clear line between the JS standards (TC39) and the Web standards (WHATWG) and all other globals that are currently added by runtimes (e.g. Buffer in Node.js). Not having this distinction makes it hard to develop isomorphic applications.

@TheLarkInn
Copy link

@TheLarkInn TheLarkInn commented Dec 20, 2018

I agree heavily and on behalf of webpack we would be strongly against a standard library.

The modern module and package management systems we have today would ensure Stdlibs are always outdated and obsolete by their alternatives. Keeping the core small enables a rich ecosystem.

@Akamaozu
Copy link

@Akamaozu Akamaozu commented Dec 20, 2018

This seems like a heavy-weight solution.

Instead of a new stdlib which is almost guaranteed to fracture the ecosystem even more, why not have a list of recommended modules on npm for different usecases?

E.g.

Date/Time - moment
HTTP request - request
HTTP server - express

This list can be updated annually, and the old list permanently archived.

Think awards show, but for packages.

@Akamaozu
Copy link

@Akamaozu Akamaozu commented Dec 20, 2018

Are any of these run or publicly endorsed and promoted by TC39?

@Pauan
Copy link

@Pauan Pauan commented Dec 20, 2018

@Akamaozu No, and to be honest they shouldn't be: the TC39 committee has its hands full.

They are already struggling just to keep their own proposals up to date.

They should focus on improvements to JavaScript, not ecosystem curation.

Having it be a community-run wiki ensures that the quality stays high and that it remains up to date.

Having it be community-run means that it's very easy for ordinary people to contribute, which is how it should be.

If you simply meant that these lists should be more widely promoted and known within the JavaScript community, then yes I agree. However, I don't think that requires TC39 promotion. Word of mouth is plenty enough.

@Akamaozu
Copy link

@Akamaozu Akamaozu commented Dec 20, 2018

So basically let them keep up the busy work of creating a new stdlib, but ...

  • if somehow these lists were popular enough they wouldn't need to

  • they don't need to publicly endorse or promote one because word of mouth is enough

Let me ask this:

What is a stdlib if not a list of recommended util libs rolled into core?

@Pauan
Copy link

@Pauan Pauan commented Dec 20, 2018

So basically let them keep up the busy work of creating a new stdlib

I never mentioned anything about whether TC39 should create a stdlib or not, I merely replied to your suggestion about creating a list of curated npm modules.

What is a stdlib if not a list of recommended util libs rolled into core?

There are many differences, especially with regards to versioning and out-of-date libraries. But I'm not trying to get into an argument about that.

@Nimelrian
Copy link

@Nimelrian Nimelrian commented Dec 20, 2018

@mcollina

I would love to see a clear line between the JS standards (TC39) and the Web standards (WHATWG) and all other globals that are currently added by runtimes (e.g. Buffer in Node.js). Not having this distinction makes it hard to develop isomorphic applications.

The stdlib proposed here would be part of the JS standards you mention and would contain nothing related to the web API or other runtimes.

@TheLarkInn

I agree heavily and on behalf of webpack we would be strongly against a standard library.

The modern module and package management systems we have today would ensure Stdlibs are always outdated and obsolete by their alternatives. Keeping the core small enables a rich ecosystem.

The stdlib would/should only implement certain core functionalities (utility functions, containers, etc.) which can be tested easily. The purpose of the stdlib wouldn't be to replace entire libraries/frameworks, but to supply small solutions to small problems.

If a third-party solution offers better features or fixes a bug in the stdlib implementation, people can use that and the stdlib can be updated in time.


I think many people here misunderstand the goal and purpose of a standard library. It wouldn't replace entire frameworks. It would only supply well-tested, single implementations for common problems in our everyday use of the language.

@calebdwilliams
Copy link

@calebdwilliams calebdwilliams commented Dec 20, 2018

@Nimelrian What, then, is the benefit of adding these sorts of utilities to a "standard library" rather than adding them to the core?

From my vantage point, those benefits seem to be almost nothing especially when weighed against the cognitive cost of introducing a new paradigm into the JavaScript ecosystem. Yes, this feature might work for other languages, but in its current conception, it doesn't seem to fit with what JavaScript is and has been.

@Nimelrian
Copy link

@Nimelrian Nimelrian commented Dec 20, 2018

They will be built "into the core" as in "They'll be part of the language specification". That doesn't mean they'll be available on the global scope, but instead have to be imported explicitly.

To add:

And I'd be glad about this. Having to pull in (sometimes multiple versions) of utility libraries like lodash hurts page performance and puts people at the mercy of the maintainers of these libraries (e,g, lodash is gonna remove the omit function, which is widely used in FP or the React world, in version 5).

Another advantage would be that implementations can optimize the standard library functions using native code. That's impossible for userspace libraries.

@calebdwilliams
Copy link

@calebdwilliams calebdwilliams commented Dec 21, 2018

@Nimelrian, so too would adding those objects and functions to an object on the global scope (even if that object were named stdlib). The idea that you should need to import something built into the creates an entirely new paradigm in JavaScript that is unnecessary and will add a significant amount of cognitive overhead that will negatively impact the cohesiveness of the language.

@Pauan
Copy link

@Pauan Pauan commented Dec 21, 2018

@calebdwilliams ES6 modules are a standard part of JavaScript, they are implemented in browsers, currently being implemented in Node, and already widely used throughout the npm ecosystem (thanks to things like Webpack). They are not some bizarre esoteric new feature.

So yes, it is technically true that using ES6 modules for browser built-ins is new, but it's certainly not unusual, and Node has been using modules for built-ins for decades. And other languages have used modules for built-ins for decades. It's not a new idea.

And since user code uses ES6 modules, having the browser built-ins be ES6 modules actually increases consistency and cohesiveness, rather than having them be these weird global things which behave differently from user code.

@Lodin
Copy link

@Lodin Lodin commented Dec 21, 2018

@calebdwilliams, why do you think so? Any update itself is a cognitive overhead, but we have a ES2015 and it's great.

We all know why god objects are wrong. Especially mutable god objects. And JavaScript used to have it by design from the very start. However, if a lot of features built around this architectural flaw doesn't make it less flaw. It just means we cannot undo it in the next release. But we still can incrementally fix it, and introducing the standard library is a great step.

And I don't really think it is a big overhead. We import things. We do it all the time. So if we can import from non-existing file, what does it change? It still keeps the paradigm introduced by ES2015 modules.

And I'm very excited to have a possibility to have a standard library onboard. I really tired to choose and install simple things that all other languages includes from the box. And I really don't see any disadvantages of it.

Poor design? It can be handled by discussions and ideas. Moreover, JavaScript is on the stage this problem is solved so many times it's almost impossible to make it really poor. Lodash, ramda, underscore, tons of them. Just take and use. Other claims to design a just a matter of taste. How much map implementation did we have before Array.prototype.map? Where have they gone now? They just became unnecessary even if someone don't like that native map don't have some abilities lodash's map has.

These functions also will be faster than analogues. There will be no additional size forces us to write a lot of webpack plugins just to keep lodash from blowing up our packages. And that's even more great.

Also, it's quite polyfillable. Just install the package with the name of standard library, and you're done.

@calebdwilliams
Copy link

@calebdwilliams calebdwilliams commented Dec 21, 2018

@Pauan You do realize that the guy who basically runs Webpack is on the record earlier in this thread against this idea, right?

I will concede the Node argument is the strongest example of why this should be implemented, but if we examine the places where Node added core functionality as a module, it was in places that only Node could feasibly use or needed Node-specific implementations.

As for @Lodin, modules are fine, they work great for ES and I literally use them every day. The success of Array.prototype.map is a great example of how the current standards process is working without the need for a "standard library." I'll reiterate the point, if something needs or might need to be versioned, it doesn't belong in the spec. Period. If we want a set of utilities like Lodash or any of the others you mentioned, they can be built in with the current global object/prototype methods that are used.

My criticism is about language cohesiveness and adding this feature would make JavaScript less cohesive.

@Pauan
Copy link

@Pauan Pauan commented Dec 21, 2018

@calebdwilliams Of course, I am well aware. His objections were about the stdlib becoming outdated, which is a very real concern (e.g. see the Python stdlib, which has been outdated). That has nothing at all to do with ES6 modules vs globals (globals would also become outdated).

To be clear, I am generally against a JS stdlib. I think certain specific functionality should be added (such as more Array.prototype methods, some generator functions, maybe a few helper functions), but a rich stdlib should be avoided. The ecosystem does a better job with that.

But if functionality is added, it should be done through ES6 modules. That is my point.

@calebdwilliams
Copy link

@calebdwilliams calebdwilliams commented Dec 21, 2018

@Pauan I completely agree, but this is kind of an anchoring argument. If we can agree that things should be on a global, then what's the use of a stdlib object? Just put it on the prototype or on the Reflect object.

@Pauan
Copy link

@Pauan Pauan commented Dec 21, 2018

@calebdwilliams As shown in the README, the stdlib is just using ES6 modules, no globals, no stdlib object, no Reflect methods:

import { Date } from "std:Date";
import { len, map } from "std:builtins";

This is what I am in support of. The only exception is for adding methods to existing objects (such as Array.prototype).

@calebdwilliams
Copy link

@calebdwilliams calebdwilliams commented Dec 21, 2018

@Pauan and that's what I'm arguing is unnecessary. len and map make more sense to me as part of Reflect, date utils make more sense as static Date methods or part of Intl. I'm not at all against standardizing these utilities, quite the opposite in fact.

What I am opposed to is adding them to the junk drawer that a "standard library" will inevitably become. If it does get added, though, I agree using modules and the layered API proposal linked in the README is the right way to do it, but I don't believe that "If" should actually come about.

EDIT: At this point, I think my position is clear. I'll leave it up to the standards body and community to continue this discussion (and only jump in if something else is added). Either way, I do sincerely appreciate the thought and work that goes into making JavaScript such a successful language. Thanks all for the insightful conversations and happy holidays!

@ljharb
Copy link
Member

@ljharb ljharb commented Dec 21, 2018

(Please note that Reflect is solely for being a 1:1 correspondence with Proxy traps; so nothing can ever be added there unless it does)

@yacinehmito
Copy link

@yacinehmito yacinehmito commented Dec 21, 2018

I've been lurking in this issue; apologies for chiming in.

So far there has been talks of "date utilities", "map", "len", and in general quoting @Nimelrian :

The stdlib would/should only implement certain core functionalities (utility functions, containers, etc.) which can be tested easily. The purpose of the stdlib wouldn't be to replace entire libraries/frameworks, but to supply small solutions to small problems.

It would be great if people would define what they mean by standard library first if we want that discussion to ever move on. Wouldn't the opinions of everybody be heavily dependent on that?

I least mine would be. @littledan what do you have in mind when you say standard library?

@littledan
Copy link
Member Author

@littledan littledan commented Dec 21, 2018

Well, we already have a standard library in JavaScript, it's just very small. Things like RegExp and Date. IMO we should do whatever is most useful; there are lots of different goals we can consider. Small solutions to small problems sounds like a great way to start.

@noncombatant
Copy link

@noncombatant noncombatant commented Dec 22, 2018

I'd love to see more things like RegExp and Date, and I'd love to see additional useful methods added to existing object prototypes, and I'd love to see the new things added into their own namespace(s) instead of added to the global namespace. To me, all those things fall under the umbrella of "standard library".

The heavy dependencies that come with Node modules really hurt safety and performance, and looking around it seems to me that a lot of those modules are filling gaps that other languages have filled in their standard libraries.

Certainly, there will be things that are not yet ready to be elevated to standard library status; Node modules and other things can continue to fill that niche, just as (e.g.) Boost has for C++. When things mature, they can be elevated (as many things from the community have been in C++ and Go). That's how other language communities are doing it, and it has proven to work, and it saves a lot of developer (and user!) time.

@AtibaBryan
Copy link

@AtibaBryan AtibaBryan commented Jan 9, 2019

I wonder how hard it would be to get an official js standard library that's published on npm so you can have a lean core for low memory environments but have an easy way to add a library that's blessed/secured by a trustworthy org.

@mwetzel7r
Copy link

@mwetzel7r mwetzel7r commented Jan 10, 2019

I wonder how hard it would be to get an official js standard library that's published on npm so you can have a lean core for low memory environments but have an easy way to add a library that's blessed/secured by a trustworthy org.

for the scientific community, this group has been making good headway on a stdlib in javascript: https://github.com/stdlib-js/stdlib (I feel they deserve a shoutout here)

maybe instead of a single stdlib, an oligarchy of stdlibs for different communities' needs would be a useful approach (like what python tries to do)

@Akamaozu
Copy link

@Akamaozu Akamaozu commented Jan 15, 2019

Saw this tweet a few minutes ago.

I think the linked discussion might be of interest to us

https://twitter.com/milessabin/status/1084783899677523968

@markcellus
Copy link

@markcellus markcellus commented Jan 15, 2019

@Nimelrian

I think many people here misunderstand the goal and purpose of a standard library. It wouldn't replace entire frameworks. It would only supply well-tested, single implementations for common problems in our everyday use of the language.

Isn't this exactly what userland libraries aim to do? I personally believe that they already do a good job of that. I don't think people are talking frameworks here, we're talking userland libraries vs this new standard library. How are the potential libraries within this standard library any different from userland libraries that also exist to provide solutions "for common problems in our everyday use of the language"?

Another advantage would be that implementations can optimize the standard library functions using native code. That's impossible for userspace libraries.

I think this is an interesting idea and maybe this could be the focus of this group, but I certainly don't think that taking userland library features across multiple libraries and putting them into what we're calling a standard library is good for either the authoring teams or their consumers. Teams have spent years of hard work on their libraries and creating derivative works from them without their support is just rude. How do you plan to convince userland lib authors to support their work being exposed in this standard library vs their own repositories and under their own control?

I think @yacinehmito is right when he says

It would be great if people would define what they mean by standard library first if we want that discussion to ever move on. Wouldn't the opinions of everybody be heavily dependent on that?

This proposal needs to definitively address what it means by "standard library" or perhaps call out a few examples. Everyone has a different idea of what a potential "standard library" would include, so I think its important to make that clear--not in this comment thread--but in the actual proposal because it's still not clear to me, even after having read the README.

@Pauan
Copy link

@Pauan Pauan commented Jan 15, 2019

How are the potential libraries within this standard library any different from userland libraries that also exist to provide solutions "for common problems in our everyday use of the language"?

  • They're a standard spec, thus widely implemented everywhere (cross-platform), so they can be relied upon.

  • They can be implemented to be faster/with less memory usage.

  • They do not need to be bundled with the user's code, thus reducing file sizes (a big deal with JavaScript).

  • They significantly ease prototyping (since you don't need to fiddle with npm, installing dependencies, bundling, etc.)

  • They significantly ease tweaking, debugging, and experimentation, since the features are available right in the Node/browser's dev console.

  • They make writing small shell scripts (or quick one-off scripts) dramatically easier.

  • They significantly ease teaching for beginners (since beginners can just jump right in and start writing code, without dealing with all the complexity of libraries and package managers right away).

Probably some other benefits that I wasn't able to think of.

but I certainly don't think that taking userland library features across multiple libraries and putting them into what we're calling a standard library is good for either the authoring teams or their consumers.

I don't think anybody is suggesting taking existing libraries (like lodash) and just copying them into the stdlib. The stdlib would be new, not a copy of existing libraries.

@markcellus
Copy link

@markcellus markcellus commented Jan 15, 2019

No offense, but it sounds like you guys just want to create a new, authoritative platform of libraries that circumvent the strict processes of current standard's bodies by creating higher-level APIs based off of existing libraries in userland. You say these are "new" standard libraries, but you can't make a "new" library of "common features" that have already been built as npm packages, which are already used by much of the open-source community without it being "standard".

Everything you just mentioned is the same thing that existing userland libraries are doing great, but they are mostly claims to "make things easier" (I get it), but that's only a valuable solution if the current ecosystem is actually "hard", but I just don't think it is. So it just sounds like a solution to a problem that doesn't really exist--not at the magnitude at which you're emphasizing at least.

Overally, I just don't see how you're going to get people to disregard userland libraries for ones that you intend to expose here. All creating this "standard library" does is add another layer of red tape to ship software that is already available in the open-source community, which adds to the fragmentation. But this has been expressed already throughout this thread, so this will be my last comment unless I have something different to contribute.

@evanplaice
Copy link

@evanplaice evanplaice commented Jan 16, 2019

Q: Rather than make new libraries that are built-in, should we standardize existing libraries and put them in the platform? These meet real needs, have adoption in existing code (so speedups will come sooner) and are more ergonomic.

👎 Big no on this one. This is essentially taking the current popular libraries and 'blessing' them with special consideration. Performance speedups would be better served to all libraries if we had a common-reliable CDN that everybody actually used, better capabilities for caching, and possibly the ability to locally cache a post-parsed version of common libs.

I'd much rather see more emphasis on using public CDNs and providing LTS releases for popular libraries. This benefits everybody, not just the lucky few libraries that garner attention from TC39 (ie or whoever grants the blessings).

And let me be 'real' here. When you open up the potential for exclusivity, it's only a matter of time before the privilege becomes a 'pay to play' endeavor. It wasn't very long ago that the Node maintainers had to fork into IO.js to regain control of their project. It's no mystery that Joyent was actively blocking progress and selling influence to the highest bidders (ex MSFT, GOOG, etc) at the time.

Q: Should the platform focus exclusively on low-level capabilities, rather than building a standard library? The ecosystem does best at higher-level things, being more agile and not having to worry about rollout across browsers.

👍 As much as I'm not in favor of a single common lib. I'm all for adding support for a few more low-level capabilities. I'd consider things like RegExp and Date to be standard features of any language and JS could potentially use a few more. Emphasis on very limited scope tho.

Q: How will built-in modules avoid the pitfalls of previous standardization efforts, which result in poor ergonomics, not really solving developers' needs, etc.?

It won't. For a language that is constantly adapting/changing to meet new demands, there's no way to accurately predict what will be required in the future (ex ES2030).

Bit rot is a very real issue. Standard library API's -- due to their ubiquity -- are essentially permanently frozen. Best case scenario, old versions are phased out on a predicable schedule to prevent breaking existing code. The sad fact is -- the more ubiquitous the API -- the harder it is to deprecate. Trying to force the ecosystem move on from obsolete code incurs a real cost (ie time/money).

Competition is Good

Falling in line with my previous comment, libraries like lodash are absolutely valuable to the community as a whole; but it's important to not forget that lodash is probably the 3rd-4th-5th successful iteration of a 'collection of functional operators' lib. It's great in large part as a result of competition. There will be more to come in the future.

From experience; at one point, one of my own libs achieved relative popularity for a while before it was eclipsed by a better alternative. What's good now will (ideally) be trash in 5 years. Not only is that a good thing for the overall quality of the lib ecosystem. It's a good lesson in humility for OSS devs. Even if you're a very talented dev; a rival project that provides quality documentation, is better marketed, is easier to use, is more inclusive to contributors, etc should win out in the long run.

If there will be a standard library, make it like any other JS library and see if it can reliably compete with the rest of the ecosystem first.

@littledan
Copy link
Member Author

@littledan littledan commented Jan 16, 2019

If there will be a standard library, make it like any other JS library and see if it can reliably compete with the rest of the ecosystem first.

I'm totally in favor of this last point. I don't think we have to wait until it's solidly winning 100% in the ecosystem, but some moderate amount of catching on would be good to see, or an understanding of why that didn't happen in a particular case or why something specific is more urgent.

@Qix-
Copy link

@Qix- Qix- commented Jan 30, 2019

This seems like a slippery slope and ultimately a solution looking for a problem.

I couldn't agree more.

I don't see any benefit of this proposal other than shifting bloat around, not to mention potential performance hits due to JS<->C++ language boundaries.

If there will be a standard library, make it like any other JS library and see if it can reliably compete with the rest of the ecosystem first.

This.

I'm pretty shocked this hasn't been posted yet, but obligatory XKCD.


EDIT:

A quick note about left-pad.

I'm not going to sugarcoat it: it was a failure of NPM as well as all of the companies that treat NPM as a single point of failure. Our community champions fault tolerance but tends to show very little of it when it comes to our development process.

When I worked at Uber, we were immune to NPM outages or silly things like packages being unpublished since we used a CouchDB mirror, which is painfully easy to set up and maintain I might add.

When I develop native code, I usually mirror repositories to my own account for similar purposes if I'm using submodules, or I'll vendor the code directly into my codebase. This has little to no overhead aside from maybe a few more seconds during a clone, which happens once every month at most.

Further, the 'problem' with unpublishes has been patched and is thus a moot point. I don't see how it has any bearing here.


Another note:

Changing the language to solve the shortcomings of the package manager is a very quick way to ruin the language.

Pip, easy_install, anaconda, virtualbox, etc. - they all provide something the other doesn't, and working with all of them in a project is a headache, sure.

But that doesn't mean Python should be changed.

I think the only viable argument here is that there are many small modules that exist to patch small pieces of functionality in the standard types. However, saying we need a full-blown standard library to fix the problems of node_modules, unpublishes, et al is a ridiculous jump to conclusions, in my opinion.


In response to the points in #19 (comment):

They're a standard spec, thus widely implemented everywhere (cross-platform), so they can be relied upon.

That's literally the point of Javascript. There's nothing new here.

They can be implemented to be faster/with less memory usage.

[citation needed]

This is speculation at best.

They do not need to be bundled with the user's code, thus reducing file sizes (a big deal with JavaScript).

While not directly in the node_modules, they're still now part of the runtime, which is just shifting bloat elsewhere.

The net effect of this is that the code is hidden elsewhere, and instead of having a single implementation that you have inspected and know works with your project, you now have multiple platforms with potentially differing implementations of the functionality.

This is a net-negative if you ask me.

They significantly ease prototyping (since you don't need to fiddle with npm, installing dependencies, bundling, etc.)

If you're "fiddling" with npm, you're probably using it wrong or are setting up your machine for the first time.

If you're prototyping, you're probably not bundling.

Installing dependencies is a fact of life, for any language.

They significantly ease tweaking, debugging, and experimentation, since the features are available right in the Node/browser's dev console.

This doesn't make sense. Aren't dependencies also in those places? You're arguing that we should add a massive development overhead to JS engine vendors for the sake of saving a few seconds to avoid a call to require().

They make writing small shell scripts (or quick one-off scripts) dramatically easier.

How?

They significantly ease teaching for beginners (since beginners can just jump right in and start writing code, without dealing with all the complexity of libraries and package managers right away).

Ease of use for the general public should be the focus of any language, not focusing on how beginners approach a topic.

Beginners are going to have to understand dependencies, importing, etc. at some point in their journey, what good is it to shield them from it? Also, how does this have anything to do with the formal specification of a major programming language?

@PJB3005
Copy link

@PJB3005 PJB3005 commented Jan 30, 2019

not to mention potential performance hits due to JS<->C++ language boundaries.

Why would the standard library HAVE to be implemented in C++ and by extension incur overhead? Surely browser authors are free to implement it in the most optimized way possible. If it were slower for the browser to implement in C++ then they could just... implement it in JS, and it'd be transparent to you anyways.

Changing the language to solve the shortcomings of the package manager is a very quick way to ruin the language.

This problem isn't due to a shortcoming of the package manager though, it's due to a shortcoming of the language.

saying we need a full-blown standard library to fix the problems of node_modules, unpublishes, et al is a ridiculous jump to conclusions, in my opinion.

While yes those are some of the problems it aims to solve, it aims to solve a lot more too.

While not literally in the node_modules, they're still now part of the runtime, which is just shifting bloat elsewhere.

Except now the bloat doesn't have to be sent over the wire constantly and can be much more efficiently loaded by the JS runtime. I don't know what kind of amazing tricks browsers employ nowadays, but I can't imagine it being slower than polyfills in pure JS.

The net effect of this is that the code is hidden elsewhere, and instead of having a single implementation that you have inspected and know works with your project, you now have multiple platforms with potentially differing implementations of the functionality.

This is true, but I think this problem can be mitigated significantly. For example a standard test suite could be made that browsers have to comply with. Also this is already the exact problem the web faces, and I don't think "we shouldn't add features to browsers because it might have buggy implementations" is a common argument.

Installing dependencies is a fact of life, for any language.

Perhaps, but in no way to the extent with JS. In something like .NET you can get very far with just the standard library. Maybe a large framework depending on what you're building.

This doesn't make sense. Aren't dependencies also in those places?

I think what was meant is the case where you want to use some quick code in your browser's dev console to try something out, but to write that you need a dependency that you didn't bundle. Which the standard library would of course avoid.

And finally for the relevant XKCD: The difference here is that that doesn't apply when the new standard will be the future universal solution. If people used this argument against USB then maybe we'd still have charger spaghetti.

@Qix-
Copy link

@Qix- Qix- commented Jan 31, 2019

This problem isn't due to a shortcoming of the package manager though, it's due to a shortcoming of the language.

Half the arguments here refer to npm problems.

While yes those are some of the problems it aims to solve, it aims to solve a lot more too.

Care to elaborate?

Except now the bloat doesn't have to be sent over the wire constantly and can be much more efficiently loaded by the JS runtime. I don't know what kind of amazing tricks browsers employ nowadays, but I can't imagine it being slower than polyfills in pure JS.

Over the wire, perhaps. That's the only valid point, as you're contradicting your prior statement - if it's included as a javascript library, it means that each isolate is going to have to load in the standard libs anyway.

For example a standard test suite could be made that browsers have to comply with

Ah yes, I love waiting for Microsoft to implement features incorrectly 2 years after they're mandated.

What will ultimately happen is that this standard library will be decree and we'll spend many extra hours writing polyfills for it because browsers/environments/runtimes don't get them all, or don't get them all correct.

we shouldn't add features to browsers because it might have buggy implementations

"Might" is too loose a term. There will be buggy environments, overhead, and periods of waiting, especially since I would assume this "standard library" would be organic.

This is no different than requesting things be added to the built-in prototypes.

There are decades of evidence this would not be a smooth rollout process.

In something like .NET you can get very far with just the standard library.

Which is why projects like Mono have a really hard time supporting all of the features. Same with OpenJDK at first. It took decades for them to get it right.

If people used this argument against USB then maybe we'd still have charger spaghetti.

Software and hardware standardization problems are quite different, firstly.

Secondly, I think you're quite wrong here. Hardware standards are competing. They allow developers to make appropriate tradoffs for various manufacturing benefits. Software has a much different landscape when it comes to standards, as we tend to favor compatibility since our "manufacturing" overhead is virtually zero outside the realm of infrastructure and labor, which aren't being address by this proposal whatsoever.

That's an apples-to-oranges comparison.

@Nimelrian
Copy link

@Nimelrian Nimelrian commented Jan 31, 2019

Half the arguments here refer to npm problems.

npm problems which stem from a lacking standard library, which motivated people to write single-digit line packages. This isn't a "ridiculous jump to conclusions", it's simply backtracing from the current mess that is the JS package ecosystem.

I really don't see the huge problem people have with adding a mechanism (Note that this proposal is about the mechanism, not about what would be added to the libraries, which can then be loaded by said mechanism) to import standardized functions / classes / whatever.

@evanplaice
Copy link

@evanplaice evanplaice commented Jan 31, 2019

npm problems which stem from a lacking standard library, which motivated people to write single-digit line packages

Strawman. The need to keep libs small (ie pre tree shaking + ESM) combined with laziness was the cause of a single line lib breaking production code.

We're apprpaching a new era of JS lib development where designing larger aggregate libs with mostly dead code will no longer have a negative affect on performance.

Once native ESM is made available. It will be perfectly feasible to create a standard lib available via NPM.

Emphasis on - NOT IN CORE.

That way:

  • universal compatibility is guaranteed
  • upgradeability is simple
  • contributing is easy
  • platform/spec bloat is minimized
  • breaking change can be made on major versions
  • the progress/versioning doesn't have to be tied to Nodes release cycle
  • lib backward compatibility can be maintained independently of platform compatibility

If the standard lib becomes so damn essential to the JS ecosystem that it would be a detriment to the ecosystem to NOT include it - then, and only then - should it be included in CORE.

@kaizhu256
Copy link

@kaizhu256 kaizhu256 commented Jan 31, 2019

tree-shaking is a marketing-gimmick / joke. it has and will never work with a dynamic-language like javascript. I've manually tree-shaked various libraries using code-coverage analysis (swagger, nedb, etc...), and concluded a non-human could not possibly figure out the many circular and dynamic code-references common integration-level javascript.

@Qix-
Copy link

@Qix- Qix- commented Jan 31, 2019

tree-shaking is a marketing-gimmick / joke.

Speaking academically, yes and no. "Tree shaking" is a marketing term for dead code removal. There are situations where you can assert that code is dead in Javascript with guarantees about side effects. But those cases are rare.

Unfortunately, Javascript has way too many ways to introduce subtle or downright hidden side effects that can affect dead code removal provability.

I do agree, however, with the sentiment that tree shaking doesn't really have a spot in this conversation - I see it as beside the point, much as I think anything related to npm is also beside the point.

Also, I don't see how "bloated node_modules omg" is a problem solvable by the language. Native applications have the same problem when you distribute source. The size comparison comes down to bundling - in native applications "bundling" is compilation. You compile to an executable, which in many cases is leagues smaller than your sources.

With Javascript, bundling is comparable to the size of the source code, sans readmes/licenses and not taking into account compression. This is because javascript doesn't have a binary format (shout out to https://github.com/tc39/proposal-binary-ast).

This is also why WebAssembly was attractive to many people.

@evanplaice
Copy link

@evanplaice evanplaice commented Jan 31, 2019

Speaking practically, not academically. Forget 'tree shaking' there is a greater issue here.

Specifying that the standard library is built into the platform implies that the spec will need to be universally applied to most/every runtime that supports JS. Including node, browsers, embedded, and any devices that runs JS. Also note, Javascript's application/usage will likely increase to cover even more platforms over time.

This compounds the maintenance overhead on a language ecosystem that is already struggling to keep up with existing/emerging standards.

Also, I don't see how "bloated node_modules omg" is a problem solvable by the language.

I don't see how shoving what is essentially a global library into the runtime solves the issue either.

This top-down, 'maintainers know what the users need' approach has been done already (ref .NET/JRE/C++). From existing examples; standard libs tend toward massive, bloated, monolithic catch-all collections of utilities that become a 'necessary evil' of working in the language.

The advantage of doing it in JS using the existing tooling/ecosystem, is that it will be easy to exclude and/or dispose of the std lib if the experiment turns out to be a failure.

I'm not saying it should never be included in the runtime. Only that it's sensible to prove that a std library is necessary to begin with, prior to specifying it as a long-term maintenance commitment for all platform creators/maintainers.

With Javascript, bundling is comparable to the size of the source code, sans readmes/licenses and not taking into account compression. This is because javascript doesn't have a binary format (shout out to https://github.com/tc39/proposal-binary-ast).

Yeah, this would be really awesome. If a warmed cache containing the (already downloaded->parsed) AST representation of libs existed, it might actually be feasible to go back to the 'old way' of providing common library LTS releases via CDN.

@Qix-
Copy link

@Qix- Qix- commented Jan 31, 2019

@evanplaice I think you misunderstand me.

This compounds the maintenance overhead on a language ecosystem that is already struggling to keep up with existing/emerging standards.

I agree.

I don't see how shoving what is essentially a global library into the runtime solves the issue either.

I agree, it doesn't. Neither the package management nor the "more is better" arguments here make any sense to me.

The advantage of doing it in JS using the existing tooling/ecosystem, is that it will be easy to exclude and/or dispose of the std lib if the experiment turns out to be a failure.

I agree.


This proposal is a solution to an X-Y problem. I don't see it any other way.

@evanplaice
Copy link

@evanplaice evanplaice commented Jan 31, 2019

@Qix- Right, my bad. I interp'd it as you were in favor of the 'JS libs suck, put it in the runtime' approach. I'll quit with the broken record then, see what other peeps have to say.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet