Less is more

[mikeal]

I'm going to try and write out some of my most recent thinking about NG. What I first thought of as an opportunity to adopt new patterns I now see as an opportunity to reduce the surface area of the platform and more effectively enable higher order platforms and ecosystems.

First, you start with nothing. import does not return a single standard library module.

Now, rather than try to build up what node has today, we try to build up a new standard library that enables the community to build up what node stdlib has today. And, you build it to function in the browser.

Let's call this next-stdlib.

Rather that try to write a promise based platform, or a callback platform, or some future pattern that might emerge, we create an ideal foundation for any such platform by not exposing a stdlib other than base primitives that can be built to the platform bindings (libuv) and the browser.

What about features that can't be supported in the browser?

We don't write them, at least not yet. We can work with standards groups to expand what is possible in the browser and write to that. Next Stdlib should be written to the latest standards and people can use polyfills if they want to support older browsers. Remember, not doing it in the stdlib doesn't mean that it won't be possible, it just means that it'll have to be done in the ecosystem and will probably not work in the client and the server.

It may also be worthwhile to break the libuv section in to libuv/libuv.js (thinnest possible JS binding to libuv). This would also give us the opportunity to do heavy performance optimizations at the libuv.js layer and tolerate slightly less ideal performance when faced with supporting an API on top of the browser's available APIs.

[benjamingr]

I'm sorry, I don't understand this at all.

I think any solution that would let users use the native language facilities for concurrency (like promises) as part of the platform would be interesting to look at.

By the way, observables are an "alternative" to streams and not callbacks.

[yoshuawuyts]

This sounds pretty cool, but also tough to wrap your head around if not familiar with what libuv exposes. Just to summarize (and check that I'm understanding your ideas correctly):

• change require to import
• expose libuv bindings through import regardless of which platform is used (?)
• expose callback / promise / observable "platforms" which are enabled explicitly
• adopt browser / standards features in addition to the libuv bindings
• develop and implement new standards together with the standards groups

Reading it left me with some questions:

• How will platforms be enabled? Will it go through package.json?
• Will all functions in a platform be required, or can it be cherry-picked? (e.g. importing a callback-based fs module vs importing a full callback based platform).
• Who will create platforms? Will this resposability fall under a working group?
• How will platforms be shipped? Would they be included in node or split off into userland completely?

[trevnorris]

@mikeal FWIW I've already been working on a new JS API to be the "thinnest possible" layer between JS and the native side. That's something I've been wanting to happen for a while.

[mikeal]

@trevnorris are you building it as a standalone module or inside of node? Is there a repo or a branch somewhere?

[trevnorris]

@mikeal Since I know node's internal interfaces I'm able to write most of it as a module. Since it does rely on internals it'll probably break with any minor version release, but since I'm only using it for a proof-of-concept I'm not too worried.

[mikeal]

change require to import

No, require must remain unchanged in order to support the existing ecosystem and avoid a large break in compatibility.

expose libuv bindings through import regardless of which platform is used (?)

We should probably build libuv.js as a seperate module with the intention of it landing in core someday.

expose callback / promise / observable "platforms" which are enabled explicitly

No, everything above the "new stdlib" line is third party. We don't implement that directly, instead we build the ideal platform for other developers to build these callback/promise/observable platforms on top of.

adopt browser / standards features in addition to the libuv bindings

Yes and no. "new stdlib" should be an API that can run on top of either libuv.js+VM-bindings or the browser but should only directly expose new standards when it is simple and makes sense.

develop and implement new standards together with the standards groups

Yes. The problem with how this plays out today is that the shim and transform layers from node -> browser are so complicated that it's actually hard to tell where something doesn't entirely work in the browser because we do so much to polyfill or noop it. If we had something a little closer to the metal on the node side that was also being written to work on top of the browser we will much more easily see the places in the browser that missing some functionality.

That said, we'll also have to write the new stdlib in a way that can account for the additional security constraints of the browser without failing to provide that functionality to the native platform. This is a hard balance but I think it's achievable.

How will platforms be enabled? Will it go through package.json?

Entirely from npm or another package manager. We don't worry about supporting it natively, we let each higher order platform worry about how to do that themselves. You can already see something like this today with ember where they maintain the global nature of their sub-platform (ember) but use npm as a publisher/installer/package manager for everything else.

Will all functions in a platform be required, or can it be cherry-picked? (e.g. importing a callback-based fs module vs importing a full callback based platform).

That's up to each platform.

Who will create platforms? Will this resposability fall under a working group?

Ideally they should be able to be created outside of the core project. If they end up needing a home we can and should bring them in to the foundation though.

How will platforms be shipped? Would they be included in node or split off into userland completely?

We should be shipping as little as possible with core. It's up to these packages how they would like to be built and distributed on top of core.

[spion]

Sounds exciting! 👍

If done right, it could also mean a huge performance boost. The reliance on callbacks means that everything must rely on closures to preserve context, even if the abstraction could potentially provide a different way (e.g. async functions). And closures are costly

Regarding "platform" type libraries, the situation should get much better once npm flattening ships. I predict that once we have flattening, having a popular platform type library (with liberal version restrictions) as a dependency will not be frowned upon as much as it is now.