Breaking the CommonJS standardization impasse

[brettz9]

What is the reason for the indifference to CommonJS? I understand you are no longer looking to adhere to it.* Are all contributors abandoning it or just you?

If you were not happy with the quality of specs or suitability to your environment, would it not be possible, as with the WhatWG group of browsers for the W3C process over HTML5, to use your weight as the apparent wide lead fore-runner and stake-holder among SSJS implementations to influence the CommonJS process (or something similar) in a way that accommodates your needs as well as offers some room for collaboration? For example, maybe you could express openness at least to a veto power to avoid dictates coming from specs you did not find to be well thought out?

I would think everyone would benefit as a result, since CommonJS too should not acquire so much adoption if the fore-runner is resistant. (I'm not speaking with any knowledge of the history or processes at CommonJS, just wishing for a break to the standardization impasse.)

Developers such as myself really like the ability of using JavaScript on the server (and there ARE plenty of cases of code reuse between client and server), as well as having cross-server-side portability, so I hope that you do not become the IE of the SSJS world by stifling compatibility through dominance (not as the IE of the stagnant years, mind you, but as the IE of the IE-Netscape browser war days when it was innovating and ultimately successful, but wasn't harmonizing with other players and thereby set the stage for years of agony for developers).

* http://www.reddit.com/r/node/comments/h1m2o/i_am_ryan_dahl_creator_of_nodejs_ama/

[bmeck]

While I find CommonJS interesting, it should be noted at least as long as I have been around, node is focused on creating a simple and efficient toolkit for making network applications. Early on, investigations into various things such as async module definitions node put some weight into talking to various individuals. Historic disagreements led to some apathy on node's side as people such as RingoJS attempted to be purely CommonJS; as RingoJS evolved along node there grew a popularity for node's module system, and the simplicity compared to the unified CommonJS system, however, with the module system you can easily recreate all of CommonJS given enough effort. See @domenic work on Promises for a stellar example of doing so.

As for the IE comment, I doubt node will have significant sway in the CommonJS field, but we have seen some sway in the WhatWG and W3C fields. CommonJS is largely community run, and the need for node to get involved with a laurel branch will be hard considering the difference in styles between existing APIs

[bnoordhuis]

What is the reason for the indifference to CommonJS?

node.js has evolved an API that is pretty much stable by now and we're not changing that this late in the game.

That's not necessary either. It's possible to build CommonJS abstractions on top of the existing API and people are doing that.

Are all contributors abandoning it or just you?

The word 'contributors' suggests that we've actively participated in the past. AFAIK, that has never been the case.

[brettz9]

Thank you for your replies!

[isaacs]

I'm prepared to make a much stronger statement on this point than @bnoordhuis. I actually was involved in many of the CommonJS discussions, even as far back as when it was called ServerJS. In fact, that's how I came to learn about Node in the first place.

In those days, it seemed like there were a bunch of "contenders" in the Server-Side JavaScript space, and it seemed like a good idea to have some consistency in our APIs so that programs could be made portable in different JavaScript environments. It even seemed like a fertile environment in which to experiment with proposals that could eventually make their way to es-discuss/TC-39, and become part of the JS language.

In practice, this turned out to be naive. The community was quickly overrun with hand-raisers, and people who were happier arguing about your program rather than writing their own. The naiveté was forgivable without the benefit of hindsight, of course, and none of this is intended to impugn the character or motivations of any of the human beings involved.

One approach that several SSJS platforms took was to emulate other systems, or build in JavaScript as a scripting layer on top of other systems. For example, Ringo and Narwhal are built on top of the JVM, and this informed many of their choices. Node was built as a JavaScript layer on top of libev/libeio. Reasonably Smart was built as a JavaScript wrapper around a perl core. v8cgi provided a v8 engine as an Apache2 plugin. The list goes on and on.

"Standardizing" before you have implementation is naive, in the most literal sense of showing a lack of experience, wisdom, or judgement. There is no reliable way to know the problems with an API before using it in a variety of different situations, and standardizing increases the commitment to this untested API, when what you ought to be doing is decreasing the commitment (and thus, the cost of fixing it later).

Furthermore, the desire to create a "standards body"-like thing, without carefully taking into account many of the costs and concerns facing actual standards bodies, was also naive. I was not involved in the decision to create ServerJS/CommonJS; I just jumped on the bandwagon because it seemed like the thing to do at the time. But over the course of a year or so, it devolved into a rather unsavory mess.

Meanwhile, Node was delivering real value to real people writing real programs. CommonJS never really got on board with asynchronous IO paradigms, cross-platform compatibility, or streaming APIs. All of these things turned out to be critical for Node. (And, we got every single one of them wrong, several times, before eventually figuring out what worked, and what didn't.) Performance was not a priority for CommonJS; it was the main priority for Node. require.async (and its descendant, AMD) turned out to be a red herring; increased complexity significantly, no one could agree on how it should work, and in the end, the only thing it did was make programs more complicated and slower to start. (And client-side modules in the real world require a build step anyway, and aren't our problem.)

The final straw for me was the absurd drama around the Packages specification. I wrote npm because I'd seen module packaging done terribly, and done brilliantly, and I wanted to make sure that Node didn't end up with something like Pear. Even CPAN, wonderful and impressive as it is, had some problems that I thought we could learn from. My real inspiration, though, was yinst, which is incredible, but also had some shortcomings that I wanted to address.

I engaged in good faith to try to make my points clear. In the end, it became obvious that the best thing for me, and for node, and for npm, was just to go off the reservation and focus on the things that I knew mattered. Ryan basically always gave zero fucks about CommonJS anyway. One evening at Joyent, when I mentioned being a bit frustrated some ludicrous request for a feature that I knew to be a terrible idea, he said to me, "Forget CommonJS. It's dead. We are server side JavaScript." That really stuck with me. I realized that I was spending mental energy in a way that was benefiting no one, and I decided then and there to just accept that Node is the only SSJS that matters. The eagle does not make war with flies.

A few good things came out of CommonJS. The module system we have now is basically indistinguishable from the original "securable modules" proposal that Kris Kowal originally came up with. (Of course, we went pretty far off the reservation in v0.4, with the whole node_modules folder thing, and loading packages via their "main" field. Maybe we should remove those features, I'm sure that Node users would appreciate us being more spec-compliant!) Some ideas about the package stuff came out of those discussions and made their way into npm, but not many.

All the proposals about specific APIs and modules, of course, we dismissed out of hand, and never had any intention of implementing. CommonJS ended up becoming effectively a body of documentation of Ringo/Helma, Narwhal, and SproutCore. And that's great. If those platforms want to have a similar API between them, then they can totally do that.

You bring up a concern about Node going the way of MSIE. We've won the SSJS war, such as it was. In fact, the only reason anyone even cares about other SSJS platforms is because Node's success has validated the approach in principle. We're competing with Ruby and Python and other server-side languages. Node IS server-side JavaScript.

On the face of it, it kind of looks like it might be something like MSIE and the browser war. However that misses many very important distinctions.

1. Technically savvy developers use server-side platforms. Not end users who don't know the difference between "msie" and "the internet", or the address bar and google.
2. Node is open source. MSIE is not. If anyone really really wants a feature, they can fork and add it. If it turns out that it really is as essential as they think, then they'll gain adoption. We won't stop them (and, in fact, we really can't).
3. Node is a minimal platform for a vibrant community of code reuse. Anything it lacks can be built on top of it. MSIE, not so much.
4. Web browsers are a many-many-many relationship. Many browsers, viewing many websites, for many users. However, in the case of a server-side platform, the developer can choose the specific binary and version, and they're the only one using it. There are issues of portability of modules, and portability across different OS's, but the landscape is MUCH simpler.

Not all success is a threat. Not all specifications are valuable.

[brettz9]

Nice to have these thoughts and history be spelled out a bit, thank you.

In the case of the IE discussion, however, my point in bringing up IE (the immediate post-browser war years), was its effect on web developers having to write for different browser environments (like the programmers writing for Node), not to speak to the quality of the product for its users (like the users of Node applications).

Also for your point 1, there is a wide spectrum of "savvy". To teach kids to program, for example, is made infinitely easier by a common syntax. Moreover, much of what is valuable on the web has been cobbled together by "script monkeys", so even if only they have trouble and find it inconvenient to have different syntaxes or make builds (and I don't think this is the case), I think harmonization and simplification is worthwhile.

I personally sympathize with the aims of AMD and find it very helpful having a single modular syntax in client-side JavaScript too (I've started a plugin moreover to allow shiv/shim/polyfill usage to be similarly modular, and have contemplated the same in my AsYouWish add-on to allow ActiveX-like privileged browser requests in Firefox and potentially other browsers via an AMD-compliant plugin). On the other hand, I'll admit I have seen first hand some confusion by developers on asynchronous requiring, but avoiding it is I think a high cost to sacrifice the potential for portability between all environments. But then again, I guess non-built client-side apps would presumably not be so performance-critical as to worry that much about performance anyways, and could be made to take advantage of shared modularity with Node packages via a synchronous API, synchronously loaded when the client-side app was not pre-built). My main point is that a lot of good can come from seamless plug-and-play, though maybe things should be converging on the synchronous API including in the browser rather than the other way around.

By the way, it is not only core JavaScript which is a candidate for modular reuse between server and client--the likes of AsYouWish (or browser add-on SDKs) might also harmonize to provide privileged client-side APIs consistent with server-side APIs (e.g., libraries for accessing files could be reused on the server or client).

Your point 3 is maybe the most valid toward potential convenience for a shared syntax by developers, and I suppose Common Node is an example of this, although if Node IS SSJS then I suppose one would view such attempts as redundant or even unnecessarily forking the predominant style.

At the least, it would seem to me to be healthy that if more viable alternatives did come along (if that is not the case now), I would hope there would in such a case at least be a willingness to collaborate toward eventual standardization along the lines of WhatWG with new APIs being prefixed, as I think it should at least be recognized that standardization is a good thing once enough experience is gained through experimentation (and of course assuming similar competitive implementations exist---as is the case with browsers).

[isaacs]

Re Common Node, meh. Just another module on npm. That's the place for doing crazy things with node. But it sort of proves the point that focusing on speed, portability, and extensibility is better than focusing on some vague normative "compliance" with a spec made up by people who are not invested in our project in the first place. They can obviously do their own thing on top of our APIs if they want.

It sounds like you're looking for a standard for Server-Side JavaScript. Let me make this more clear: as far as we're concerned, Node is the standard for server-side JavaScript platforms. We are defining it, as a community, in reality. You want a spec? Fine. Here it is.

[ghost]

I have taken to labeling node's module system as "commonj-esque" since it gives a more concrete name for the idea of "however node does modules". If you want to build a new tool that needs to interoperate with existing code written for node, then just copy how node works. You don't need to ask a standards body for permission. The node module system in particular is a great informal spec to code against because ever since 0.4 it hasn't changed at all and there is a built-in mechanism for ensuring that this stability endures: existing programs will break en masse with even minor amounts of tinkering. The heaps of inter-dependent software built up around how modules presently work is more than adequate to enforce stability.

I didn't need the blessing of a standards body, node or npm to build browserify. I just read the docs and built it. No standards necessary. Standards for their own sake is very silly. Standards bodies should need to elaborate a very clear reason for their own existence and that self-justification should be subject to review at constant intervals. Except in rare cases they are just paperwork-driven programming.

[jbergstroem]

Don't want to ruin the party, but could we not pollute notifications with standards discussions? If there's a bug or feature, attach a test case and aim at fixing it. There's probably a mailing list somewhere that's better suited for this.

[brettz9]

Wasn't disagreeing with you, btw. Your argument seems well-reasoned to me and apparently backed up by the high adoption rate of Node. I just hope that if viable alternatives do come along that developers won't be forced into different styles. But I think you all have convinced me to start with Node as the base.

[isaacs]

@jbergstroem The OP here is about standards, so it's really not pollution. Stop watching the thread if you don't want to hear it. Note the little button on the lower-left.

[jbergstroem]

@isaacs Wow. Always scroll to the bottom. Thanks.

[brettz9]

I thought it also might be insightful to point out (along the lines of your argument perhaps of Node being the standard) that even the editor of HTML stated to the effect that they tried to adhere to IE behavior wherever possible.

Practical standardization (even up to international governance!) does seem to need to begin with a need to accommodate those in a position of success (even while remaining open to consultation and possible eventual changes).

Anyhow, maybe a request could be turned out of this issue to create some FAQ entry to address the issue of your approach to standardization.

Thanks again.

[mikeal]

It's important to note that, while many people like to label us as such, node people are not a paramilitary force bent on destroying standards bodies and trained from birth in an Al-Qaeda-esque anti-standards training camp.

The node community is actually filled with a lot of standards cast offs, people who spent time in the system and left out of frustration. I've been involved in standards at Calconnect, the IETF, and eventually CommonJS where I gave up entirely. The mantra of "implementation first, standardization last and maybe never" comes from the many years that many people in this community spent trying to make progress through what is considering the blessed process and coming up short.

Node's success is because all of us, at one point or another, gave up on standards and process and just built something. The biggest mistakes and warts in the platform are from where we continued to try and maintain compatibility with what everyone else was doing and now we're stuck with those warts forever.

Although our indifference is often characterized as antagonism we very truly just don't give a shit what everyone else is doing and are committed to doing the best that we can here, and now, and in real tangible improvements and successes people can use.

[kriskowal]

The hypothesis of CommonJS was that the reason no server-side JavaScript framework had taken off (by 2009) was the lack of continuity or completeness among the many existing platforms. @dangoor enticed the owners of existing server-side JavaScript engines to band together and solve those problems. This gave me an opportunity to pitch my module system (with Ihab Awad), and my continued involvement was to ensure that enough API’s existed to bootstrap the module ecosystem, and to take advantage of a forum to air my petty grievances with similar API’s in other languages.

Node (Ryan) took CommonJS modules and the insight that we needed async IO, which is obvious in retrospect. That insight invalidated a lot of my designs. I would have pivoted differently, adjusting my existing designs to use promises. I pitched the idea to Ryan at the SFSSJS meetup @tlrobinson and I organized after JSConf. He went with callbacks and streams in the interest of speed and reducing GC churn. It was for the best. Ryan very wisely pushed anything he did not grok out to user-space. Based on the “Promises” design in v0.1, he clearly did not grok promises. It has taken some years to educate a critical mass of people who can recognize a good promise design. We have our enclave in user-space and it is just fine.

I am thrilled that all the work beyond the modules and packages specifications was unnecessary to ensure the proliferation and survival of CommonJS modules. The modules and packages specifications were deliberately incomplete to avoid trying to solving problems that needed more exploration, like how to compose the module identifier namespaces of packages. Narwhal tried one approach and @isaacs and Ryan iterated on that with Node and NPM profitably a couple times. Node has extensions to the specifications that are clever, judicious, and have good emergent qualities (with the exception of directory index.js). Every other thing I would have done differently amounts to coloring bike sheds and I really don’t have time. I said my piece more than once.

We achieved ignition and CommonJS went home. We have a great trajectory with Node. The next big thing will not be backward compatible, so don’t get hung up on specs.

[dangoor]

Looking at the OP, I'm not even sure what would be standardized and between which implementations. What would the goal be? To elevate AMD (which I see referenced a few comments up)? That seems unnecessary, especially given that JavaScript itself is on track to get modules.

I agree with pretty much everything written by @isaacs, @mikeal and @kriskowal on this thread. In my 4-year-old blog post, I was largely write about what Python gave me that JS did not. I was wrong about multi-implementation standards being the right path to get there, and I'm delighted that we've reached a point with Node that the ecosystem is as rich as Python's (modulo some differences that I think can be attributed to age).

I also think that Node's tight core with a huge ecosystem is very cool and a testament to the value of not standardizing too much prematurely.

(I hate to pile on with a "me, too", but I honestly don't get what problem is trying to be solved here.)

[isaacs]

Wow. Well, there you have it. I don't think you could find two pillars of CommonJS that were more fundamental to the whole enterprise than @dangoor and @kriskowal, and they both seem to agree that the direction we've ended up taking is mostly for the best.

Case closed, I guess.

In retrospect, it seems to me like the goal of CommonJS was (or at least, should have been) to discover Node, and enable what we've built here. Some mistakes were made, because hindsight doesn't go in the direction you'd like it to, but overall, I think the whole CommonJS project can be considered a success.

[L2L2L]

Hey @issacs ... I read your post on common js... And can't it just be an add on to node js? Features that are consider low APIs but are still not the ideal of Node programming. Who say the body have to come to an argument? You can break common js in to section so the body won't be arguing over something that matter to the other half. I believe that you and the developers who follow are paying the way for use. We don't care what the others say, we are looking toward what you all are saying... Sorry to just post like this, I was on wiki and was lead here after reading the title to the link. Curiosity is what kill the cat--good thing my curiosity is aim toward learning in books--