Update export mechanism (to support a switch from Closure Compiler to UglifyJS)

[brianmhunt]

What the Closure Compiler is doing for Knockout is very clever, and quite interesting. I wonder though if this design is getting a little long in the tooth, as many things have changed in the Javascript world in the past couple years. The core hurdle to switching from Closure Compiler to another minifier (e.g. UglifyJS) would seem to be that Closure Compiler chooses a subset of symbols to not mangle, which subset becomes the 'exported' symbols.

The typical development style nowadays is to use a build script like Grunt or Gulp, get source code via a package manager like npm or bower, and create debugging and production scripts it with a modular compiler like browserify or r.js and run the result (a single .js file) through a minification system - UglifyJS, possibly generating source maps for the entire codebase with respective links to original files.

Closure Compiler's use does not play well in this pattern, which impedes the introduction and use of Knockout by the mainstream community. It makes it harder to package Knockout (e.g. Java is required); the conflation of minifying with crystallizing of certain symbols for export is confusing and there is no warning of this issue when using a packaging system like Bower; exploiting Closure Compiler's advanced options bogs down linters with issues; users must decide between including a minified version or not be able to debug what is going on internally; there is no source map of the production version and including a generated one via browserify/r.js then uglify.js is complicated and brittle.

It is my understanding that Closure Compiler is desirable because it does a better job of compiling things than UglifyJS i.e. the size of the result is less.

A point @domenic made in #495 is I think worth repeating:

... A difference of even 10 KB isn't going to change my opinion of your project's usefulness (I can get more than that by using pngcrush). But the cost to source code clarity and contributor-unfriendliness is very high.

My main concerns here stem from the use of Closure Compiler (curl in a build script, really??) and how it makes you litter the code with quoted property names, use crazy stuff like ko.exportSymbol, and introduce IIFE wrappers into many files.

Switching to Uglify would be a huge improvement. It actually respects JavaScript's rules, like, say, a["b"] === a.b. It doesn't mangle names (which makes Knockout objects a gross pain to debug in production). It doesn't require maintaining a ko.exportSymbol list at the bottom of every file. It just works like normal JavaScript.

Not to belabour the point — I am sure you get the idea. 😃 However, to add to @domenic's note: the essential long-term issue with any open source project is the vibrance of the ecosystem in which it resides, as without a healthy and regular injection of sincere interest by capable people the volunteer maintainers will suffer fatigue and the project will ultimately wither. Principles are fine, but existence is finer. :) Not that fatigue is necessarily an issue with Knockout - just something to be mindful of.

As noted above the cost of this freedom from Closure Compiler is refactoring the export mechanism. This is a really nifty bit of code to kill. As the writers say, though: "kill your darlings".

With the export mechanism updated, one could readily switch between UglifyJS and Closure Compiler. Here is a sample of issues that could be resolved in the result:
#1211 #835 #1312 #1078 #1184 #1362 ; Also worth noting #437.

Sorry for the length – I had just hoped to give a thorough explanation in the assertion. :)

[soundofjw]

We (www.loudr.fm) use the original knockout.js in our closure compiler build chain (with advanced compilation, no less!). We love that it's compatible and built with exports in mind.

While I can appreciate the desire to move away from Closure, not maintaining the exports will break things for us.

Please, for the love of all things knockout, do not kill your darlings!

[brianmhunt]

Not knowing Closure well enough, I wonder if there would be a way to retain the benefits of the exports style yet make it more canonical and accessible for the "rest of us". :)

The core concern is community involvement and project sustainability. If I could quote Bob Beck (of OpenBSD) in his BSDCan 2014 presentation on LibreSSL because I think he hits the nail on the head:

That horrible code that you don't want to look at actively discourages outside involvement. It actively discourages developers and the community from looking at it. Why? We all look at it and say "I don't have the time and energy to invest to understand this arcane stuff ... just to see if it's ok. I'm going to hope that the person that writes in this arcane style understands it enough and cares enough to take care of it."

Which is not to compare Knockout to OpenSSL, but the generic problem is the same: the more arcane the code, the harder it is to get community contributions. Closure has all kinds of arcanum that exists outside the canon for Javascript. (As if Javascript weren't bad enough!) I am afraid by using Closure the result is that many Javascript developers see the Knockout code and their thought process is what Bob said. It has been so with me, FWIW, which is perhaps the main reason why my contributions have skirted the core (e.g. the #1360 build proposal).

Here is an interesting Google Trends: knockout.js | angular.js | closure compiler | uglify | browserify. The graph shows Knockout trending down substantially since March 2013, closure compiler has been stagnant since basically 2010, and angular.js, browserify and uglify have skyrocketed since 2013. Closure has not caught on the way Browserify and Uglify have.

Here is an alternative trend that illustrates how Closure compiler has not been "trending": Google Trends: knockout js | closure js | browserify js | uglify js. I took Angular.js out because skews the chart. You can see everything there going up quite exponentially, save Closure which is dropping. That may be a temporary trend, but I think it ought to be concerning. Fewer people using Closure means a smaller community of individuals that will not be put off by its peculiar arcanum.

The key to a healthy self-sustaining ecosystem is one that is inviting to new developers. With some exceptions, I think Closure generally creates a hostile environment. This is not to say anything about the technical capabilities of Closure, which are quite amazing, only the environment it sets for community involvement.

All that said, it would not be fair to change away from Closure and break things for folks like @soundofjw in a minor or patch release. I feel this is a major change that ought to be accompanied by a major version change or API compatible fork.

Cheers.

[PeteDevoy]

The core hurdle to switching from Closure Compiler to another minifier (e.g. UglifyJS) would seem to be that Closure Compiler chooses a subset of symbols to not mangle, which subset becomes the 'exported' symbols.

Apologies if I misunderstand you but I believe it is more complex then that. As far as I understand it, exported symbols are still 'mangled' in the sense that they are minified but the crux is they are not treated as 'dead code' by the compiler running in ADVANCED_COMPILATIONS mode if they are not referenced in the compiled JavaScript. E.g. https://developers.google.com/closure/compiler/docs/api-tutorial3?hl=en

I develop with a lot with OpenLayers3 which is fully type-annotated for the Closure compiler and they also have a dev workflow which uses Node for 'live' debugging of sources. Admittedly though I think they are looking at moving away from this workflow (but still using the closure typing system and Closure Compiler).

Albeit popularity important for open source projects I think when it comes to the wider development trends. I think it is dangerous to equate 'popular' with 'good' or 'right'. Totally unfounded personal opinion but the current web development paradigm seems a bit ridiculous and reminiscent of .dll hell to me... There are about half a dozen JS libs for almost everything imaginable and, in my experience, many of them fall short of being useful for any project that is more than another admin dashboard.

On that note, I think it is worth thinking about the divide and conquer effect of a 'healthy ecosystem' on Open Source ( similar to this XKCD on standards https://xkcd.com/927/ ) and also the wider context of a lot of package development in the Node ecosystem. https://www.youtube.com/watch?v=bzkRVzciAZg

Regardless, if looking to find out the barriers to contributions surely the best way would just be to put a simple poll on the KO homepage. People may not be contributing because the code base is not documented enough, or because they are not comfortable enough with MVVM, etc.

If we only did what was popular Knockout would be another MVC framework ;)

[tjhiggins]

+1