Migrate Chiasm to JSPM

[curran]

Chiasm plugins should each live in their own separate git repositories, and be included in a project via JSPM. This is a similar approach to the Gulp system and plugin architecture.

[Hypercubed]

Are you planning to switch completely to JSPM/SystemJS? Do you/we need to support requirejs in chiasm and chiasm plugins?

[curran]

Yes, I want to eliminate RequireJS completely from the Chiasm project, and use JSPM and SystemJS instead. The main reasons are:

• no more headaches with manually tweaking the RequireJS config to get it to work with dynamic loading as well as bundling (hopefully), and
• the ability to load and bundle Chiasm plugin modules authored in many formats (ES6, CommonJS, AMD).

The one thing I'm not exactly sure about is how to bundle a Chiasm application for inclusion in a production flow, like the Rails Asset Pipeline (which Chiasm must support). It should be possible to use JSPM to create a self-executing bundle that exposes Chiasm as a Browser global AND supports dynamic instantiation of plugins included in the bundle. I haven't seen this work yet, but because JSPM creates bundles that include a version of System.import that can load bundled modules, I'm convinced that it should be feasible.

I would like to go down this road, so if you have the energy to try swapping out RequireJS for JSPM and SystemJS completely, then go for it!

Also, you might want to check out the tooling that the new version of D3 is using - see d3-selection and d3-bundler. Based on this work, it feels like Rollup would be a great bundling solution, but it's not clear to me if it's possible to use JSPM with Rollup, or if that would give a significant advantage over the JSPM/SystemJS bundling strategy.

Here are a few related threads:

• Rollup - Resolvers and transformers (possibly using JSPM and Rollup together)
• D3-bundle - List (and install) indirect dependencies?

[curran]

On second thought, I might just want to go with CommonJS and NPM for publishing Chiasm-related modules, then use Browserify for bundling. It seems these tools have a lot more traction than JSPM, they are more mature and pleasant to work with, and they achieve almost the same thing as JSPM (single install, single require).

After having tried out JSPM a bit, I find the following two things to be the most questionable aspects:

• JSPM automatically installs Traceur as a dev dependency. I can see how this makes sense in a way, for supporting ES6, but if a modules does not use ES6 I don't think it should be forced to depend on Traceur. Related: SystemJS Google Group: Why is Traceur a default dependency with JSPM?, jspm-cli issue: Allow no transpiler option. Every time I jspm install something, Traceur is installed and I need to manually delete it from config.js, package.json, and jspm_packages.
• The bundles created by JSPM seem to have a lot of extraneous stuff in them as compared to the bundles created by Browserify. Also, JSPM bundles only target System.register, which is future looking, but also precludes publishing the bundle for use in AMD or CJS systems. Also to expose the bundle as a browser global, one needs to write custom code, whereas Browserify does this for you.

Even for ES6, it seems that many people are using a workflow where they transpile to ES5 first, then publish to NPM (see this thread Reddit: What's the best way to publish ES6 modules?.

[curran]

In evaluating all of the options for bundling and plugin authoring for Chiasm, I ended up created this screencast on JS modules and build tools.

I'm leaning more and more toward NPM + Browserify for splitting up the Chiasm codebase across repositories, mainly because this is a relatively mature toolchain that works extremely well. This means Chiasm will impose NPM as a plugin publishing and install mechanism, and force developers to deliver plugins as NPM modules (CommonJS). This does not preclude authoring plugins as ES6, because the ES6 code can be transpiled to ES5 before publishing to NPM, which is already a fairly common pattern.

In a way, this is less tooling to impose as compared to JSPM, because to use JSPM, one already needs to be using NPM, and additionally JSPM introduces the "config.js" file, and dependencies on ES6-related things. I'm just not convinced JSPM is stable/mature enough for use as the Chiasm bundler, and the space for authoring modules in ES6 and transpiling to ES5 is vast.

I'm planning to start breaking out Chiasm plugins soon, using NPM.

[Hypercubed]

This is a reasonable approach. NPM/CJS modules are easy to install using JSPM as well, if the developer chooses. I would hope, at least, that the plugin system continues to support SystemJS.

Although I don't know if NPM + Browserfy + RequireJS is any less tooling that NPM + JSPM + SystemJS.

[curran]

Thank you for your input. This new approach would eliminate RequireJS, so
the tools would only include NPM + Browserfy.

I would however be happy to keep support for dynamic loading with SystemJS
if that is something you'd like to use.

My main concern is proper bundling for production, and splitting out the
codebase into modules in such a way that bundling is straightforward. The
outline for the bundling strategy is here in this README -
https://github.com/curran/chiasm-bundle

On Wed, Jun 17, 2015 at 7:07 PM, Jayson Harshbarger <
notifications@github.com> wrote:

This is a reasonable approach. NPM/CJS modules are easy to install using
JSPM as well, if the developer chooses. I would hope, at least, that the
plugin system continues to support SystemJS.

Although I don't know if NPM + Browserfy + RequireJS is any less tooling
that NPM + JSPM + SystemJS.

—
Reply to this email directly or view it on GitHub
#29 (comment).

[curran]

The more I consider various use cases, I think that loading plugins on-demand is actually a fundamental requirement for Chiasm. In some cases it makes sense to bundle the required plugins, but for more general use cases, it may not be possible to know ahead of time which plugins will be required. If the set of plugins grows to hundreds some day, it would not make sense to bundle all of them with all of their dependencies if only a few are ever used.

I think I'm going again down the road of evaluating JSPM, and trying to make it work for Chiasm both with bundling and with dynamic module loading. I was turned off initially by the huge bundle produced by JSPM, and how it just felt a little bloated all around, but I think since that time there has been tons of work on JSPM that address some of the issues I had.

[Hypercubed]

I think the benefit of JSPM bundle vs other tools is that:

1. unlike gulp-wiredep (for example) it only includes modules that are actually used by analyzing the module tree. So at the app level you can JSPM install as many modules as you like and only ones that are used are bundled. Of course, as we found, this is difficult for Chiasm plugin loading. Shomewhere you need to specify what plugins are used (https://github.com/Hypercubed/Project-Chi/blob/master/gulpfile.js#L12).
2. Other tools, from what I can see, just do a textual include. jspm bundle keeps modules as modules. This means some added noise in the jspm bundle. However, I agree at the plugin level rollup or Esperanto should probably be used.

Two things to note about the jspm bundle. You can turn on minify and make sure you source maps are not inline.

[curran]

Here's a new idea: I think it would be cool to make a SystemJS module loader plugin for Chiasm. When the plugin is instantiated in a Chiasm configuration, it can be set up with local (possibly in jspm_packages) or remote (CDN) paths for other Chiasm components. For this, there will need to be a hook in the Chiasm API that would allow plugins to register and unregister themselves as module loaders. This would also make it possible to, for example, build another totally separate loader plugin that uses RequireJS.

[Hypercubed]

Honestly I haven't looked at the project in detail lately so I am falling behind. I'm not sure I follow how a SystemJS plugin can help. If plugins are all JavaScript files then one can load all plugins explicitly:

      var chiasm = Chiasm();

      chiasm.plugins.layout = require('ChiasmLayout');
      chiasm.plugins.links = require('ChiasmLinks');
      chiasm.plugins.dataLoader = require('DataLoader');
      chiasm.plugins.pieChart = require('PieChart');

JSPM builder would love this. Each plugin can load and register it's dependencies the same way, possibly under an alias to avoid collision.

[Hypercubed]

@curran Reading this thread and looking back at our previous discussions I can't recall why lazy loading of plugins was needed. I added the SystemJS loading support to the Chiasm code and went through some trouble in project-χ (here: https://github.com/Hypercubed/Project-Chi/blob/dc759095ad1cff7cd4ad850857389f084d2bcedd/gulpfile.js#L12) to get the chiasm plugins bundled so that they can be loaded by the loadPlugin method (https://github.com/chiasm-project/chiasm/blob/master/src/index.js#L166).

But why?

The approach you are using in your examples (simply add the plugin constructors to the chiasm instance plugins hash) seams much more maintainable and robust. A developer that has installed chiasm and plugins (using jspm or whatever) needs to create a chiasm instance in the application. For each instance the necessary plugins should be added. Am I missing something?

[Hypercubed]

I've updated project-χ with the latest chiasm version (including new layout, links, and bar chart from your example): https://github.com/Hypercubed/Project-Chi/tree/chiasm/app/components/examples/chiasm

Everything installed fine using jspm specifying the npm registry (jspm install chiasm=npm:chiasm). Installing directly from github would require adding the registry to package.json. I'm explicitly importing the plugins (chiasm-layout and chiasm-links) and adding to my chiasm instance. So loadPlugin is not needed at all. I had to jspm install chiasm-component also because the barChart plugin is local and requires it. I had to add lodash to window because we are missing the import in chiasm-layout (see chiasm-project/chiasm-layout#1).

[curran]

Right, I think this has to do with the history of the project. Originally, Chiasm was using Require.js to dynamically load plugins. Then the need for bundling came up, and I had to re-think how the plugins were being loaded. At this point the chiasm.plugins object was added, so bundling tools can know which plugins to bundle. Currently, using chiasm.plugins is the way I'm making new examples, and this works well. Indeed, I think it is more maintainable and robust than the previous solutions that depended on either Require.js or System.js.

However, in a way it would be nice to be able to include a plugin in the configuration only and have everything "just work". With the new examples, each plugin needs to be defined in three places:

1. loaded with a <script> tag.
2. assigned to a name in chiasm.plugins
3. invoked in the configuration.

I suppose this is fine, but I have a vague idea that it would nice to bring back dynamic plugin loading for cases where bundling is not required (e.g. in bl.ocks.org examples). This way, to use a new plugin, all you'd need to do is change the configuration (maybe the plugin property could refer to a CDN URL, like "http://chiasm-project.github.io/chiasm-links/chiasm-links-v0.2.0.js"). I'm not sure if this is a good idea or not, but this is what I was thinking with the SystemJS module loader plugin comment.

I'm happy to hear you are trying out the latest Chiasm stuff in project-χ. Thanks for filing the issue on the missing lodash dependency, I will fix it.

[Hypercubed]

Your message got cut off... but I can guess. No matter what the developer needs to create an instance of chiasm (var chiasm = new Chiasm();) that seams like an appropriate time to add the desired plugins. Not all chiasm instances need every plugin and in fact may use different versions of the the same plugin. For example chiasm.plugin.barChart = require('myExperimentalBarChartPlugin').

My vote would be to remove the System.amdRequire section from the loadPlugin. I added it but it's not needed anymore and introduces a loading method that doesn't bundle easily. Chiasm supports JSPM just fine now. Maybe even close this issue.

[curran]

Great, my thoughts exactly. Regarding the System.amdRequire section, I will remove it from the core Chiasm code, but I'll think about maybe keeping a hook in there so custom dynamic loaders can be defined in the future.

Closing the issue - the Gulp-like plugin organization has been achieved, and JSPM is supported via explicit plugin setup.

Thanks for your input on this!

[curran]

Done in 4b53ec1