Configure paths by pattern or module name rewriter?

[mk-pmb]

Hi, thanks for curl! (and when, too.)
Could there be a way to rewrite module names or configure paths by a function? I'm looking for a pattern matching (here: every module name that starts with "phorkys-") mass solution to replace the long list:

  [ 'phorkys-core',
    'phorkys-dialog',
    'phorkys-quiz',
    'phorkys-scenetype-floor',
    'phorkys-scenetype-landscape',
    /* ... */
    'phorkys-ui'
  ].forEach(function (mod) {
    paths[mod] = 'lib/' + mod + '/' + mod.replace(/^\S+\-/, '');
  });

My first idea was to replace require() with a proxy that does regexp replace on the module name and then passes it on to the original require(), but then I realized that it is probably not used as a real function but rather a macro. So where could I hook into, or what would be a better way to give a pattern instead of a long list?

Yours, MK

Edit: API might be something like

  curl.config({ resolvers: [
    function resolvePhorkys(resName) {
      if (resName.substr(0, 8) !== 'phorkys-') { return; }
      return ('lib/' + resName + '/' + resName.replace(/^\S+\-/, ''));
    }
  ] });

[unscriptable]

Hey @mk-pmb,

This is a fun problem. First, I have to ask if there's an easier solution: how hard would it be to refactor these modules so they use normal CommonJS/AMD module ids? For instance, use "phorkys/scenetype/landscape" instead of "phorkys-scenetype-landscape"?

Replacing require() would almost work. The hard part would be replacing it inside every module since each module gets its own copy. The curl/shim/dojo18 shim does this. You would also have to override define() since cram.js removes inline require() calls and replaces them with dependencies in the define(). Overriding define() is actually easier. You just have to remember to set define.amd = {}.

I think the best way to resolve this is with a shim (like the dojo18 shim above) or a loader (such as these. First, you'll have to decide if the transformation is an id transform or a url transform. There's a subtle difference:

• id transforms happen in the context of the requesting module. These are often used to provide special versions of libraries. For instance, a require("jquery") might be get remapped from "jquery" to "zepto". These are rarer.
• url (path) transforms happen in the context of the requested module. These are often used to point to different versions. For instance a require("jquery") might be get remapped to "lib/jquery-1.9.3/jquery.min" or "lib/jquery-2.1.1/jquery.min".

I'm guessing you need a path transform.

A loader can be applied to an entire package, but it has to handle every module in that package. You cannot handle some with the loader (e.g. you can't remap the path of some modules) and not others.

A shim overrides some internal function of curl. These are much more flexible, but rely on internals so you'll want to take note during major version changes.

Any ideas, thoughts?

-- John

[mk-pmb]

Hi John,

thanks for the quick reply. Refactoring will be possible, but must remain compatible to browserify and nodejs. (lib/is an alias to node_modules/.) I'm not an expert in node modules, so maybe there is a structure that can serve both.

Since the path config works and I'm not sure about possible side effects of id transforms, I tend to agree on path transform being the best choice. Do I understand correctly that

A loader can be applied to an entire package

means I'd have to move all of phorkys' dependencies into lib/? If that is the only limitation, the re-org part should be easy.

Yours, MK

Edit: While we're at it, currently all the phorkys modules have a wrapper in them to detect whether they are loaded from AMD or nodejs, because I failed to configure legacy mode for them. (Probably would have worked with "js!" in front of every module name inside, but that would break node.) With an extra loader, could I enable this legacy mode to save those redundant wrappers in each module? Does legacy mode come at a cost?

[mk-pmb]

Checked again, nodejs compatibility is already available the other way around, using RequireJS in node. Browserify compatibility should probably be reduced to making browserify work with node's RequireJS if not yet. The remaining compatibility problems seem to be only about future npm publishing, which I'm not yet used to.

[unscriptable]

Hey @mk-pmb,

I'm getting pretty confused by this conversation. :P

I'm not sure I'm on the right path, but here's an attempt at a recommendation/solution...

1. AMD and CommonJS (node.js) share a common specification for module ids and paths. Whatever solution you provide for one will work with the other.
2. Your phorkys modules seem to be organized into well-defined categories. (This is good!) In CommonJS and AMD, we call these "packages". You should consider whether you have a single package, "phorkys", or several packages: "phorkys-core", "phorkys-ui", "phorkys-scenetype", etc. It's also possible for "phorkys" to be a combined package for the other packages. I don't really know your code, but it feels like "phorkys-scenetype-floor" is too fine-grained, so you should leave the package at "phorkys-scenetype" and "floor" modules should be under "phorkys-scenetype/floor".
3. Decide whether to use AMD, CommonJS/node, or UMD format in your modules.
   a. AMD: AMD works great in browsers. Unfortunately, using AMD in node requires bypassing the built-in requier and using r.js or curl.js, instead.
   b. UMD: UMD works in AMD and CommonJS/node. Below is the boilerplate we like to use. Unfortunately, most node.js developers are snooty about module format. They don't like to see UMD.
   c. CommonJS/node: These will work fine in node, of course. However, to run them in a browser, you either need to use browserify to convert them to an AMD bundle, which makes the development more cumbersome, or to use curl.js's cjsm11 loader. The cjsm11 loader will convert the modules to AMD on-the-fly, if needed. There si no need for boilerplate. When used with cram.js, it will convert them at build time, like browserify.
4. Decide which package manager(s) to publish to.
   a. npm: The only choice for node, but not so friendly for front-end development.
   b. bower: Friendly to front-end development, but not as nice for node dev.
   c. both npm and bower: the best choice, imho!
   d. others: there are still others, but I don't know much about them.

UMD boilerplate:

// boilerplate starts with an IIFE...
(function (define) {
define(function (require, exports, module) {
    // inside here is where your code goes. it should be commonjs format
    var dep = require('phorkys/core/foo');
    module.exports = function bar (val) {
        return dep(val);
    };
   // more boilerplate below...
});
}(typeof define == 'function' && define.amd ? define : function (factory) { module.exports = factory(require, module, exports); }));

I hope this helps?

-- John

[mk-pmb]

Thanks for the overview and suggestions! I'll see how far I get with these, probably starting with the cjsm11 loader, and ask again when I find a wall in front of me.

I'll try and stay with just npm until I discover its limits. The modules are very fine-grained because there are quite some many types of scenes, furniture, transportation and so on, and I only need a few of those categories in most projects. (For example, I haven't yet had chariots and jetpacks together.)
I like mini modules like defos as well as collections like lodash.

Yours, MK

[unscriptable]

Cool. I'm going to close this issue. Feel free to continue to comment on it. If you'd like to start a lengthy discussion, you could also try the #cujojs freenode channel or our google group.