Extension API Evolution: module loader

[jrburke]

I came across the Extension API research:
https://github.com/adobe/brackets/wiki/Extension-API-Research

and curious to know if you need any background or ideas on how to use an AMD loader more effectively for what you need.

The main issue seemed to be using modules installed via NPM in brackets for brackets extensions. This would work out better if there was a process that added a bridge module in the node_modules directory to the "main" module in the node_modules area. Then setting the loader's baseUrl config to 'node_modules' would work out.

This assumes that the packages installed do not have nested node_modules dependencies. At first thought I would expect you would not want that anyway given that it meant extra code getting downloaded for in-browser use. However, they could be supported in an AMD loader by using map config if they were desired.

I have an in-process project to write out those adapter modules for node_modules, then also to convert them to AMD format by just adding a define() around the modules installed. Although, I think it is completely fine to also just author the modules in AMD format. People are already using npm for non-node, browser only code, so a concern about module format should not be an issue. But either way, a post-install tool that sets up the bridge modules could also optionally wrap the modules.

If you need some of those same modules to run in a backing node process, r.js can be used in node to load those modules in node, and it could reuse the same config you use for the front end.

This way you could take advantage of the callback require() available in AMD loaders as well as loader plugins, for things like text files, template and transpiled languages. Both of those features have companions in ES6 Modules space, so I expect it will be easy to transition use of them to ES6. The callback require for delayed loading of front end code is also a pretty standard feature request for a project the size of Brackets.

You may have other requirements I have missed, so just offering help in case you think an additional exploration down this path might be useful. No worries though if you have it sorted.

[pthiess]

@peterflynn

[pthiess]

@jrburke Thanks for reviewing the proposal and your feedback. @peterflynn I'm having a hard time to prioritize this - I'm even inclined to see this as a non issue, rather a discussion but I also know that you guys like to discuss things on github. Please use your judgment to steer this in the right direction.

[peterflynn]

@jrburke Thanks for reaching out! And sorry for the slow reply...

@dangoor can probably explain the thinking on NPM-style resolution the best. There are also notes here: https://github.com/adobe/brackets/wiki/ModuleLoader. We're definitely not set on an implementation path yet. We're planning a research story to investigate the options, where I imagine we'll spend a good deal of time looking at Require & Cajon more deeply.

But we do have a number of other goals beyond the NPM stuff -- some immediate, some longer-term -- that collectively make it look like we might need a more custom-tailored approach to module loading. Many of our needs have to do with Brackets' extensibility mechanism:

• Extension-specific require contexts that are able to load both from the extension's source tree as well as the main "Brackets core" source tree (we currently get around this by providing an alternate brackets.getModule() API for the latter).
• Ability to return proxy/wrappers around Brackets core modules, per extension context
• Enable modules in one extension context to require modules from a different extension context (using an extension id without knowledge of the exact relative path). Plus filtering -- if an extension's metadata doesn't indicate the dependency, we want to pretend the module can't be resolved.
• Also, not extension-related: inject mock objects in place of modules for unit testing scenarios (we experimented with some Require features that sounded promising here, but hit various roadblocks that I haven't remembered yet)

It seems to me like some of those needs might be addressable via custom RequrieJS plugins, but I don't know enough about plugin capabilities to have a clear picture of that (yet!). We'd definitely love to hear any thoughts or suggestions you have!

[peterflynn]

Btw, any chance you'll be at JSConf.EU? Both @dangoor and I will be there, so we could actually talk more face to face :-)

[jrburke]

I am coming back to this late, but the scenarios sound like map config may be useful for the scenarios you mention.

My first thought is that an extension-specific require would load modules from its source tree via relative require() calls (starting with './') and use top level names. But map config may be useful here if you know the extension has an override for a top level name.

For proxies and mocks, map config can be used for that, are great for those cases. If you need ideas on how to set that up, feel free to let me know.

For extensions that require modules from another extension, I would expect each extension to have a top level module ID prefix (so for a module ID 'some/thing/here', the 'some' part is the top level ID prefix), but that too could be set up via map config for a particular extension if it should only apply to that extension.

Unfortunately I was not at JSConf.EU, sounds like it was fun though!

[dangoor]

Hey @jrburke... sorry for chiming in 6 months later, but I just spent a little time thinking about this and I wanted to pass along a couple of thoughts.

access to node modules

What if Brackets could provide a directory listing to RequireJS and then Require could do Node-style resolution against that directory listing? This could even work in a more typical browser context, if someone has a way to provide Require with that list of files. It seems to me that the big problem with Node-style module resolution in the browser is that you can't get directory listings or quickly look in a bunch of different locations for a file, which is why having a list of available files would help.

With this plus Cajon, it seems possible to npm install something and have it work right away.

providing a façade to modules

Brackets extensions currently access core Brackets features by doing something like var CommandManager = brackets.getModule("command/CommandManager");. I believe that brackets.getModule is just the require function that is given to the first Brackets module loaded.

What I'd like to be able to do is something like this in an extension:

var CommandManager = require("brackets/CommandManager");
CommandManager.addCommand(...);

The first interesting part is something like a mapping, but there's a different base URL. (It's more like a package...)

The hidden, but more interesting bit, is that CommandManager.addCommand(...) is actually a façade that calls CommandManager.addCommand(extensionName, ...) so that commands can be transparently registered as having come from the extension named. So, basically, I'd like to be able to wrap the module object that gets passed in to the extension's module.

This could probably be done today using config, but:

module.config().CommandManager.addCommand(...)

is kind of ugly. Perhaps it could be done with a Require plugin?

var CommandManager = require("brackets|CommandManager");

It seems like this could work pretty easily and isn't particularly ugly. What do you think? (any opinion on this, @peterflynn?)

It would be nice to be able to do the equivalent by configuring the behavior of requiring anything in the brackets.* namespace so that a require("brackets.CommandManager") just provided the custom behavior we want.

cross-extension require

We want extensions to be able to provide services to one another. So,

var someservice = require("some-other-extension/someservice");

would need to not only know that some-other-extension is in a different subdirectory (likely a sibling to the baseURL of that require), but would also need to return the same module instance from that other extension. It's important that it's the same instance for things like the TypeScript extension which does expensive parsing/compiling operations that you want to share the results of between extensions.

It seems like the module cache would ensure that modules requested with the same URL get the same module object back, but is that true if each extension has its own require context? (It may be, in which case figuring out how to do the mapping is all that's needed!)

RequireJS (and Cajon) is very full-featured so everything above seems in the realm of possibility, but I wanted to get your take on how what we're looking for intersects with what Require already does and what would be straightforward (or more complex!) additions to require.

[dangoor]

Oh yeah, and thanks, James, for all of your input on this already!

[jrburke]

Hi @dangoor! Good to talk with you again. Notes below:

access to modules

This could be possible in two ways:

1. If you can get a shallow installation of the node modules, such that all packages are in node_modules/packageName, and not nested, then all you would need to do is to write an adapter module at node_modules/packageName.js and have it just have one dependency, on 'packageName/main' (substitute "main" for whatever the main module ID is inside the package). Then no special config is needed to pass to requirejs, if baseUrl is set to node_modules.

2. If you cannot avoid the nested node_modules installs, then a config could be constructed for requirejs that uses the package config to set up the package main wiring, then use map config to point module IDs in the nested folder to their nested location.

An example:

• node_modules
  • foo
    • index.js (the 'main' for the 'foo' package)
    • node_modules
      • bar
        lib/index.js (the 'main' for this 'bar' package)

Then, if baseUrl is set to the topmost 'node_modules', then the config would look like so:

requirejs.config({
  baseUrl: 'node_modules',
  packages: [
    {
      name: 'foo',
      main: 'index'
    },
    {
      name: 'foo/node_modules/bar',
      main: 'lib/index'
    }
  ],
  map: {
    'foo': {
      'bar': 'foo/node_modules/bar'
    }
  }
})

You could skip the package config section if you do write out those adapter modules a 'node_modules/packageName.js' as described in option 1), and just use the map config to find the nested node_modules dependencies.

I recommend trying for pathway 1), as it also means you are likely to avoid duplicate code if you can flatten all the dependencies in node_modules, and it avoids a bigger config block. But your use case will be the ultimate guide to what path makes sense.

providing a façade to modules

This sounds like another job for the map config. So for a module with ID of 'pluginOne':

requirejs.config({
  map: {
    pluginOne: {
      'brackets/Command': 'plugins/brackets/Command/pluginOne'
    }
  }
  }
});

Then in `plugins/brackets/Command/pluginOne':

define(function(require, exports, module) {
  // This assumes plugin name is the last segment of the ID, which may
  // not be true, just illustrating a generic method to get actual
  // plugin's ID
  var pluginName = module.id.split('/').pop();

  return {
    addCommand: function(...) {
      return require('brackets/command').addCommand(pluginName, ...);
    }
  }
});

I think that module body could be made generic, and you could even programmatically
seed those define() calls in whatever module decides what plugins to load. So before doing the require([]) for the plugin (I assume plugins are all lazily loaded):

var standardDefineWrapper = function(require, exports, module) {
  var pluginName = module.id.split('/').pop();
  return {
    addCommand: function(...) {
      return require('brackets/command').addCommand(pluginName, ...);
    }
  }
};

define('plugins/brackets/Command/' + pluginName, standardDefineWrapper);
require([pluginName]);

cross-extension require

I would avoid different module loader contexts and instead lean on the map config, as it allows better reuse of the module cache.

[dangoor]

Thanks for the quick and detailed reply, @jrburke.

Ahh, I hadn't thought of just calling define like that to create synthetic variations of the modules. Very interesting.

Your suggestion is that rather than having multiple contexts as we do now, we basically just have one top-level require that handles loading for all of Brackets and its extensions. Is it possible to change/augment the map after initial configuration? Most of Brackets fires up and then extensions are loaded after that.

I assume there's no real concern about the map being kind of large? The brackets repo currently has almost 2,000 .js files in src.

[jrburke]

requirejs.config() calls can happen at any time, and the configs are merged (most recent config wins), up to about two levels deep. Very deep module config for example likely does not get a merge, more of a replacement, but map config is merged.

So as long as you call requirejs.config() before any loading starts of a module that depends on that config, then it should be fine.

As far as size concerns: I would not expect it to be an issue, as the storage of the module exports are likely the bigger cost, although if you have a large map config such that every js file has 2000 entries in a map config, I am sure there would be costs.

On performance, I also try to be efficient internally: try to avoid cycling over arrays for example, but convert any array config to hashmap/object property lookup style configs.

Performance is an area where I would definitely want to fix bugs or take pull requests if something was found to be inefficient or slow.

You could try module contexts, but I believe it would mean having to do more manual copying over of exports from one context to another if you wanted to reuse those exports across contexts.

[dangoor]

I'm pretty sure we can avoid mapping all 2000 js files to each other JS file 😁

Thanks for the details. I think my next step would be to try with a single Require context and see how that works out!

[jrburke]

Oh, I should have mentioned this before: I would use the latest requirejs or cajon, as it has some small fixes around node-style packages and their internal module resolution.