Convert to Webpack Plugin

[ericclemmons]

I just found out about a similar project:

https://github.com/renke/auto-install-webpack-plugin

From what I can gather, this may be a better solution than what this project is doing.

Let's see if I can work up an overview (please check this @renke!)

• npm-install-loader:
  • Is a loader, intended to be used during the processing of a JS file.
  • Because of this, it only has access to the source, map, and webpack configuration.
  • As such, it actually parses the AST to determine which dependencies need installation.
  • It operates synchronously via child_process.spawnSync to ensure the thread & build is blocked until installation completes & any other loaders (e.g. eslint-loader, style, etc.) take over.
  • Unknown how this would affect .css files (e.g. import "~bootstrap"), which was next on the punch list.
  • Plans to support all CLI flags (Use --save or --save-exact flags when installing based on loader query config #3).
  • Because it operates per-file, if there are 10 missing dependencies in that single file, they're all installed at once.
  • How does this contrast with a plugin?
• auto-install-webpack-plugin:
  • Is a plugin, which means it has access to the compiler and a great deal more.
  • Hooks directly into the module resolver, meaning it can do an async installation.
  • Looks to hook into all module resolution, including those of stylesheets.
  • Does it block the build?
  • Does it install one dependency at a time?
  • How does this contrast with a loader?

I'll await feedback, but my initial thoughts are to:

1. Complete any outstanding issues.
2. Test out & send a PR to auto-install-webpack-plugin for similar behavior.
3. Deprecate this project & link users to it instead.

Frankly, this was to scratch an itch and @renke's project didn't show up in my results, but I'm not a fan of competing for the sake of competing.

[ericclemmons]

The plot thickens:

https://twitter.com/rygu/status/682133402368274432

Maybe a Babel plugin akin to Philip's?

[latentflip]

Here's rough-and-ready source for a babel-plugin https://gist.github.com/latentflip/966b86ffa06f3f894050

I think the downside of a babel-plugin, as against a webpack plugin, is that at the babel stage, we're not aware of what webpack is doing - so if it's got module aliases etc, I don't think that we'll know about them, so we might try and install "modules" which are actually being aliased to something else by webpack.

[latentflip]

I think a loader (as against a webpack plugin) suffers from the same problem I mentioned above.

[latentflip]

Sorry to spam your issues 😬 but I see now that even the plugin suffers from this problem.

Specifically what I am looking at is react-heatpack, it has a few aliased modules that it inserts into the entry file, which are replaced by module.alias in the webpack config to point to an actual file on disk. These aren't real npm modules and so shouldn't be installed, however, as far as I can tell, all of our modules will try and install them anyway.

I would have thought the webpack plugin, would have avoided this problem, since the aliasing is part of webpack, but it doesn't seem to. Will keep digging.

[latentflip]

Okay, after prodding webpack plugins for as long as I can stand, I can't see how to actually get a module resolver to run after everything else, to deal with the problem of trying to install aliased/otherwise resolved modules. If it's doable at all, and is a consideration (maybe it's just not really that critical for the type of thing we're talking about), it feels like a plugin is the only way to handle it, but I think it'll take someone with more webpack internals knowledge than I to figure it out.

Again, sorry for the comment thinking-out-loud :)

[ericclemmons]

@latentflip Actually, you make great points! If you check the source, you'll see that I too use Webpack-specific features (e.g. resolve.root and resolve.moduleDirectories).

This alone may be the reason for it existing.

It is probably worth having both, though, if someone goes "all in" on Babel plugins:

https://www.npmjs.com/package/babel-plugin-module-alias

[latentflip]

Yeah I saw that you were using webpack stuff (which is good), my gut was guessing that if we need to do full on resolving (via plugins) to determine whether a module is a valid and missing npm package, we probably have to do that during plugin time, rather than loader time.

But that's just a gut feeling from prodding, and this feels like not the main consideration anyway :)

On 30 Dec 2015, at 17:23, Eric Clemmons notifications@github.com wrote:

@latentflip Actually, you make great points! If you check the source, you'll see that I too use Webpack-specific features (e.g. resolve.root and resolve.moduleDirectories).

This alone may be the reason for it existing.

It is probably worth having both, though, if someone goes "all in" on Babel plugins:

https://www.npmjs.com/package/babel-plugin-module-alias

—
Reply to this email directly or view it on GitHub.

[ericclemmons]

I think you & have share the same sentiments. The problem with the loader
is that it ends up having to re-implement some webpack features around
module resolution, which is not ideal at all.

It would require some testing or someone in the know to answer if loader &
plugin are largely intercheable. Frankly, I just don't know myself. Being
able to leverage "alias" is necessary IMO.

[ericclemmons]

I've been testing out ways to solve Babel's rigidness compared to Webpack's flexibility (but configuration overload), and decided that it'd be best to, once a few PRs are merged, to take combine the effort with @latentflip's https://gist.github.com/latentflip/966b86ffa06f3f894050 prototype here:

ericclemmons/babel-plugin-npm-install#1

I'll be working out a blog post or something about the effort sometime, but experience is pushing me towards:

• Babel manages transpilation of all forms. It's better suited for AST parsing & conversion.
• Webpack for bundling, building, & optimizing.

This alone will greatly simplify things, IMO, as it decentralizes the documentation problem from Webpack to individual Babel plugins.

[ericclemmons]

On a side note, this ultimately skirts the need to determine if this should be a plugin or not.

[insin]

I see that webpack Compiler objects have createChildCompiler() and runAsChild() methods, which I assume would to play a part in using webpack's own machinery to determine if a module can be resolved using all the tweaks webpack allows you to apply to that process, but I have no idea how to even begin determining if that's possible looking at the webpack codebase

You would just want to use the "can I find this thing?" part rather than going ahead and doing anything with it, which again I don't know where to start with.

auto-install-webpack-plugin doesn't currently handle this either because it uses substack/node-resolve to check if a module can be resolved from the path it's being requested from.

Something interesting I found while digging is that extract-text-webpack-plugin creates a child compiler inside its loader: https://github.com/webpack/extract-text-webpack-plugin/blob/d34d0fb60b9522532f4406ddb255ea95b5c92fe7/loader.js#L46

[ericclemmons]

Also, the primary limitation of a babel plugin is that CSS @imports are kinda outta the question.

[insin]

Useful resource: eslint-plugin-import implements a webpack resolver it uses to check if imports can be resolved: https://github.com/benmosher/eslint-plugin-import/tree/master/resolvers/webpack

[ericclemmons]

Or maybe this should be a Webpack plugin after all?

Just tested out CSS modules and it's possible with some tweaks:

/* layout.css */
@import url("~bootstrap");

converts to:

exports = module.exports = require("./node_modules/css-loader/lib/css-base.js")();
// imports
exports.i(require("-!./node_modules/css-loader/index.js!bootstrap"), "");

Which can be parsed out of the AST if this project is adjusted to do a .split("!").filter(...) to retrieve bootstrap using the following loader sequence:

      {
        loader: "style-loader",
        test: /\.css$/,
      },
      {
        loader: "npm-install-loader",
        test: /\.css$/,
      },
      {
        loader: "css-loader",
        test: /\.css$/,
      },

Or:

{
  loader: "style!npm-install!css",
  test: /\.css$/,
}

[renke]

Hi,

I would say that a Webpack plugin is definitely the best way to go because it in theory it should work for every loader without adding a special loader. I was pretty happy with the result when I wrote auto-install-webpack-plugin but I actually didn't use it myself in larger projects because it failed in some ways I didn't really understand.

I also noticed today that there are some other problems that I didn't encounter when I developed it. All in all the plugin needs a lot of work to be in an acceptable shape (for me it was more like a proof of concept anyway).

Regarding the problem with aliases, I am pretty sure it's possible to hook into the alias mechanism. The webpack documentation on the resolver tappables (see resolvers) is a bit lacking though. I had to do a lot of trial by error to accomplish what I want (which apparently it not even enough).

Currently, my plugin checks if the a module is resolvable (basically by looking in to the file system) when webpacks tells us the module is required somewhere. However, it currently doesn't let other resolvers try to resolve the module (afaik the alias mechanism is one of those resolvers) which means it will try to install the module it can't resolve even though it might actually be resolvable by other resolver plugins.

[ericclemmons]

@renke Thanks for jumping in! I've begun doing some research myself & you're absolutely correct: this isn't an easy problem to solve well, though it's easy to get started.

Would you mind if I forked this project into a plugin (since a lot of the tests & code would come along with it), or would you prefer we put the effort into auto-install-webpack-plugin?

This is something that we've already begun using a lot within my team, so we're eager for it to scale to the complex setups several of our projects already have.

Thanks!

[renke]

@ericclemmons sure, go ahead! I am happy as long as the idea of my plugin lives on.

[ericclemmons]

Alright, I'm combining this, #2, #5, #11, & #12 together and working on this today.

If you'd like to follow-along, #2 is where most of the effort will be taking place, since it's the end-user experience.

The ideal is that the following are supported:

• import ... from ...
• require(...)
• require(...).polyfill();
• express.use(require(...))
• @import url(...); (CSS)

[ericclemmons]

Also, this will mean renaming the project:

• Rename repo to npm-install-webpack-plugin.
• Rename name in package.json to npm-install-webpack-plugin.

[ericclemmons]

Awesome. Apparently compiler.resolvers.normal.plugin("module") doesn't hook into anything for resolving. Thanks Webpack.

[ericclemmons]

@bebraw Alright, this is about as done as it can be for now.

I'll record another video (none of the current ones were too my liking) tomorrow.

It's pretty darn incredible IMO having this stuff "just work" while you code. Especially if you do:

1. import express from "express";
2. -S
3. app.use(...).get(...).etc.
4. app.listen(3000)

By the time you finish using the dependency, it's already been installed, saved, & reloaded :)

[bebraw]

@ericclemmons Ok, I'll blast it once you merge the PR and publish a new version. 😄