Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow selective compilation of packages and modules within node_modules. #9771

Merged
merged 12 commits into from Mar 27, 2018

Conversation

@benjamn
Copy link
Member

@benjamn benjamn commented Mar 26, 2018

The problem

Ever since Meteor 1.3, we have assumed that code published to npm and installed inside node_modules should be consumed by Meteor without any further compilation.

However, as the comments in meteor/meteor-feature-requests#6 make clear, not every npm package author compiles their code for full compatibility with older browsers, and there are cases where you might like to use Meteor compiler plugins to compile resources installed in node_modules, even if that's not something the package author intended.

A blanket policy of recompiling all packages in node_modules would be extraordinarily expensive, not to mention error-prone, because no single Babel configuration can safely process the immense variety of npm package code. Not even if you use the package's own .babelrc configuration, as the Parcel project has discovered: parcel-bundler/parcel#13

The obvious alternative to compiling everything or nothing is to let the developer specify which packages should be compiled, but that approach increases the configuration burden of Meteor applications and steepens the learning curve for Meteor developers, which is something we take very seriously. Meteor was a "zero-configuration" build tool long before that terminology was hip.

With that background in mind, we are proud to offer a technique for selectively recompiling packages and modules within node_modules that is simultaneously easy to use, efficient, and flexible enough for any conceivable use case.

Our solution

Because Meteor refuses to compile code within node_modules by default, the trick is to expose npm package code outside of node_modules using symbolic links, so that the code will be compiled as part of your application, using whatever compiler plugins you have installed.

This trick has worked for a long time, but it was never really satisfying, because you'd have to import the exposed code using a relative import like import stuff from "./imports/linked/the-package" rather than just doing import stuff from "the-package".

Even if you were very careful to stick to the relative import style, you couldn't control the way other npm packages imported the-package, so this approach was always doomed to failure—until now!

This pull request makes both of these import styles work in exactly the same way.

Technically speaking, if two modules share the same fs.realpath, but one of them is compiled by Meteor, this pull request ensures that both modules will use the compiled code.

In other words, if Meteor compiles some code as part of your application, you can create symbolic links to that code elsewhere in your application (including inside node_modules), and the same compiled code will be used everywhere.

Packages that need modification before compilation

If you want to use an npm package that needs to be recompiled, but you have to make some modifications to the package source code before the compilation happens, first clone the package repository somewhere in your application:

cd path/to/application
mkdir -p imports/linked
cd imports/linked
git clone git@github.com:acornjs/acorn.git
rm -rf acorn/.git
git add acorn

Tip: you can use git submodules to include other git repositories in your application, instead of committing the clone.

Now you can edit the package however you like, perhaps by replacing the main field of acorn/package.json with src/index.js instead of dist/acorn.js, or adding a custom .babelrc file to the root of the package.

Once you've modified the package to your liking, simply npm link it into your node_modules directory:

cd path/to/application
meteor npm link imports/linked/acorn

The npm link command works by creating a symbolic link from node_modules/acorn to the package source directory (imports/linked/acorn).

Thanks to this linkage, import { parse } from "acorn", require("acorn"), and import("acorn") will resolve to code within node_modules/acorn/..., but that code will be compiled by the Meteor, because it was also exposed via imports/linked/acorn/....

If you accidentally require("/imports/linked/acorn") instead of just require("acorn"), that's fine, because the Meteor module system conveniently aliases the modules together. To see how this aliasing works, try checking the following relationships:

require.resolve("/imports/linked/acorn") === require.resolve("acorn")
require("acorn").parse === require("./imports/linked/acorn").parse

Packages that only need to be compiled

If you want to use an npm package that isn't adequately compiled for older browsers, but you don't need to make any modifications to the package, that's even easier!

Instead of using npm link to create a symbolic link from node_modules/... to somewhere else in your application, first npm install the package into node_modules as usual, then create a symbolic link to the installed package from somewhere else in the application:

meteor npm install lodash-es
mkdir imports
cd imports
ln -s ../node_modules/loash-es .

As you might imagine, lodash-es makes liberal use of ECMAScript import and export declarations, which need to be compiled in most JS environments. Because imports/lodash-es is now compiled as Meteor application code, code within node_modules/lodash-es/... will also be compiled, and can be imported as usual.

In some sense, this is the opposite of the previous example, because you're making a symbolic link to node_modules/lodash-es instead of making node_modules/lodash-es a link to somewhere else, but the common idea is the same: every module inside lodash-es shares the same fs.realpath with a module that was compiled by Meteor, so the direction of the symbolic linkage doesn't really matter. The module will be compiled by Meteor regardless of how you import it.

Packages that only need, like, one file to be compiled

Just as you can make a symbolic link between directories in node_modules and directories elsewhere in your application, you can also make symbolic links to individual files or subdirectories:

meteor npm install some-npm-package
mkdir -p imports/linked
cd imports/linked
ln -s ../../node_modules/some-npm-package/that/file/with/class/syntax.js .

If that/file/with/class/syntax.js is the only file in some-npm-package that needs to be recompiled, you can compile it by exposing it as imports/linked/syntax.js without compiling any of the other modules in the package.

tl;dr

To compile specific npm packages, use symbolic links to expose node_modules code within your application, outside of node_modules. Meteor will compile the exposed code as if it was part of your application, using whatever compiler plugins you have installed, and also guarantee that you get the compiled code when you import from node_modules (this is they key new feature).

benjamn added 9 commits Mar 24, 2018
Now that symlinks can be used to enable selective compilation of
node_modules, it's important to preserve them.
Once this logic is established, the install.js library will no longer need
to know anything about module.useNode():
https://github.com/benjamn/install/blob/6412f4aabbb44501ad557f578d8ab39f78f22d2c/install.js#L322-L325
If a package in node_modules needs to be compiled for older browsers,
simply symlink the package directory into your application somewhere, and
then import the package as you normally would.

Because of the symlink, code within the package will be compiled as if it
was part of your application, and any imports that refer to modules in the
package will automatically use the compiled code rather than the raw code
from node_modules.

Note that you can also symlink individual files to make them be compiled
like application modules, rather than linking an entire package directory.

Creating symlinks could be considered a form of configuration, but
otherwise this is a zero-configuration solution to selectively compiling
packages within node_modules, which has been something of a holy grail in
the JavaScript community lately.

meteor/meteor-feature-requests#6
Copy link
Member

@abernix abernix left a comment

Just one flag, but LGTM!

Thank you for the excellent compartmentalization of commits within this PR!

// Although the options.from directory should probably be a
// node_modules directory, the only essential precondition here is
// that the destination directory is a node_modules directory.
// assert.strictEqual(files.pathBasename(options.from), "node_modules");

This comment has been minimized.

@abernix

abernix Mar 27, 2018
Member

Just flagging this commented out line: The comment explains the situation, but not sure if you intended to leave as historical evidence of what "once was".

This comment has been minimized.

@benjamn

benjamn Mar 27, 2018
Author Member

Yeah, I decided it to leave it there because it makes the comment more concrete. Also, we might want to reinstate the assertion at some point in the future, once we're confident there shouldn't be any more node_modules1 directories hanging around.

Copy link
Member

@hwillson hwillson left a comment

This looks awesome @benjamn! Making sure this works with npm linked and modified packages is such a great value add as well. I agree, LGTM! 👍

benjamn added 2 commits Mar 27, 2018
@benjamn
Copy link
Member Author

@benjamn benjamn commented Mar 27, 2018

I'm worried that the description for this PR is overcomplicated, and might obscure the elegance of this feature, so I did my best to simplify and clarify the explanation in History.md.

@benjamn benjamn merged commit c30bdbc into devel Mar 27, 2018
19 checks passed
19 checks passed
CLA Author has signed the Meteor CLA.
Details
ci/circleci: Clean Up Your tests passed on CircleCI!
Details
ci/circleci: Docs Your tests passed on CircleCI!
Details
ci/circleci: Get Ready Your tests passed on CircleCI!
Details
ci/circleci: Isolated Tests Your tests passed on CircleCI!
Details
ci/circleci: Test Group 0 Your tests passed on CircleCI!
Details
ci/circleci: Test Group 1 Your tests passed on CircleCI!
Details
ci/circleci: Test Group 10 Your tests passed on CircleCI!
Details
ci/circleci: Test Group 2 Your tests passed on CircleCI!
Details
ci/circleci: Test Group 3 Your tests passed on CircleCI!
Details
ci/circleci: Test Group 4 Your tests passed on CircleCI!
Details
ci/circleci: Test Group 5 Your tests passed on CircleCI!
Details
ci/circleci: Test Group 6 Your tests passed on CircleCI!
Details
ci/circleci: Test Group 7 Your tests passed on CircleCI!
Details
ci/circleci: Test Group 8 Your tests passed on CircleCI!
Details
ci/circleci: Test Group 9 Your tests passed on CircleCI!
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
@benjamn benjamn mentioned this pull request Mar 27, 2018
@devongovett
Copy link

@devongovett devongovett commented Mar 30, 2018

FYI we're considering adding support for a source field in package.json to enable selective compilation of symlinked node_modules, and aliasing of files to replace compiled versions with source. Would be curious to hear your feedback - maybe we can come up with a standard that many tools can adopt! parcel-bundler/parcel#1101

benjamn added a commit that referenced this pull request Apr 5, 2018
Although there was a comment in this code about not applying .meteorignore
files to the contents of node_modules directories, I'm pretty sure that
was the wrong decision, because .meteorignore merely limits what Meteor
tries to compile as application code, and does not actually modify or hide
node_modules files from other parts of Meteor (or Node).

In other words, there's no harm in letting .meteorignore apply to
node_modules, and there may be a LOT of benefit, because it allows the
developer to fight back when compilation descends unexpectedly into an npm
package that contains non-.js[on] files for which a compiler plugin has
been registered, an obscure but not uncommon behavior originally intended
to allow importing CSS assets from npm packages:

* #6037
* 43659ff
* a073280
* #5242
* #6846
* #7406

However, we now have a much more powerful tool for selectively compiling
specific npm packages: #9771. In light of this new approach, we should
probably remove the promiscuous node_modules compilation behavior
altogether, as it might speed up rebuild times for many applications whose
developers don't know or care that this behavior exists. For example,
abandoning this behavior would prevent the problem reported here:
#6950

In the meantime, this commit makes .meteorignore work for node_modules.
benjamn added a commit that referenced this pull request Apr 5, 2018
…es. (#9800)

Although there was a comment in this code about not applying .meteorignore
files to the contents of node_modules directories, I'm pretty sure that
was the wrong decision, because .meteorignore merely limits what Meteor
tries to compile as application code, and does not actually modify or hide
node_modules files from other parts of Meteor (or Node).

In other words, there's no harm in letting .meteorignore apply to
node_modules, and there may be a LOT of benefit, because it allows the
developer to fight back when compilation descends unexpectedly into an npm
package that contains non-.js[on] files for which a compiler plugin has
been registered, an obscure but not uncommon behavior originally intended
to allow importing CSS assets from npm packages:

* #6037
* 43659ff
* a073280
* #5242
* #6846
* #7406

However, we now have a much more powerful tool for selectively compiling
specific npm packages: #9771. In light of this new approach, we should
probably remove the promiscuous node_modules compilation behavior
altogether, as it might speed up rebuild times for many applications whose
developers don't know or care that this behavior exists. For example,
abandoning this behavior would prevent the problem reported here:
#6950

In the meantime, this commit makes .meteorignore work for node_modules.
benjamn added a commit that referenced this pull request Apr 18, 2018
This functionality was originally intended to allow importing
compiled-to-JS modules from `node_modules`, by precompiling any modules
found in top-level npm packages, as long as a Meteor compiler plugin was
registered for the module's file extension.

As discussed in #9800, this extra compilation burden can be non-trivial,
especially if you happen to install an npm package such as `less`, which
contains hundreds of `.less` files in the `node_modules/less/test/`
directory.

More generally, this functionality was an early attempt to enable
selective compilation of `node_modules` directories, but it was not a good
solution to that problem, because in almost all cases the extra
compilation was unwanted.

Meteor 1.7 (formerly known as 1.6.2) will give full control over selective
compilation of `node_modules` back to the application developer (#9771),
which should afford a much better solution to this problem. If you've
installed some `.less` or `.scss` or `.ts` files from npm into your
`node_modules` directory, just create a symlink to the package directory
within your application, and those modules will be compiled and become
importable by other JS modules, as if they were part of the application.
benjamn added a commit that referenced this pull request Apr 18, 2018
…#9825)

This functionality was originally intended to allow importing
compiled-to-JS modules from `node_modules`, by precompiling any modules
found in top-level npm packages, as long as a Meteor compiler plugin was
registered for the module's file extension.

As discussed in #9800, this extra compilation burden can be non-trivial,
especially if you happen to install an npm package such as `less`, which
contains hundreds of `.less` files in the `node_modules/less/test/`
directory.

More generally, this functionality was an early attempt to enable
selective compilation of `node_modules` directories, but it was not a good
solution to that problem, because in almost all cases the extra
compilation was unwanted.

Meteor 1.7 (formerly known as 1.6.2) will give full control over selective
compilation of `node_modules` back to the application developer (#9771),
which should afford a much better solution to this problem. If you've
installed some `.less` or `.scss` or `.ts` files from npm into your
`node_modules` directory, just create a symlink to the package directory
within your application, and those modules will be compiled and become
importable by other JS modules, as if they were part of the application.
@coagmano
Copy link
Contributor

@coagmano coagmano commented May 6, 2018

Is there a recommendation for how to store these symlinks in git and share them with devs working on Windows?
Last time I tries to work with symlinks between both OS it was a real pain

@benjamn benjamn mentioned this pull request May 28, 2018
@kashifnazar
Copy link

@kashifnazar kashifnazar commented Jul 4, 2018

I am trying to run my meteor app on IE11, which doesn't support ES6. Some of the node_modules need to be transpiled for this. I have just added a third symbol link inside my imports directory but when I run the meteor run command, I get the error saying too many symbolic links encountered, stat 'D:\Work\git\meteor_app\imports\utf-8-validate'. Is there a limit of two symbol links that meteor works with?

@pmcochrane
Copy link

@pmcochrane pmcochrane commented Jul 15, 2018

@benjamn can you please confirm what version of meteor has this pull request merged. The history.md doc indicates v.NEXT but I'm assuming it is in the 1.7 release?

I've spent the last week trying to figure out why my app works perfectly fine in development but fails miserably when trying to deploy with angular AOT. The @ng-bootstrap npm is distributed in ES6 and I get a completely different problem when trying to import the compiled bundled version (which I also cannot resolve).

Thanks for your efforts in trying to help us mere mortals.

@tafelito
Copy link

@tafelito tafelito commented Aug 16, 2018

I'm running an app with Meteor 1.7.0.1 and I found that when I use the symlink solution there is something not working. The app itself does build and doesn't complain about the missing modules but the source code of the module doesn't seem to be transpiled. On the other hand, when I clone the repo of the lib I'm trying to use, transpiled with babel and then linking it, it does work, but only if I link the lib from outside my meteor app. If I clone the repo inside imports/linked/... then meteor does not compile, it throws error While processing files with ecmascript (for target web.browser):

Is there anything that has to be done in order to have this working?

@awatson1978
Copy link
Contributor

@awatson1978 awatson1978 commented Aug 27, 2018

Bravo! Fantastic work!

lorensr added a commit to meteor/guide that referenced this pull request Oct 14, 2018
* Update using-npm-packages.md

Add documentation for new feature in PR #9771 and Feature #6 released in Meteor 1.7 and some general clean up.

meteor/meteor#9771

* Update using-npm-packages.md

Found it confusing to introduce the name of the package there—seemed like a typo.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

9 participants
You can’t perform that action at this time.