Support lazy inputFile.addJavaScript for substantial (re)build time savings. #9983
Conversation
…resources. One limitation of Meteor's current compiler plugins system is that every file we *might* use must be compiled before we know whether it *will* be used by the application (which is something we only find out when we later run the `ImportScanner`). More specifically, when inputFile.addJavaScript is called, any information relevant to the current file must already have been computed, even if the file will never be used. This commit begins the process of supporting a lazy version of the `inputFile.addJavaScript` method. For consistency, this interface is supported by other methods like `inputFile.addStylesheet`, though it may not make as much sense for non-JavaScript resources. In this very basic initial implementation, the `lazyFinalizer` function is called immediately, so that subsequent code can keep pretending that all relevant information was eagerly provided. The next step will be waiting to call `lazyFinalizer` until the last possible moment, so that we can skip a potentially huge amount of unnecessary compilation time.
If you're subclassing `CachingCompiler` or `MultiFileCachingCompiler`, you can now implement a `compileOneFileLater` (emphasis on `Later`) to opt into the new lazy compilation strategy. If you implement this method, and `inputFile.supportsLazyCompilation` is true, then the `addCompileResult` will not be called, though it is probably a good idea to keep any existing `addCompileResult` methods, just in case `inputFile.supportsLazyCompilation` is not truthy. This will be an important part of a proper solution to the issues I described (but failed to fix) in my broken PR #9968.
We should really update to the latest version of the less npm package (3.0.4 at the time this commit message was written).
Now that compilation of compile-to-CSS files in imports/ and node_modules/ is actually lazy, we can safely call compileOneFileLater for all inputFiles without worrying about accidental compilation.
| data: result.css, | ||
| sourceMap: result.sourceMap, | ||
| }; | ||
| }); |
benjamn
Jun 12, 2018
Author
Member
If you're already using CachingCompiler or MultiFileCachingCompiler to implement your compiler plugins, you should implement a new method called compileOneFileLater that works roughly like this.
If you're already using CachingCompiler or MultiFileCachingCompiler to implement your compiler plugins, you should implement a new method called compileOneFileLater that works roughly like this.
| ).then(result => { | ||
| Object.assign(this._initialOptions, result); | ||
| this._finalizerPromise = null; | ||
| })).await(); |
benjamn
Jun 12, 2018
Author
Member
It's too bad this implementation has to use Fibers to await the result of the lazyFinalizer function, but that's an implementation detail we can revisit in future versions of Meteor without changing the API.
It's too bad this implementation has to use Fibers to await the result of the lazyFinalizer function, but that's an implementation detail we can revisit in future versions of Meteor without changing the API.
| @@ -825,14 +818,11 @@ export default class ImportScanner { | |||
| // Set file.imported to a truthy value (either "dynamic" or true). | |||
| file.imported = forDynamicImport ? "dynamic" : true; | |||
|
|
|||
| if (file.error) { | |||
| if (file.reportPendingErrors && | |||
| file.reportPendingErrors() > 0) { | |||
benjamn
Jun 12, 2018
Author
Member
This tends to be the first place the ImportScanner forces full compilation of file.
This tends to be the first place the ImportScanner forces full compilation of file.
| @@ -249,19 +249,6 @@ export default class ImportScanner { | |||
| // something plausible. #6411 #6383 | |||
| const absPath = pathJoin(this.sourceRoot, file.sourcePath); | |||
|
|
|||
| const dotExt = "." + file.type; | |||
| const dataString = file.data.toString("utf8"); | |||
benjamn
Jun 12, 2018
•
Author
Member
Accessing file.data here (in addInputFiles) would have forced every file to be fully compiled, regardless of whether it was scanned.
Accessing file.data here (in addInputFiles) would have forced every file to be fully compiled, regardless of whether it was scanned.
|
@pagesrichie These changes will be available starting in 1.7.1-beta.1, which should be published in the next half hour or so. As long as packages and application code are compiling In general, we merge PRs into Simple! |
|
I just saw the METEOR@1.7.1-beta.1 release on the https://github.com/meteor/meteor/releases page. I will go ahead and update my meteor and hopefully it fixes the issue with https://github.com/Semantic-Org/Semantic-UI-Meteor files compilation! And @benjamn yes that package is dependent on the meteor core less package. So I will try it out with this new meteor release and let you know. |
|
@benjamn Not sure where to post this.. but there's an issue with the 1.7.10beta.2 release, getting an error here when the less package is enabled when I have my less files in the client dir, i even disabled semantic-ui package and if it has any less files to parse it crashes. (To re-enact, I added all my semantic-ui client files somewhere inside the client dir and then then simply enable less package or the less@2.8.0-beta171.2 package (that automatically got upgraded to when upgrading to this meteor release, and it just crashes with the following error): /.meteor/packages/static-html/.1.2.2.wp43cd.99etr++os+web.browser+web.cordova/plugin.compileStaticHtmlBatch.os/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:190 TypeError: First argument must be a string, Buffer, ArrayBuffer, Array, or array-like object. |
@pagesrichie In general, the best place to post it would be in a new issue. Though you're right in your identification of this issue as a suspect. The problem stems from this code: meteor/tools/isobuild/compiler-plugin.js Lines 921 to 924 in 0ca6202 ...and a result of I suspect that meteor/packages/less/plugin/compile-less.js Lines 51 to 55 in 0ca6202 ...possibly because of a compilation failure or potentially just because of a completely empty file (I'm not sure what We should probably guard against this within this I'll defer to @benjamn since this is fresh on his mind. |
|
@pagesrichie @abernix That problem should be fixed by #9998. In short, we were applying the CSS source map to a JS module that dynamically appended the CSS to the |
I should have paid more attention to @abernix's analysis here, as it was exactly right: #9983 (comment) cc @pagesrichie
More info on lazy compilation here: meteor/meteor#9983
More info on lazy compilation here: meteor/meteor#9983
More info on lazy compilation here: meteor/meteor#9983
More info on lazy compilation here: meteor/meteor#9983
When I implemented support for the "module" entry point in package.json files for client code in #10541, I modified PackageSource#_findSources to include files found in node_modules that need to be compiled, but my implementation considered only "local" node_modules directories, like the one in the application root directory, while neglecting the private .npm/package/node_modules directories that many Meteor packages have. This commit includes .npm/**/node_modules when _findSources is scanning a Meteor package, which should solve issues like #10544, where a Meteor package imports an npm package that was installed with Npm.depends, and that npm package has a "module" field in its package.json file, pointing to an ESM entry point module, but the ESM syntax was not appropriately compiled, leading to parse errors like "Unexpected token export". Before lazy compilation was introduced in Meteor 1.7 (#9983), including the node_modules directories of Meteor packages would likely have been a big problem for build performance, since there would be that many more modules to compile. It's still worth making sure this change doesn't regress build performance for other reasons, but I'm reasonably confident lazy compilation will save us here, unless there are just too many npm packages installed via Npm.depends that export ESM modules.
In Meteor 1.7.0.1, multiple developers noticed that the
lesspackage takes longer than expected to compile.lessfiles (#9957). There are a number of potential reasons for this deviation, but by far the most important reason is that Meteor compiler plugins in general (not just thelesspackage) are expected to compile every file before Meteor has determined (in theImportScanner) whether or not those files are actually used by the application.The original reason for this architecture was that existing build plugins widely used by the community were accustomed to receiving a complete batch of all the files they needed to process, up front, all at once. This was especially important for compiler plugins that support importing other files of the same type, such as
lessandfourseven:scss. In Meteor 1.2, @glasser found a clever way to continue supporting that behavior, and we've stuck with that solution ever since.By contrast, any system that attempts to scan the dependency graph of an application needs to be able to compile one file at a time. Most recently, I was reminded of this limitation when #9968 did not work, because my attempt to skip compiling
.lessfiles found innode_modulesorimportshad the unfortunate consequence of preventing them from being imported later by JavaScript modules usingrequireorimport.For a long time, I thought these two approaches were irreconcilable. Either Meteor would have to maintain backwards compatibility despite the performance cost, or we would have to overhaul the compiler plugin system and deprecate existing plugins.
As this PR demonstrates, it turns out we can have it both ways. When calling
inputFile.addJavaScript(or any otheraddWhatevermethod, such asaddStylesheet), a compiler plugin may now provide an initial set of options (such as thepathof the file) as well as a lazy finalizer function that can be called to finish compilation—but only if the file is actually used by the application.Here's what that looks like in the
babel-compilerpackage:In other words, compiler plugins still receive a complete list of all files they should process, in case they depend on that behavior, but they have the option of postponing costly compilation until later.
Note that the compiler plugin system has the freedom to invoke the callback as soon as it likes (even immediately, as I initially implemented it, just to check my sanity), though in practice the
ImportScanneris careful to avoid triggering the callback until it encounters the file in the dependency graph.As a demonstration of the impact of this change, here's a fresh build of a new
meteor create --fullapplication without this optimization:and here's the same profiling output with this optimization enabled:
If you do the math, the average time per file is💇 💇♂️
approximately the samesometimes more, sometimes less; but the lazy version always wins because it compiles less than half as many files.If you're a compiler plugin maintainer (cc @sebakerckhof @GeoffreyBooth et al), please take note of this new API, and especially the
inputFile.supportsLazyCompilationfeature detection (so you don't have to stop supporting older versions of Meteor).If you're an application developer, you won't see these benefits until you update to Meteor 1.7.1, but we hope this gives you an additional reason to help test the next 1.7.1 beta release.