Exclude specific files to prevent build to fail with "FATAL ERROR: CALL_AND_RETRY_LAST" #4055
Comments
|
We crossed some threshold as well (in uglify I would guess) and in the heat of battle started using the following:
HTH
|
|
@jschilli Thank you for the hint :) but there is something really odd... because --max-old-space-size=2400 is still too less. --max-old-space-size=5000 results in a never ending script (It runs since you posted the answer). Hope the issue can be resolved. It is quite urgent because we wanted to release a new version of our app today :p EDIT: now, after more than 4 hours the build failed with --max-old-space-size=5000 with the same error... any ideas how we could track down this issue? |
|
Okay, I was able to trace down the problem. We are using code which is compiled for multiple platforms. The source code is C++ and is transpiled to emscripten (Javascript). The emscripten files are quite large and we load it on demand. But we put it into the assets folder to have it in "application cache manifest". When I remove the emscripten files, the build makes no problem and we can build without the --max-old-space-size Whats happening with the files which are located in public/assets while building the ember-cli app in production mode? Is there any way to tell ember-cli to not do things with the emscripten files, which crashes the build? Moving the files from the assets folder to another folder would be an option but it's not the best option for us, because there are many people involved who would have to update their paths. Any ideas how we could resolve this issue? |
|
I would imagine that uglify is complicit here. see http://www.ember-cli.com/asset-compilation/ http://www.ember-cli.com/asset-compilation/ for how to disable minification. if that works you can then evaluate how to exclude the emscripten file from uglification
|
|
@jschilli yes it is uglification. But I don't want to turn off uglification for all of my files... Right now I didn't find any way to exclude just the emscripten files from uglification. Has anyone an idea? |
tschoartschi
changed the title
|
We still don't have a good solution... now we just put the emscripten files into a directory which ember-cli don't know (we call it emscripten_components) and for deployment we need to copy these files manually. This doesn't feel like a real solution but it works for now. If someone has a better suggestion let me know. Meanwhile I opened a thread on disucss.ember.com because I think our problem is not really an issue with ember-cli. |
|
what version of node are you using on jenkins server |
|
We are on 0.12.1 IIRC
|
|
cc @chadhietala this will likely be something the packager work will enable nicely |
|
@stefanpenner we are on node 0.12.2, but an option to exclude specific files from the build (especially uglification) would be great 👍 For us, minifying or uglifying the emscripten files again makes no sense. In our case it just slows down the build process and now even made it impossible. |
leave them in |
|
in theory adding file exclude/include support to https://github.com/ef4/broccoli-uglify-sourcemap/blob/master/index.js would also help fix this. |
|
@stefanpenner leaving them in public (and not put them into the assets folder) seems to work 👍 I'll check that in more detail tomorrow :-) |
👍 Awesome, but like I said, i think the packager work will expose more flexibility to this part of the build. Stay tunned |
|
@stefanpenner I was to enthusiastic yesterday. I think I made a typo when trying to run "ember build --environment production". Just putting the emscripten files outside of the assets folder didn't help. I implemented an in-repo-addon which does the copying stuff. Folder structure looks as follows:
At first I tried to add some express middleware to serve files from emscripten_components but I didn't managed it to work. So I decided to copy the emscripten into the public folder when I'm in development mode and otherwise copy it into dist folder. If someone needs the same thing or a place to start, here is my in-repo-addon (yes I know it's not a super clean solution, I know it is not generic and yes I know it is just a hack ;-) but it works. If someone has tips or tricks I'm happy to hear from you). var fse = require('fs-extra');
var path = require('path');
var config = require('../../config/environment');
module.exports = {
name: 'emscripten-handling',
src: '',
distDev: '',
isDevelopingAddon: function () {
return true;
},
preBuild: function () {
console.log('[EMSCRIPTEN HANDLING] Warning: make sure there is no other build running!! Check ember serve also');
this.setPaths();
if (!this.isDevelopment()) {
if (fse.existsSync(this.distDev)) {
fse.removeSync(this.distDev);
}
return;
}
if (!fse.existsSync(this.distDev)) {
fse.mkdirsSync(this.distDev);
}
console.log('[EMSCRIPTEN HANDLING] preBuild copy to folders');
fse.copySync(this.src, this.distDev);
},
postBuild: function (result) {
if (this.isDevelopment()) {
return;
}
this.setPaths();
fse.copySync(this.src, result.directory + '/' + this.getEmscriptenAppComponentFolder());
},
setPaths: function () {
var env = process.env.EMBER_ENV;
var actualConfig = config(env);
this.src = actualConfig.APP.EMSCRIPTEN_APP_COMPONENT_FOLDER;
this.distDev = 'public/' + actualConfig.APP.EMSCRIPTEN_APP_COMPONENT_FOLDER;
},
getEmscriptenAppComponentFolder: function () {
var env = process.env.EMBER_ENV;
var actualConfig = config(env);
return actualConfig.APP.EMSCRIPTEN_APP_COMPONENT_FOLDER;
},
isDevelopment: function () {
return process.env.EMBER_ENV === 'development';
}
}; |
|
@stefanpenner Is there any news on this topic? I am experiencing the same trouble. I added a large, already minified js file which causes the ember build to go from two to ninety minutes. |
nothing yet. Although, sounds like you should just move that file into public, and not send it through another minification step. |
|
@stefanpenner Thank you for the reply. I tried moving the file to the public folder, however in a production build it still seems to do some labor intensive things (I guess uglification) to it because the build is still taking about ninety minutes. |
|
@stefanpenner It actually seems to be the compression done by broccoli-uglify-js which is taking so much time. Since it is not supported to disable this for some files, I have disabled this for the whole application in Brocfile.js: var app = new EmberApp({
minifyJS: {
options: {
compress: false
}
}
}); |
we should likely do the following:
|
|
@stefanpenner Thanks a lot for your effort! I wanted to ask if there are any news on this issue? |
+1 |
Sorry not yet |
|
for those interested, come help populate the stress app ember-cli/stress-app#2 |
|
Exclude/include on uglify has become essential for us too, resorting to some mega hacks to get around the lack of exclude. Adding something like tiny mce bumps the build up to about twenty minutes when all that is required is a passthrough copy. Definitely need some way to tell ember that specific files are passthrough only, as otherwise the build gets out of control unnecessarily |
|
I believe the feature only needs to be added to the upstream dependency: https://github.com/ef4/broccoli-uglify-sourcemap Basically just: check opts.blacklist here and skip the minify step if matched. It may be nice to make the blacklist support globs, but not a requirement (that can be added later) I don't think I have time, but it should be relatively easy for someone to land what i describe above. |
|
@stefanpenner thanks for the heads up, that doesn't look too bad. I'm going to spend an hour or two and see what I can come up with. glob or glob-like syntax is probably necessary to make this a useful feature. Does ember-cli/broccoli etc have a preferred glob library? Can I add a dependency for it, or is there an existing way to include that functionality in this module? |
minimatch we use it via https://github.com/joliss/node-walk-sync |
|
Sorry I didn't see your edit until now, probably not too hard to switch it to node-walk-sync but I ended up using minimatch anyway. It seems to work on our build, here is the squashed commit: soxhub/broccoli-uglify-sourcemap@d33fe38 accepts an exclude: [] parameter to minifyJS: {}, an array of globs to check against. I did also discover that another major bottleneck in our build is in using broccoli-funnel to pickFiles, even though all we are doing is a straight copy (of bower_components/tinymce to assets/tinymce)... Funnel seems to be doing its own sort of processing (and tinymce has tons of files in it). Is there a better way to copy an entire directory from bower_components without Funnel? Moving this to a post build addon seems much more performant |
funnel has the ability to merely link in entire trees, i suggest that might be the approach you wan to take. https://github.com/broccolijs/broccoli-funnel/blob/master/index.js#L102-L104 |
|
I also have a POC (nearly ready to primetime – merely lacking support for windows users without symlinks) that dramatically improves performance of funnel even in crazy circumstances hjdivad/broccoli-funnel#2 The work there is meant as the foundation for planned dramatic perf boosts to broccoli-filter broccoli-persistent-filter broccoli-merge-trees etc. |
|
Yeah, I just removed the include: [] parameter from my Funnel() (it was basically collecting everything anyway), and this seems to have sped up the build a lot, in conjunction with the new exclude param on ugilify, build times are normal again. Thanks for the help! Should I submit that commit as a PR or something? TBH I've never really submitted a PR before so not even sure how to go about it lol |
|
@kjhangiani yup PR sounds good. a TL;DR if you have questions can be found here |
|
@stefanpenner thanks, submitted |
|
passing exclude information to uglfier is now fully supported. |
Hey!
We are using ember-cli since almost over a year and it always worked great. Today we encountered an issue when building our app on Jenkins in production environment. It seems we are hitting some memory limit of node.js or more specifically a limit of V8. During the build the error "FATAL ERROR: CALL_AND_RETRY_LAST" is displayed and then the build is aborted. We don't know how to resolve this issue. Maybe you can help us fixing this problem.
The exact output on the console looks as follows:
Our package.json:
The text was updated successfully, but these errors were encountered: