Compressing meteor project takes forever

[mcoenca]

I was used to very fast deployments on modulus but lately my "modulus project deploy" stays in "Compressing project" state before uploading for soooo long. ( > 4minutes)

Am I the only one getting this ?

Meteor 1.1.6
Modulus 5.0.1
Project compressed about 52 MB (Maybe that's the root of the issue :) )

thanks !

[dbnoble]

I am also having this issue. "Compressing project..." for up to 10 minutes and pegging CPU at 100%.

That has only been happening for about a week for me.

[rclai]

Same thing. I think the problem is that the modulus CLI is compressing the node_modules folder for some reason?? Shouldn't this be done on the server being deployed to?

[ashburnham]

Same for me.. takes about 15 minutes.
Meteor 1.2.1
Modulus 5.0.1

Project compressed to 28.6 MB and it's not a large project but it does use a few npm modules..

Other projects take < 2mins..

[ashburnham]

Would be great to merge this - killing me that it takes to long to deploy and sure it's hurting other people's productivity too!

[rclai]

Ah nice.

[fiveisprime]

The real solution to this is to have Modulus create a proper build image for meteor which will do the conversion at build time. I'll get that added to the new feature list.

In the meantime, here's a stopgap: use Demeteorizer@2.3.1 then deploy the generated source.

$ npm install demeteorizer@2.3.1
$ cd /path/to/project
$ demeteorizer
$ modulus deploy .demeteorized

The older version of demeteorizer attempts to get all of your dependencies into a generated package.json. This works most of the time but, because of the way Meteor manages packages, has a tendency to be unpredictable, inconsistent, and broken. That's where demeteorizer@3 comes in. It solves that by have Meteor include its dependencies.

Moving this step to the build pipeline makes a lot of sense so we'll see about making that happen. The problem there is that takes control away from you as you no longer have control over how the build is done. This is an issue if you are using the meteorhacks npm package with compiled modules which won't work with meteor build.

I'll keep this thread updated as we work through this.

[rclai]

Thanks. Also keep in mind that Meteor is going to have a more official support for NPM in Meteor 1.3 which will do away with having to use meteorhacks:npm.

[ecwyne]

@fiveisprime
Is this the best place to stay up to date on any progress you've made?
I'm very interested in seeing improvements on time to deploy. Is there any way we can help?

[cryptoquick]

Maybe the system tgz/gzip tools could be used, if available, for the compression part? For those experimenting with this option, it's kind of ridiculous how long this takes.

[cryptoquick]

Just for anyone who's wondering, it takes over an hour and a half to compress the build and upload a 110mb node_modules directory, in addition to our application code (which is a drop in that bucket). The compressed build was 55mb. So, be warned, if using the -m option.

[lytstephen]

I just upgrade my project to 1.3 and the compressing takes forever now, which was not the case before I updated.

Any updates on this issue, particularly related to deploying meteor 1.3 apps?

[ajschmaltz]

@lytstephen I'm having the same problem. Also, uploading to Modulus fails. Have you been able to successful complete your modulus deployment (even though it takes forever)?

[flean]

I'm having the same issue and it's hit and miss on the deploy. Sometimes it works and sometimes it fails for random reasons.

[jackboberg]

Please try using our new meteor image, and an updated CLI:

npm i modulus@next -g

[charleshan]

Learned my lesson. Always having a staging environment (INCLUDING Modulus).
npm i modulus@next -g does not work.

Note: I'm using 1.3 as well.

[ajschmaltz]

@IamCharlesHan & company - there is a new environment on Modulus.io called "Meteor" (you're probably using NodeJS). Switch to that then reset your server. Finally, redeploy. If you are having jquery warnings on your console remember to go to the root of your project and run npm init before the suggested meteor npm install --save jquery. At least that's what ended up working for me.

[charleshan]

@ajschmaltz That worked for me. Thank you.

[charleshan]

Actually no it didnt. It fails to deploy.

Uploading project...
Upload progress [===================] 100%
Deploying project into Meteor runtime...
Starting build.
Creating directories for build environment.
Downloading source.
Executing build.
Demeteorizing application...
mop, this is your first time using Meteor!
Installing a Meteor distribution in your home directory.
######################################################################## 100.0%

/mnt/home/.meteor/packages/meteor-tool/.1.3.0_3.p3s7qv++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/lib/node_modules/meteor-promise/promise_server.js:116
      throw error;
            ^
Error: ENOSPC, mkdir '/mnt/tmp/.tmpl0uhx2/less-2.5.6/plugin.compileLessBatch.os/npm/node_modules/meteor/babel-runtime/node_modules/regenerator/node_modules/commoner/node_modules/glob/node_modules/minimatch/node_modules/brace-expansion/node_modules/concat-map/example'
Conversion failed.

[flean]

Sorry, wrong issue answer I'm also stuck in hell with this.

On Apr 6, 2016, at 3:17 PM, Charles Han notifications@github.com wrote:

Actually no it didnt. It fails to deploy.

Uploading project...
Upload progress [===================] 100%
Deploying project into Meteor runtime...
Starting build.
Creating directories for build environment.
Downloading source.
Executing build.
Demeteorizing application...
mop, this is your first time using Meteor!
Installing a Meteor distribution in your home directory.
######################################################################## 100.0%

/mnt/home/.meteor/packages/meteor-tool/.1.3.0_3.p3s7qv++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/lib/node_modules/meteor-promise/promise_server.js:116
throw error;
^
Error: ENOSPC, mkdir '/mnt/tmp/.tmpl0uhx2/less-2.5.6/plugin.compileLessBatch.os/npm/node_modules/meteor/babel-runtime/node_modules/regenerator/node_modules/commoner/node_modules/glob/node_modules/minimatch/node_modules/brace-expansion/node_modules/concat-map/example'
Conversion failed.
—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub #66 (comment)
Untracked with Trackbuster https://trackbuster.com/?sig

[ajschmaltz]

@IamCharlesHan - I was slightly frustrated and am by no means an expert, I was just relaying my experience and hoped it helped someone... BUT ... I forgot to mention I also upgraded to Meteor 1.3.1 based on something I read. I don't remember why & I don't know if that matters. I do know that I was having all the issues above and now everything runs smoothly for me.

[charleshan]

@ajschmaltz I think I'm in your shoes as well. I think that worked (for now)... I really appreciate it! Thank you.

[BruceHubbard]

We've been having all kinds of problems as well after upgrading modulus cli versions. We rolled back to a previous version and things are working well now:

sudo npm remove -g modulus
sudo npm install -g modulus@4.0.2

[flean]

@BruceHubbard Are you on Meteor 1.3+ and do you build for mobile platforms? If so what's working for you now. I'm stuck ...

[BruceHubbard]

We're not doing cordova or anything like that, just straight up Meteor web app. We are using 1.3 though. Downgrading the modulus cli seemed to do the trick for us.

[csillag]

We are hitting this one, too.
Meteor 1.3.1, "meteor" runtime on Modulus.io, 5.0.1 version of modulus-cli...
...compression takes forever.

[BruceHubbard]

@csillag Try downgrading your modulus-cli version and see if that fixes it. It's not a perfect fix but it allowed us to deploy again.

[csillag]

@BruceHubbard I already tried that, and it failed.

Compressing project...
2.4 MB written
Uploading project...
Upload progress [===================] 100%
Deploying project into Meteor runtime...
Starting build.
Creating directories for build environment.
Downloading source.
Executing build.
Demeteorizing application...
mop, this is your first time using Meteor!
Installing a Meteor distribution in your home directory.
######################################################################## 100.0%
Upload progress [===================] 100%
build: You're not in a Meteor project directory.

To create a new Meteor project:
  meteor create <project name>
For example:
  meteor create myapp

For more help, see 'meteor --help'.
Conversion failed.
Build attempt failed, attempting again.
Starting build.
Creating directories for build environment.
Downloading source.
Executing build.
Demeteorizing application...
mop, this is your first time using Meteor!
Installing a Meteor distribution in your home directory.
######################################################################## 100.0%
build: You're not in a Meteor project directory.

To create a new Meteor project:
  meteor create <project name>
For example:
  meteor create myapp

For more help, see 'meteor --help'.
Conversion failed.
Build attempt failed, attempting again.
Starting build.
Creating directories for build environment.
Downloading source.
Executing build.
Demeteorizing application...
mop, this is your first time using Meteor!
Installing a Meteor distribution in your home directory.
######################################################################## 100.0%
build: You're not in a Meteor project directory.

To create a new Meteor project:
  meteor create <project name>
For example:
  meteor create myapp

For more help, see 'meteor --help'.
Conversion failed.
Build failed, reverting project.

[Error] There was a problem deploying your project.

[tzmanics]

@csillag have you tried using our new meteor image, and updated CLI:
npm i modulus@next -g

You'll also want to change your runtime image to Meteor on our project's administration page.

I also noticed this error in your logs build: You're not in a Meteor project directory. Sometimes that happens if you don't have have all the correct meteor files in your project's directory.

[csillag]

@tzmanics:

have you tried using our new meteor image,

I created the project yesterday, on the modulus web ui, using the meteor image.

and updated CLI: npm i modulus@next -g

I have tried 4.0.2 and 5.0.1, but not the unreleased 6.0.0. Now I tried that, too. With that, the results are different:

Errors prevented bundling:
While processing files with flemay:less-autoprefixer (for target web.browser):
less/bootstrap.less:1: Unknown import: imports/bootstrap/less/bootstrap.less
less/main.less:4: Unknown import: imports/bootstrap/less/variables.less

Conversion failed.

I only get these errors here, otherwise, the build runs successfully.

You'll also want to change your runtime image to Meteor on our project's administration page.

Isn't this the same thing as you said in your first sentence? ("using our new meteor image")

I also noticed this error in your logs build: You're not in a Meteor project directory. Sometimes that happens if you don't have have all the correct meteor files in your project's directory.

I could use a bit more detail here. What specifically is missing? The project is OK; I can successfully deploy it to (for example) Galaxy.

[csillag]

Also, I always get this:

Installing a Meteor distribution in your home directory.

Why does it do this? I have a perfectly functional Meteor distribution on my path..

[tzmanics]

@csillag if you send an email to help@modulus.io I can look into your specific project.

The first part of installing npm@next updates your Modulus CLI to the newest version that supports the Meteor image. Changing the runtime image selection in your project's dashboard is setting up your project to always use the Meteor image.

It's hard to say what's causing Errors prevented bundling: without seeing the file list but I would double check to make sure the bootstrap.less and variables.less exist in that file path.

I searched around for people's solutions for the You're not in a Meteor project directory. error and that's when I found a few articles relating it to missing files (1, 2 These aren't exactly your problem but I thought it may be a worthwhile bug hunt path).

Now seeing the new log output you mentioned, Installing a Meteor distribution in your home directory., makes me wonder if everything is set up correctly. This person got that error and found that re-installing meteor helped.

[csillag]

@tzmanics:

It's hard to say what's causing Errors prevented bundling: without seeing the file list but I would double check to make sure the bootstrap.less and variables.less exist in that file path.

I understand the problem now, but it's unclear how to solve it best.
We are using the bootstrap NPM package, which ships with bootstrap.less. We are importing this less file in our less code. However, importing less files directly from npm packages to the Meteor app is not really supported (see meteor/meteor#6037 ), so we are using a workaround: we have a symlink pointing from less/imports/boostrap' to 'node_modules/bootstrap. This way, we can successfully import the less file, while running in development mode.

However, demeteorizer chokes on this file, because it tries to re-create the build environment somehow, and the symlink will miss its target....

[csillag]

I will probably get rid of this packaging trick, but than requires some work in other areas...

[csillag]

OK, now I have a new error message (channeled to me from the Modulus build servers):

/mnt/home/.meteor/packages/meteor-tool/.1.3.2_2.38wtkx++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/lib/node_modules/meteor-promise/promise_server.js:116
      throw error;
            ^
Error: ENOSPC, open '/mnt/home/.meteor/packages/flemay_less-autoprefixer/.build245582..1.2.0.1ddzgsx++os+web.browser+web.cordova/plugin.compileLessAddAutoprefixer.os/npm/node_modules/meteor/babel-compiler/babel-compiler/node_modules/meteor-babel/node_modules/acorn/src/whitespace.js'
Conversion failed.
Build attempt failed

Usually, ENOSPC means out of disk space... but it must be something else now, right?

[ajschmaltz]

I just totally dropped modulus and went with galaxy.

[twastvedt]

Ditto, but with DigitalOcean. It's not as hard as I thought, and I have full control.

[tzmanics]

@ajschmaltz & @twastvedt Thanks for the time you have spent with us. We hope to make things even better in the future and get the Meteor Beta to a better and stable state. Good luck & happy coding!

[ajschmaltz]

@tzmanics I shouldn't carelessly speak, let me follow your kind gesture with a recommendation for others:

Modulus is great. Deploying many projects in many environments is very well done, competitively priced and all in once place. I recommend it to all. Meteor's in house solution just obviously is going to handle bleeding edge cases with more stability. I do plan on switching back once things stabilize. That's the great thing about how portable this stuff is these days.

[tzmanics]

Thank you @ajschmaltz we really do appreciate that & are working to get these issues resolved :)

[csillag]

OK so now it's clear that I am hitting a separate problem, I will open a new issue about it.

[workflow]

As an addition to @tzmanics' solution (ended up working for us, thank you!):

After changing the runtime image to Meteor you need to STOP the project, then deploy, instead of merely restarting it. Had missed that one.

[oliverlloyd]

The solution given by @jackboberg here works but it did take us a little while to get there. In the end we reset the project, removed node_modules made sure there were no .demoeteorised folders hanging around and then followed the recommended steps (as above, install v6 of CLI and use the new Meteor image type).

[HynesIP]

It seemed like I didn't have a choice but to move to stay with CLI 6.0 and update my DEV env to a Meteor image. I really didn't want to move to a Meteor image while it was being called Beta. But, here I am and I'll be moving to the Meteor image for prod on my next deployment 6/16/16.

[timmakken]

I havent found a solution for the [Error] There was a problem deploying your project. problem.
Have tried old modulus CLI versions , the newest one with next and a meteor container but so far i havent been able to deploy my new app.

I wanted it to run on the new Meteor beta container but i cant seem to make it work. Any tips?