New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release 1.4.2 #7668

Merged
merged 353 commits into from Oct 25, 2016

Conversation

@benjamn
Member

benjamn commented Aug 18, 2016

As promised in the 1.4.1 release announcement, Meteor 1.4.2 will be all about rebuild performance, as well as the usual bug fixes post-1.4.1.

The easiest way to get involved is to keep an eye on this pull request, and run meteor update --release 1.4.2-beta.n in a test app whenever you see a new beta.n version appear. Technically, you should wait at least a half hour after you see the version appear on the releases page, since it takes a bit of time to publish the meteor-tool package for all architectures. If you've helped out with previous beta testing cycles, all of this should be familiar.

A more challenging way to get involved is to take matters into your own hands, and help improve Meteor rebuild performance. If you're up for that, we encourage you to identify specific performance problems, open issues to invite discussion of those problems, and then submit pull requests based on the outcomes of those discussions. Your pull requests should target the https://github.com/meteor/meteor/tree/release-1.4.2 branch instead of devel.

Meaningful performance work is not easy, so we really must insist on the process outlined above. Contributions that come out of nowhere are not likely to be considered or accepted, because they tend not to take the whole picture into consideration.

With that caveat out of the way, I don't think I need to convince you that build performance improvements are among the highest-impact work you can possibly do for the Meteor community. More than just improving your own developer experience, you’ll be a hero.

Here are two great documents to read to get started thinking about Meteor performance optimization:

To be clear, we are not expecting the community to do all (or even very much) of this work. Without a doubt, the core team will be working on these problems harder than anyone. If you're up for a challenge, though, we're here to support your efforts.

@benjamn benjamn added the in progress label Aug 18, 2016

@benjamn benjamn added this to the Release 1.4.2 milestone Aug 18, 2016

@benjamn

This comment has been minimized.

Show comment
Hide comment
@benjamn

benjamn Aug 19, 2016

Member

Here's a task that could have a big impact on rebuild times: #7673

Member

benjamn commented Aug 19, 2016

Here's a task that could have a big impact on rebuild times: #7673

@nicolaslopezj

This comment has been minimized.

Show comment
Hide comment
@nicolaslopezj

nicolaslopezj Aug 19, 2016

@benjamn what do you think about this?

Expose a option to not watch for changes on files on server or client. With the new file structure, where all files are in a common folder every time I change a client file all the server restarts taking more than 40 seconds.

With this option, we could prevents rebuilds when changing files in the ui folder for the server and the api folder for client.

And since most of the files changes are for the client (in my case), most rebuild times will be improved in 30 seconds or more.

A much better solution would be to watch only the files that are imported, on client and server separately. But I don't know how difficult could this be.

nicolaslopezj commented Aug 19, 2016

@benjamn what do you think about this?

Expose a option to not watch for changes on files on server or client. With the new file structure, where all files are in a common folder every time I change a client file all the server restarts taking more than 40 seconds.

With this option, we could prevents rebuilds when changing files in the ui folder for the server and the api folder for client.

And since most of the files changes are for the client (in my case), most rebuild times will be improved in 30 seconds or more.

A much better solution would be to watch only the files that are imported, on client and server separately. But I don't know how difficult could this be.

@laosb

This comment has been minimized.

Show comment
Hide comment
@laosb

laosb Aug 19, 2016

Collaborator

I guess rebuild is always needed for server side changes, since we can't really "hotreload" the server bundle yet.

Collaborator

laosb commented Aug 19, 2016

I guess rebuild is always needed for server side changes, since we can't really "hotreload" the server bundle yet.

@veered

This comment has been minimized.

Show comment
Hide comment
@veered

veered Aug 25, 2016

Contributor

I've added a ton more detail to my document on how to speed up reloads. In particular, I've added much more information on what I think is slowing things down and how I think we might be able to fix these things. I highly recommend that anyone interested in this problem take a look at this document. If you read the document previously, the new info starts in the Potential Problems section.

I've created a topic for this document on the Meteor forums

Contributor

veered commented Aug 25, 2016

I've added a ton more detail to my document on how to speed up reloads. In particular, I've added much more information on what I think is slowing things down and how I think we might be able to fix these things. I highly recommend that anyone interested in this problem take a look at this document. If you read the document previously, the new info starts in the Potential Problems section.

I've created a topic for this document on the Meteor forums

@benjamn

This comment has been minimized.

Show comment
Hide comment
@benjamn

benjamn Aug 26, 2016

Member

Great ideas, @veered. Thanks for adding those additional details.

I should mention that I'll be off the grid (at burning man, my first vacation since this time last year) until after Labor Day (back September 6th), but I'm really excited to work through these possibilities when I get back.

Feel free to create issues for any of the improvements you have in mind. I'll defer to @tmeasday (even more than usual) to answer questions while I'm gone.

Member

benjamn commented Aug 26, 2016

Great ideas, @veered. Thanks for adding those additional details.

I should mention that I'll be off the grid (at burning man, my first vacation since this time last year) until after Labor Day (back September 6th), but I'm really excited to work through these possibilities when I get back.

Feel free to create issues for any of the improvements you have in mind. I'll defer to @tmeasday (even more than usual) to answer questions while I'm gone.

@Akryum

This comment has been minimized.

Show comment
Hide comment
@Akryum

Akryum Aug 26, 2016

@benjamn Would you be interested in discussing a way to officially integrate hot-reloading of components (react or vue) and javascript modules to improve the developper experience? For example, I used a socket server and blocked automatic refresh under certain conditions for akryum:vue-component, but we could also use a special collection like what autoupdate does (perhaps holding component/module content).
For now, I must say what I did to integrate vue and hot-reloading feels quite hacky (but it works!).

Akryum commented Aug 26, 2016

@benjamn Would you be interested in discussing a way to officially integrate hot-reloading of components (react or vue) and javascript modules to improve the developper experience? For example, I used a socket server and blocked automatic refresh under certain conditions for akryum:vue-component, but we could also use a special collection like what autoupdate does (perhaps holding component/module content).
For now, I must say what I did to integrate vue and hot-reloading feels quite hacky (but it works!).

@tdesc

This comment has been minimized.

Show comment
Hide comment
@tdesc

tdesc Aug 26, 2016

@benjamn I use Google Polymer as ui for meteor application and code in /public folder. And for "hot-code" reload I forced to start "meteor --once" and copy /public in .meteor/build/web
I'm interested in some option for meteor, to stop restarting server on /public changes

tdesc commented Aug 26, 2016

@benjamn I use Google Polymer as ui for meteor application and code in /public folder. And for "hot-code" reload I forced to start "meteor --once" and copy /public in .meteor/build/web
I'm interested in some option for meteor, to stop restarting server on /public changes

@clayne11

This comment has been minimized.

Show comment
Hide comment
@clayne11

clayne11 Aug 31, 2016

@Akryum we already have a pretty promising community solution for HMR. It needs some work, and it seems like @gadicc has stopped working on it, but I think it's got some good building blocks.

clayne11 commented Aug 31, 2016

@Akryum we already have a pretty promising community solution for HMR. It needs some work, and it seems like @gadicc has stopped working on it, but I think it's got some good building blocks.

@Akryum

This comment has been minimized.

Show comment
Hide comment
@Akryum

Akryum Sep 1, 2016

@clayne11 maybe we could work on bringing hot-reload into core Meteor.

Akryum commented Sep 1, 2016

@clayne11 maybe we could work on bringing hot-reload into core Meteor.

@veered

This comment has been minimized.

Show comment
Hide comment
@veered

veered Sep 1, 2016

Contributor

@clayne11 @Akryum Does it work for Blaze?

Contributor

veered commented Sep 1, 2016

@clayne11 @Akryum Does it work for Blaze?

@clayne11

This comment has been minimized.

Show comment
Hide comment
@clayne11

clayne11 Sep 1, 2016

@veered HMR (hot module replacement) doesn't care what view layer you're using. They're completely independent of each other. I'm sure you can make it work with Blaze, however you'll lose all of your component state because as far as I know there isn't a hot-reloading solution for Blaze so you'll have to remove and re-render your entire view.

clayne11 commented Sep 1, 2016

@veered HMR (hot module replacement) doesn't care what view layer you're using. They're completely independent of each other. I'm sure you can make it work with Blaze, however you'll lose all of your component state because as far as I know there isn't a hot-reloading solution for Blaze so you'll have to remove and re-render your entire view.

@gadicc

This comment has been minimized.

Show comment
Hide comment
@gadicc

gadicc Sep 3, 2016

Contributor

Thanks @clayne11, as usual :)

I was so hoping to get a final "1.0.0" release out for meteor-hmr before I stopped working on it, but I just ran out of time, with a lot of pressure from other projects :( The good news though is that the final beta was really really close, and as expected, the new design behind it allowed it to keep up with later versions of Meteor.

Re Blaze, I think someone who knows Blaze well could probably add Blaze hotloading (with state) using meteor-hmr in under an hour. I know with the "fview-lab" I had for famous-views (wish all the *.meteor.com sites were still up, but basically it looked like this), we did a kind of hotloading on every keystroke while retaining state, it was a lot of fun :)

Re adding to core, well, unlike most of my other projects, because this was a combination of a lot of different experiments/hacks, the code base is a little bit all over the place. But, the main client-side runtime implements the most important HMR features, has a few tests, and could definitely be reused with a little cleanup. And all the server side really does is send the updated bundle across, that wouldn't be too hard to do in core. Ccurrently we do it in a separate process for speed, but with all the build time improvements, hopefully no need for that. As you can see though, there are a lot of higher priority updates for Meteor to come first.

Contributor

gadicc commented Sep 3, 2016

Thanks @clayne11, as usual :)

I was so hoping to get a final "1.0.0" release out for meteor-hmr before I stopped working on it, but I just ran out of time, with a lot of pressure from other projects :( The good news though is that the final beta was really really close, and as expected, the new design behind it allowed it to keep up with later versions of Meteor.

Re Blaze, I think someone who knows Blaze well could probably add Blaze hotloading (with state) using meteor-hmr in under an hour. I know with the "fview-lab" I had for famous-views (wish all the *.meteor.com sites were still up, but basically it looked like this), we did a kind of hotloading on every keystroke while retaining state, it was a lot of fun :)

Re adding to core, well, unlike most of my other projects, because this was a combination of a lot of different experiments/hacks, the code base is a little bit all over the place. But, the main client-side runtime implements the most important HMR features, has a few tests, and could definitely be reused with a little cleanup. And all the server side really does is send the updated bundle across, that wouldn't be too hard to do in core. Ccurrently we do it in a separate process for speed, but with all the build time improvements, hopefully no need for that. As you can see though, there are a lot of higher priority updates for Meteor to come first.

@gadicc

This comment has been minimized.

Show comment
Hide comment
@gadicc

gadicc Sep 3, 2016

Contributor

@benjamn, this is something I would have loved to have gotten involved in only I had more time :( Some of the things I had thought about while working on meteor-hmr were:

  • Keep everything in memory, and only write to disk asynchronously after rebuild is complete
  • Parallelize build plugin runs (I had thought of a separate build plugin manager process)
  • Avoid rebuilding files that haven't changed, re-use cached builds when reconstructing bundle.

Good luck! Hope you got a lot out of Burning Man and your well deserved break!

Contributor

gadicc commented Sep 3, 2016

@benjamn, this is something I would have loved to have gotten involved in only I had more time :( Some of the things I had thought about while working on meteor-hmr were:

  • Keep everything in memory, and only write to disk asynchronously after rebuild is complete
  • Parallelize build plugin runs (I had thought of a separate build plugin manager process)
  • Avoid rebuilding files that haven't changed, re-use cached builds when reconstructing bundle.

Good luck! Hope you got a lot out of Burning Man and your well deserved break!

@laosb

This comment has been minimized.

Show comment
Hide comment
@laosb

laosb Sep 4, 2016

Collaborator

Totally want to see HMR in core! But does it give us fast rebuild times out of box? I guess bringing HMR to core is an separated topic and should be in an another issue.

Collaborator

laosb commented Sep 4, 2016

Totally want to see HMR in core! But does it give us fast rebuild times out of box? I guess bringing HMR to core is an separated topic and should be in an another issue.

@laosb

This comment has been minimized.

Show comment
Hide comment
@laosb

laosb Sep 4, 2016

Collaborator

Keep everything in memory would slightly increase the required memory space, which is already too large now. Maybe it can be an option, but shouldn't be the default.

Parallelizing is a good idea though.

Collaborator

laosb commented Sep 4, 2016

Keep everything in memory would slightly increase the required memory space, which is already too large now. Maybe it can be an option, but shouldn't be the default.

Parallelizing is a good idea though.

@veered

This comment has been minimized.

Show comment
Hide comment
@veered

veered Sep 8, 2016

Contributor

@benjamn Any movement on this PR?

Contributor

veered commented Sep 8, 2016

@benjamn Any movement on this PR?

@clayne11

This comment has been minimized.

Show comment
Hide comment
@clayne11

clayne11 Sep 8, 2016

@laosb I completely disagree. Running from memory should absolutely be the default. The performance of the Meteor build tool is absolutely abhorrent at the moment. Running the build from memory could significantly improve it.

Using a bit more memory in the name of performance is a phenomenal idea. Webpack is able to do it and I see no reason why Meteor shouldn't as well.

clayne11 commented Sep 8, 2016

@laosb I completely disagree. Running from memory should absolutely be the default. The performance of the Meteor build tool is absolutely abhorrent at the moment. Running the build from memory could significantly improve it.

Using a bit more memory in the name of performance is a phenomenal idea. Webpack is able to do it and I see no reason why Meteor shouldn't as well.

@veered

This comment has been minimized.

Show comment
Hide comment
@veered

veered Sep 8, 2016

Contributor

@clayne11 I agree that keeping things in memory is the way to go. The tradeoff between marginal increases in memory usage and large increases in developer productivity seems pretty one-sided. Have you read my article about how to get faster reloads?

Contributor

veered commented Sep 8, 2016

@clayne11 I agree that keeping things in memory is the way to go. The tradeoff between marginal increases in memory usage and large increases in developer productivity seems pretty one-sided. Have you read my article about how to get faster reloads?

@clayne11

This comment has been minimized.

Show comment
Hide comment
@clayne11

clayne11 Sep 8, 2016

I have, and I've done everything except migrate to Webpack and use Meteor packages. I tried using the webpack:webpack package but I had trouble getting my app to build.

I'm not going to use Meteor packages because I'm trying to get away from proprietary Meteor solutions when clearly the Meteor team is not fixing core issues like build-time and code-splitting.

If these issues are not fixed soon I'll be abandoning Meteor in the near future. I have 85k lines of JS and 15k lines of SCSS. My app takes 30 seconds to rebuild on a late 2015 15" rMBP and 15 seconds to load on a phone in production which is completely unacceptable.

I'm not even able to upgrade to 1.4 because Meteor 1.4 completely broke PostCSS (each SCSS file takes ~2.5 seconds to compile vs ~30ms in 1.3).

I'm considering migrating completely off of Meteor and using asteroid to connect to my Meteor server, although I'm concerned with the lack of development on the asteroid repo.

At this point I really wish I had never used Meteor and simply built my app on Webpack and Express / GraphQL. I've completely hit the limits of Meteor and it's holding back my company from moving forward at the speed we'd like to, as well as the 15 second load time seriously hurting our SEO ranking.

I definitely can't recommend Meteor to anyone at this point in time, but maybe these issues can be resolved in the near future.

clayne11 commented Sep 8, 2016

I have, and I've done everything except migrate to Webpack and use Meteor packages. I tried using the webpack:webpack package but I had trouble getting my app to build.

I'm not going to use Meteor packages because I'm trying to get away from proprietary Meteor solutions when clearly the Meteor team is not fixing core issues like build-time and code-splitting.

If these issues are not fixed soon I'll be abandoning Meteor in the near future. I have 85k lines of JS and 15k lines of SCSS. My app takes 30 seconds to rebuild on a late 2015 15" rMBP and 15 seconds to load on a phone in production which is completely unacceptable.

I'm not even able to upgrade to 1.4 because Meteor 1.4 completely broke PostCSS (each SCSS file takes ~2.5 seconds to compile vs ~30ms in 1.3).

I'm considering migrating completely off of Meteor and using asteroid to connect to my Meteor server, although I'm concerned with the lack of development on the asteroid repo.

At this point I really wish I had never used Meteor and simply built my app on Webpack and Express / GraphQL. I've completely hit the limits of Meteor and it's holding back my company from moving forward at the speed we'd like to, as well as the 15 second load time seriously hurting our SEO ranking.

I definitely can't recommend Meteor to anyone at this point in time, but maybe these issues can be resolved in the near future.

@laosb

This comment has been minimized.

Show comment
Hide comment
@laosb

laosb Sep 8, 2016

Collaborator

@clayne11 I totally agree that we should have in-memory cache, but it seems that some of Windows users have already started complaining of the memory usage. So I suggested not being the default.

I guess most people there is using Meteor in production, and some of them have the same or even larger size of codebase when compared to yours. Most people have the same problem, including me. But Meteor is still the very framework (or platform) that you can do a quick start with and has not bad performance in production with realtime data transfer. There's no best choice fits all, but only the one suits your requirements best. Just use the one for the right thing. At this time, I hope we can improve what we use everyday, no matter w/ or w/o MDG, instead of just complaining and have a hard time choosing "the best". It's even easier than switching to another stack, which is totally waste of time in most cases.

Collaborator

laosb commented Sep 8, 2016

@clayne11 I totally agree that we should have in-memory cache, but it seems that some of Windows users have already started complaining of the memory usage. So I suggested not being the default.

I guess most people there is using Meteor in production, and some of them have the same or even larger size of codebase when compared to yours. Most people have the same problem, including me. But Meteor is still the very framework (or platform) that you can do a quick start with and has not bad performance in production with realtime data transfer. There's no best choice fits all, but only the one suits your requirements best. Just use the one for the right thing. At this time, I hope we can improve what we use everyday, no matter w/ or w/o MDG, instead of just complaining and have a hard time choosing "the best". It's even easier than switching to another stack, which is totally waste of time in most cases.

Show outdated Hide outdated .gitmodules
@benjamn

This comment has been minimized.

Show comment
Hide comment
@benjamn

benjamn Sep 15, 2016

Member

Pretty excited about the two most recent optimizations: d618cc6 and d5f239e

Member

benjamn commented Sep 15, 2016

Pretty excited about the two most recent optimizations: d618cc6 and d5f239e

@veered

This comment has been minimized.

Show comment
Hide comment
@veered

veered Sep 15, 2016

Contributor

YESSSSSSS :)

Contributor

veered commented Sep 15, 2016

YESSSSSSS :)

@davidworkman9

This comment has been minimized.

Show comment
Hide comment
@davidworkman9

davidworkman9 Sep 16, 2016

Contributor

Not sure if this is the right place to say this (or if you even want feedback like this, but I'm super excited about getting some productivity back so I can't help myself!).

Personal testing, current beta(3) build times for client are cut in half it seems, server rebuilds are the same if not a bit worse.

Contributor

davidworkman9 commented Sep 16, 2016

Not sure if this is the right place to say this (or if you even want feedback like this, but I'm super excited about getting some productivity back so I can't help myself!).

Personal testing, current beta(3) build times for client are cut in half it seems, server rebuilds are the same if not a bit worse.

@laosb

This comment has been minimized.

Show comment
Hide comment
@laosb

laosb Sep 16, 2016

Collaborator

Sounds good! Looking forward to next beta release!

Collaborator

laosb commented Sep 16, 2016

Sounds good! Looking forward to next beta release!

@hwillson

This comment has been minimized.

Show comment
Hide comment
@hwillson

hwillson Sep 16, 2016

Member

Will the 1.4.2 release address issue #7434? Quite a few devs are starting to move their application structure around to leverage something like imports/client, to prevent unnecessary server rebuilds. I'm just wondering if we should be updating the Guide + todos to recommend this approach, or if this will be unnecessary after 1.4.2.

Member

hwillson commented Sep 16, 2016

Will the 1.4.2 release address issue #7434? Quite a few devs are starting to move their application structure around to leverage something like imports/client, to prevent unnecessary server rebuilds. I'm just wondering if we should be updating the Guide + todos to recommend this approach, or if this will be unnecessary after 1.4.2.

@arggh

This comment has been minimized.

Show comment
Hide comment
@arggh

arggh Sep 16, 2016

Contributor

Just took the new 1.4.2-beta.3 for a spin, and I can already feel the difference. Better times ahead! Thank you @benjamn and everyone else on this: the improvement on the quality of life in general is huge ☀️ 😎

Contributor

arggh commented Sep 16, 2016

Just took the new 1.4.2-beta.3 for a spin, and I can already feel the difference. Better times ahead! Thank you @benjamn and everyone else on this: the improvement on the quality of life in general is huge ☀️ 😎

@benjamn

This comment has been minimized.

Show comment
Hide comment
@benjamn

benjamn Sep 16, 2016

Member

If you have a lot of local packages, 1.4.2-beta.4 should be noticeably faster still (coming soon).

Member

benjamn commented Sep 16, 2016

If you have a lot of local packages, 1.4.2-beta.4 should be noticeably faster still (coming soon).

@arggh

This comment has been minimized.

Show comment
Hide comment
@arggh

arggh Sep 19, 2016

Contributor

TL;DR Regarding local packages, 1.4.2-beta.X fixes a problem, where a local package's unused node modules would mess with the deployment/build process when using Meteor Up. Is this a known fact and fix?


Long story here

I recently refactored my Semantic-UI styles to a separate local package (semantic:ui was taking ~10s minimum on every refresh). The local package has semantic-ui npm module as a devDependency in package.json and lots of libraries in node_modules from semantic-ui's dependencies. I generate the dist/*.js and dist/*.css by running a gulp task.

Package file structure is approximately this:

package/
  node_modules
  semantic
     dist
        sui.js
        sui.css
     themes
        ...icons
  src
     ..loads of .less and .js
  gulpfile.js
  package.js
  package.json

In my package.js, I only add/use the dist-files:

Package.describe({
  name: 'my-custom-semantic-ui',
  summary: '',
  version: '1.0.0'
});

Package.onUse(function(api) {
  api.addFiles([
    'semantic/dist/sui.css',
    'semantic/dist/sui.js'
  ], 'client');
  api.addAssets([
    'semantic/dist/themes/default/assets/fonts/icons.eot',
    'semantic/dist/themes/default/assets/fonts/icons.svg',
    'semantic/dist/themes/default/assets/fonts/icons.ttf',
    'semantic/dist/themes/default/assets/fonts/icons.woff',
    'semantic/dist/themes/default/assets/fonts/icons.woff2',
  ], 'client');
});

Running the app locally I had no problems, but with Meteor 1.4.1, my Meteor Up -deployment would fail:

[server]> meteor-dev-bundle@0.0.0 install /bundle/bundle/programs/server
[server]> node npm-rebuild.js
[server]
[server]
[server]> fsevents@1.0.14 install /bundle/bundle/programs/server/npm/node_modules/meteor/my_custom_semantic_ui/node_modules/fsevents
[server]> node-pre-gyp install --fallback-to-build
[server]
[server]node-pre-gyp ERR! Tried to download: https://fsevents-binaries.s3-us-west-2.amazonaws.com/v1.0.14/fse-v1.0.14-node-v46-linux-x64.tar.gz 
[server]node-pre-gyp ERR! Pre-built binaries not found for fsevents@1.0.14 and node@4.4.7 (node-v46 ABI) (falling back to source compile with node-gyp) 
[server]make: Entering directory '/bundle/bundle/programs/server/npm/node_modules/meteor/my_custom_semantic_ui/node_modules/fsevents/build'
[server]  SOLINK_MODULE(target) Release/obj.target/.node
[server]  COPY Release/.node
[server]make: Leaving directory '/bundle/bundle/programs/server/npm/node_modules/meteor/my_custom_semantic_ui/node_modules/fsevents/build'

so, it seems the unused node modules were not ignored. (Maybe I've misunderstood something about Meteor packages?)

However, with 1.4.2-beta.4, my deployment completes without problems. So, whatever @benjamn did to change the way these unused node_modules are handled in packages, it seems to fix this problem.

Contributor

arggh commented Sep 19, 2016

TL;DR Regarding local packages, 1.4.2-beta.X fixes a problem, where a local package's unused node modules would mess with the deployment/build process when using Meteor Up. Is this a known fact and fix?


Long story here

I recently refactored my Semantic-UI styles to a separate local package (semantic:ui was taking ~10s minimum on every refresh). The local package has semantic-ui npm module as a devDependency in package.json and lots of libraries in node_modules from semantic-ui's dependencies. I generate the dist/*.js and dist/*.css by running a gulp task.

Package file structure is approximately this:

package/
  node_modules
  semantic
     dist
        sui.js
        sui.css
     themes
        ...icons
  src
     ..loads of .less and .js
  gulpfile.js
  package.js
  package.json

In my package.js, I only add/use the dist-files:

Package.describe({
  name: 'my-custom-semantic-ui',
  summary: '',
  version: '1.0.0'
});

Package.onUse(function(api) {
  api.addFiles([
    'semantic/dist/sui.css',
    'semantic/dist/sui.js'
  ], 'client');
  api.addAssets([
    'semantic/dist/themes/default/assets/fonts/icons.eot',
    'semantic/dist/themes/default/assets/fonts/icons.svg',
    'semantic/dist/themes/default/assets/fonts/icons.ttf',
    'semantic/dist/themes/default/assets/fonts/icons.woff',
    'semantic/dist/themes/default/assets/fonts/icons.woff2',
  ], 'client');
});

Running the app locally I had no problems, but with Meteor 1.4.1, my Meteor Up -deployment would fail:

[server]> meteor-dev-bundle@0.0.0 install /bundle/bundle/programs/server
[server]> node npm-rebuild.js
[server]
[server]
[server]> fsevents@1.0.14 install /bundle/bundle/programs/server/npm/node_modules/meteor/my_custom_semantic_ui/node_modules/fsevents
[server]> node-pre-gyp install --fallback-to-build
[server]
[server]node-pre-gyp ERR! Tried to download: https://fsevents-binaries.s3-us-west-2.amazonaws.com/v1.0.14/fse-v1.0.14-node-v46-linux-x64.tar.gz 
[server]node-pre-gyp ERR! Pre-built binaries not found for fsevents@1.0.14 and node@4.4.7 (node-v46 ABI) (falling back to source compile with node-gyp) 
[server]make: Entering directory '/bundle/bundle/programs/server/npm/node_modules/meteor/my_custom_semantic_ui/node_modules/fsevents/build'
[server]  SOLINK_MODULE(target) Release/obj.target/.node
[server]  COPY Release/.node
[server]make: Leaving directory '/bundle/bundle/programs/server/npm/node_modules/meteor/my_custom_semantic_ui/node_modules/fsevents/build'

so, it seems the unused node modules were not ignored. (Maybe I've misunderstood something about Meteor packages?)

However, with 1.4.2-beta.4, my deployment completes without problems. So, whatever @benjamn did to change the way these unused node_modules are handled in packages, it seems to fix this problem.

@benjamn benjamn changed the base branch from devel to master Sep 20, 2016

@sebakerckhof

This comment has been minimized.

Show comment
Hide comment
@sebakerckhof

sebakerckhof Sep 21, 2016

Contributor

Could this make it in 1.4.2, since it's a blocker for our builds and approved by some other people by now: #7637

Contributor

sebakerckhof commented Sep 21, 2016

Could this make it in 1.4.2, since it's a blocker for our builds and approved by some other people by now: #7637

@mitar

This comment has been minimized.

Show comment
Hide comment
@mitar

mitar Oct 25, 2016

Collaborator

That said, I'm willing to be creative about what constitutes a "bug." For example, #6002 will have to wait for 1.4.2.1, but a simpler version that ignores hidden (.*) files could be considered, for the sake of consistency with existing logic.

Can we then get skipping of . files at least? ;-)

Collaborator

mitar commented Oct 25, 2016

That said, I'm willing to be creative about what constitutes a "bug." For example, #6002 will have to wait for 1.4.2.1, but a simpler version that ignores hidden (.*) files could be considered, for the sake of consistency with existing logic.

Can we then get skipping of . files at least? ;-)

@mitar

This comment has been minimized.

Show comment
Hide comment
@mitar

mitar Oct 25, 2016

Collaborator

I ran meteor --release 1.4.2-beta.13 on my project and after changing one file it now took 15 seconds for client to reload. This is a pretty good improvement over before.

Nooo, 1.4.2-rc.4 works again so slow for my project. :-( (Like more than minute.)

Also, it does not seem to reload on changes to .idea. Was this added? Or am I running against some old version?

Collaborator

mitar commented Oct 25, 2016

I ran meteor --release 1.4.2-beta.13 on my project and after changing one file it now took 15 seconds for client to reload. This is a pretty good improvement over before.

Nooo, 1.4.2-rc.4 works again so slow for my project. :-( (Like more than minute.)

Also, it does not seem to reload on changes to .idea. Was this added? Or am I running against some old version?

@pozylon

This comment has been minimized.

Show comment
Hide comment
@pozylon

pozylon Oct 25, 2016

@mitar: Same is true with my project, maybe this is because of 03e798a. My dev dependencies const of a huge bunch of eslint and plugins. What if only meteor test includes dev dependencies and meteor run does not? Or maybe hard-cache dev dependencies (no recompilation after initial build)?

I don't really understand the underlying build pipeline, sorry if I don't get the bottlenecks, maybe writing stuff that doesn't help you.

Identifying my code I can say that:

  • app code, tests, local atmosphere packages, npm-link npm packages change every minute.
  • private & public folders, remote atmosphere packages, normal npm modules, meteor release change every few days.

Is it possible to trigger a full server restart if packages or node_modules dir's except for the local packages and symlinked/npm-link node_modules change and use completely cached versions of those files on normal rebuilds?

  • Start (initial build)
  • Full Rebuild (= Start, but triggered through watchers) taking care of changes usually happen every few days.
  • Rebuild (current server-side + cached npm_modules & packages except local/symlinked)
  • Refresh (current client-side + cached npm_modules & packages except local/symlinked)

I mean if somebody adds an atmosphere or npm package dependency it's fine to wait a minute, right?

pozylon commented Oct 25, 2016

@mitar: Same is true with my project, maybe this is because of 03e798a. My dev dependencies const of a huge bunch of eslint and plugins. What if only meteor test includes dev dependencies and meteor run does not? Or maybe hard-cache dev dependencies (no recompilation after initial build)?

I don't really understand the underlying build pipeline, sorry if I don't get the bottlenecks, maybe writing stuff that doesn't help you.

Identifying my code I can say that:

  • app code, tests, local atmosphere packages, npm-link npm packages change every minute.
  • private & public folders, remote atmosphere packages, normal npm modules, meteor release change every few days.

Is it possible to trigger a full server restart if packages or node_modules dir's except for the local packages and symlinked/npm-link node_modules change and use completely cached versions of those files on normal rebuilds?

  • Start (initial build)
  • Full Rebuild (= Start, but triggered through watchers) taking care of changes usually happen every few days.
  • Rebuild (current server-side + cached npm_modules & packages except local/symlinked)
  • Refresh (current client-side + cached npm_modules & packages except local/symlinked)

I mean if somebody adds an atmosphere or npm package dependency it's fine to wait a minute, right?

@benjamn

This comment has been minimized.

Show comment
Hide comment
@benjamn

benjamn Oct 25, 2016

Member

What if only meteor test includes dev dependencies and meteor run does not?

i agree that might be a safer change at this stage. I saw an opportunity to fix #7953 and I went for it, perhaps against my better judgment. Thanks for catching the problem!

Member

benjamn commented Oct 25, 2016

What if only meteor test includes dev dependencies and meteor run does not?

i agree that might be a safer change at this stage. I saw an opportunity to fix #7953 and I went for it, perhaps against my better judgment. Thanks for catching the problem!

@benjamn

This comment has been minimized.

Show comment
Hide comment
@benjamn

benjamn Oct 25, 2016

Member

Also, it does not seem to reload on changes to .idea. Was this added? Or am I running against some old version?

I tried creating a reproduction where .* files mattered and I wasn't able to, so I guess that's good news, though I don't know what might have changed. If you can put together an example where it's still a problem, we can fix it in 1.4.2.1.

Member

benjamn commented Oct 25, 2016

Also, it does not seem to reload on changes to .idea. Was this added? Or am I running against some old version?

I tried creating a reproduction where .* files mattered and I wasn't able to, so I guess that's good news, though I don't know what might have changed. If you can put together an example where it's still a problem, we can fix it in 1.4.2.1.

@veered

This comment has been minimized.

Show comment
Hide comment
@veered

veered Oct 25, 2016

Contributor

I'm glad you included the new profiling information; 7.3 seconds is spent in Meteor server startup for me.

Also, I'm very curious if you have had a chance to look at _initializeCatalog, it is still taking about 5s for me and it will sometimes get stuck and take 1+ minutes. I think running it in serial to avoid fiber-swapping hell might fix this issue (or figuring out why the logging it's doing gets so backed up).

Contributor

veered commented Oct 25, 2016

I'm glad you included the new profiling information; 7.3 seconds is spent in Meteor server startup for me.

Also, I'm very curious if you have had a chance to look at _initializeCatalog, it is still taking about 5s for me and it will sometimes get stuck and take 1+ minutes. I think running it in serial to avoid fiber-swapping hell might fix this issue (or figuring out why the logging it's doing gets so backed up).

@mitar

This comment has been minimized.

Show comment
Hide comment
@mitar

mitar Oct 25, 2016

Collaborator

@benjamn: Please try building this: https://github.com/peer/mind This is my bigger issue for now because it takes more than a minute to build a change to a Blaze template. :-(

Collaborator

mitar commented Oct 25, 2016

@benjamn: Please try building this: https://github.com/peer/mind This is my bigger issue for now because it takes more than a minute to build a change to a Blaze template. :-(

@benjamn

This comment has been minimized.

Show comment
Hide comment
@benjamn

benjamn Oct 25, 2016

Member

@veered I wasn't able to reproduce that problem, but I have an idea for an environment variable that might help you evaluate the impact of fibers on _initializeCatalog (coming soon).

@mitar With cadf113, rebuilds take about 9 seconds for me when I change one of your templates.

Member

benjamn commented Oct 25, 2016

@veered I wasn't able to reproduce that problem, but I have an idea for an environment variable that might help you evaluate the impact of fibers on _initializeCatalog (coming soon).

@mitar With cadf113, rebuilds take about 9 seconds for me when I change one of your templates.

@Akryum

This comment has been minimized.

Show comment
Hide comment
@Akryum

Akryum Oct 25, 2016

Yes, it's quite tedious to work on a npm package with meteor right now. I
have to restart meteor every time I make a change.

Le ven. 21 oct. 2016 22:12, Nicolás López notifications@github.com a
écrit :

@ElegantSudo https://github.com/elegantsudo I agree, but with the
package.json, we should also watch the folders that are added with npm link


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#7668 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACqyfF9xO5YUpOOfOV5a0kzLlSu5WCDAks5q2RzLgaJpZM4Jn6cU
.

Akryum commented Oct 25, 2016

Yes, it's quite tedious to work on a npm package with meteor right now. I
have to restart meteor every time I make a change.

Le ven. 21 oct. 2016 22:12, Nicolás López notifications@github.com a
écrit :

@ElegantSudo https://github.com/elegantsudo I agree, but with the
package.json, we should also watch the folders that are added with npm link


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#7668 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACqyfF9xO5YUpOOfOV5a0kzLlSu5WCDAks5q2RzLgaJpZM4Jn6cU
.

@mitar

This comment has been minimized.

Show comment
Hide comment
@mitar

mitar Oct 25, 2016

Collaborator

rebuilds take about 9 seconds for me when I change one of your templates.

I will check. But you do agree that 9 seconds is a bit a lot for changing one file only. ;-)

Collaborator

mitar commented Oct 25, 2016

rebuilds take about 9 seconds for me when I change one of your templates.

I will check. But you do agree that 9 seconds is a bit a lot for changing one file only. ;-)

@aaronjudd

This comment has been minimized.

Show comment
Hide comment
@aaronjudd

aaronjudd Oct 25, 2016

I've been testing various RC candidates, but for us, we're always seeing a failure in startup using autoprefixer. This error persists in the latest release meteor --release METEOR@1.4.2. Meteor does not start.

W20161025-13:39:29.718(-7)? (STDERR) Error: Cannot find module './browsers'
W20161025-13:39:29.719(-7)? (STDERR)     at require (packages/modules-runtime.js:109:19)
W20161025-13:39:29.719(-7)? (STDERR)     at packages/modules.js:2899:14
W20161025-13:39:29.720(-7)? (STDERR)     at meteorInstall.node_modules.autoprefixer.lib.autoprefixer.js (packages/modules.js:2978:4)
...
=> Exited with code: 1
=> Your application is crashing. Waiting for file change.

aaronjudd commented Oct 25, 2016

I've been testing various RC candidates, but for us, we're always seeing a failure in startup using autoprefixer. This error persists in the latest release meteor --release METEOR@1.4.2. Meteor does not start.

W20161025-13:39:29.718(-7)? (STDERR) Error: Cannot find module './browsers'
W20161025-13:39:29.719(-7)? (STDERR)     at require (packages/modules-runtime.js:109:19)
W20161025-13:39:29.719(-7)? (STDERR)     at packages/modules.js:2899:14
W20161025-13:39:29.720(-7)? (STDERR)     at meteorInstall.node_modules.autoprefixer.lib.autoprefixer.js (packages/modules.js:2978:4)
...
=> Exited with code: 1
=> Your application is crashing. Waiting for file change.
@benjamn

This comment has been minimized.

Show comment
Hide comment
@benjamn

benjamn Oct 25, 2016

Member

@aaronjudd Please file a separate issue with a reproduction. The autoprefixer package works for me, so I suspect the root cause may not be related to the release.

Member

benjamn commented Oct 25, 2016

@aaronjudd Please file a separate issue with a reproduction. The autoprefixer package works for me, so I suspect the root cause may not be related to the release.

@benjamn

This comment has been minimized.

Show comment
Hide comment
@benjamn

benjamn Oct 25, 2016

Member

Speaking of the release…

I want to thank everyone who pitched in on this pull request and elsewhere, in related issues and PRs. You've set a great example of community collaboration, and I'm really happy I've had this chance to engage with you and your ideas.

We've come a long way since Meteor 1.4.1, and while there's still lots more work to be done, I think it's important to pause for a moment, package up all this progress, and make it available to the wider community. I hope the release of Meteor 1.4.2 means you can finally update your important apps without hesitation, and please rest assured Meteor 1.4.3 is right around the corner!

At this point, I believe the only steps left are to publish the blog post about Meteor 1.4.2 and recommend the release, but of course you can update now by running meteor update --release 1.4.2 in any app.

Member

benjamn commented Oct 25, 2016

Speaking of the release…

I want to thank everyone who pitched in on this pull request and elsewhere, in related issues and PRs. You've set a great example of community collaboration, and I'm really happy I've had this chance to engage with you and your ideas.

We've come a long way since Meteor 1.4.1, and while there's still lots more work to be done, I think it's important to pause for a moment, package up all this progress, and make it available to the wider community. I hope the release of Meteor 1.4.2 means you can finally update your important apps without hesitation, and please rest assured Meteor 1.4.3 is right around the corner!

At this point, I believe the only steps left are to publish the blog post about Meteor 1.4.2 and recommend the release, but of course you can update now by running meteor update --release 1.4.2 in any app.

@benjamn benjamn merged commit 7614c0d into master Oct 25, 2016

4 checks passed

CLA Author has signed the Meteor CLA.
Details
ci/circleci Your tests passed on CircleCI!
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
@markoshust

This comment has been minimized.

Show comment
Hide comment
@markoshust

markoshust Oct 25, 2016

For those having trouble downloading:
METEOR_WAREHOUSE_URLBASE=https://d3fm2vapipm3k9.cloudfront.net meteor update --release 1.4.2

I agree it is a ways to go to improve build times, but this is a great incremental update to get faster build times, today. Thanks for all the hard work guys.

markoshust commented Oct 25, 2016

For those having trouble downloading:
METEOR_WAREHOUSE_URLBASE=https://d3fm2vapipm3k9.cloudfront.net meteor update --release 1.4.2

I agree it is a ways to go to improve build times, but this is a great incremental update to get faster build times, today. Thanks for all the hard work guys.

@benjamn

This comment has been minimized.

Show comment
Hide comment
@benjamn

benjamn Oct 25, 2016

Member

Update: Meteor 1.4.2 is now recommended, and we've published a short blog post about it: http://info.meteor.com/blog/announcing-meteor-1.4.2

Member

benjamn commented Oct 25, 2016

Update: Meteor 1.4.2 is now recommended, and we've published a short blog post about it: http://info.meteor.com/blog/announcing-meteor-1.4.2

@theosp

This comment has been minimized.

Show comment
Hide comment
@theosp

theosp Oct 26, 2016

Contributor

Where can I learn more about the what defines a version as stable/recommended ?

To me it seems that 24 hours since last rc.4, and 7 days since rc.0 might not be enough for the community to run enough tests on ~15k lines of code version, and report bugs properly .

Would be nice if a planned release date was announced to encourage us to speed up our checks.

Contributor

theosp commented Oct 26, 2016

Where can I learn more about the what defines a version as stable/recommended ?

To me it seems that 24 hours since last rc.4, and 7 days since rc.0 might not be enough for the community to run enough tests on ~15k lines of code version, and report bugs properly .

Would be nice if a planned release date was announced to encourage us to speed up our checks.

@veered

This comment has been minimized.

Show comment
Hide comment
@veered

veered Oct 26, 2016

Contributor

@benjamn Wow running with METEOR_DISABLE_FS_FIBERS=1 consistently saves me 7+ seconds per reload! It didn't seem to effect the LocalCatalog#_loadLocalPackages section very much though (I think it helped a bit, but not a ton).

I will certainly always be running with this environment variable enabled :) I have a feeling that this might help a lot of people with large projects.

Contributor

veered commented Oct 26, 2016

@benjamn Wow running with METEOR_DISABLE_FS_FIBERS=1 consistently saves me 7+ seconds per reload! It didn't seem to effect the LocalCatalog#_loadLocalPackages section very much though (I think it helped a bit, but not a ton).

I will certainly always be running with this environment variable enabled :) I have a feeling that this might help a lot of people with large projects.

@veered

This comment has been minimized.

Show comment
Hide comment
@veered

veered Oct 26, 2016

Contributor

@benjamn Woo! I figured out how to bring LocalCatalog#_loadLocalPackages down from 4+ seconds to ~.1 seconds. I tried this a while ago but it didn't really help until METEOR_DISABLE_FS_FIBERS=1.

All you need to do is replace the forkJoin with a normal loop like this: veered@f143683

Another example of fiber hell! I think anything that was creating a new Fiber for every file (or in this case, even every package) was wrecking performance. There are just too many files in large projects.

Edit: I opened a PR for this change #7972

Contributor

veered commented Oct 26, 2016

@benjamn Woo! I figured out how to bring LocalCatalog#_loadLocalPackages down from 4+ seconds to ~.1 seconds. I tried this a while ago but it didn't really help until METEOR_DISABLE_FS_FIBERS=1.

All you need to do is replace the forkJoin with a normal loop like this: veered@f143683

Another example of fiber hell! I think anything that was creating a new Fiber for every file (or in this case, even every package) was wrecking performance. There are just too many files in large projects.

Edit: I opened a PR for this change #7972

@sebakerckhof

This comment has been minimized.

Show comment
Hide comment
@sebakerckhof

sebakerckhof Oct 26, 2016

Contributor

@veered Seeing the same results. Thanks for the tip! What's the reason this is currently behind a flag?

Contributor

sebakerckhof commented Oct 26, 2016

@veered Seeing the same results. Thanks for the tip! What's the reason this is currently behind a flag?

@veered

This comment has been minimized.

Show comment
Hide comment
@veered

veered Oct 26, 2016

Contributor

The flag was intended to help me debug why LocalCatalog#_loadLocalPackages was so slow for me, not as a potential performance improvement in of itself. It doesn't effect correctness in any way.

Contributor

veered commented Oct 26, 2016

The flag was intended to help me debug why LocalCatalog#_loadLocalPackages was so slow for me, not as a potential performance improvement in of itself. It doesn't effect correctness in any way.

@davidworkman9

This comment has been minimized.

Show comment
Hide comment
@davidworkman9

davidworkman9 Oct 26, 2016

Contributor

I notice a small improvement with METEOR_DISABLE_FS_FIBERS=1. From 21 seconds down to 19 for a full server refresh.

Contributor

davidworkman9 commented Oct 26, 2016

I notice a small improvement with METEOR_DISABLE_FS_FIBERS=1. From 21 seconds down to 19 for a full server refresh.

@mitar

This comment has been minimized.

Show comment
Hide comment
@mitar

mitar Oct 26, 2016

Collaborator

@mitar With cadf113, rebuilds take about 9 seconds for me when I change one of your templates.

I would so love if this would be the case for me as well. :-( With released 1.4.2 it is still beyond 1 minute when I for example add a character to packages/peermind/discussion/list.html or whatever.

But METEOR_DISABLE_FS_FIBERS is amazing. It brings it down to 10 seconds!

The forkJoin loop patch makes scanning packages faster visually, but it still takes 10 seconds for a refresh for my app. The strange thing is that it reloads the server side, not just the client side. Despite me changing a client-side only file. It might be that the latter is a bad interaction with Blaze Components, because they support compiling Blaze on server side as well. But I do not know why it would reload server side if file is still made only for the client side. Is maybe this an issue with #7573?

Collaborator

mitar commented Oct 26, 2016

@mitar With cadf113, rebuilds take about 9 seconds for me when I change one of your templates.

I would so love if this would be the case for me as well. :-( With released 1.4.2 it is still beyond 1 minute when I for example add a character to packages/peermind/discussion/list.html or whatever.

But METEOR_DISABLE_FS_FIBERS is amazing. It brings it down to 10 seconds!

The forkJoin loop patch makes scanning packages faster visually, but it still takes 10 seconds for a refresh for my app. The strange thing is that it reloads the server side, not just the client side. Despite me changing a client-side only file. It might be that the latter is a bad interaction with Blaze Components, because they support compiling Blaze on server side as well. But I do not know why it would reload server side if file is still made only for the client side. Is maybe this an issue with #7573?

@mitar

This comment has been minimized.

Show comment
Hide comment
@mitar

mitar Oct 26, 2016

Collaborator

After looking into it, I think that #7573 is a source of a huge performance hit if you use local packages because it effectively disables reloading only client-side for client-side only changes. (Interestingly, when changing CSS, it does refresh client side only.)

I hope #7573, #7972, #6002, and disabling FS fibers by default could be fixed for 1.4.2.1 and released as soon as possible because that would be a real improvement.

About Fibers. I am surprised that blocking fs calls are faster than fiber-based fs calls. Is it possible that fibers just use some bad data structure internally, which has maybe a liner performance, which then explodes as number of fibers grow? Maybe we should increase fiber pool size?

Collaborator

mitar commented Oct 26, 2016

After looking into it, I think that #7573 is a source of a huge performance hit if you use local packages because it effectively disables reloading only client-side for client-side only changes. (Interestingly, when changing CSS, it does refresh client side only.)

I hope #7573, #7972, #6002, and disabling FS fibers by default could be fixed for 1.4.2.1 and released as soon as possible because that would be a real improvement.

About Fibers. I am surprised that blocking fs calls are faster than fiber-based fs calls. Is it possible that fibers just use some bad data structure internally, which has maybe a liner performance, which then explodes as number of fibers grow? Maybe we should increase fiber pool size?

@veered

This comment has been minimized.

Show comment
Hide comment
@veered

veered Oct 26, 2016

Contributor

@mitar Could you report the output of METEOR_PROFILE=1 with and without the forkJoin patch?

In addition to the fiber pool size you referenced, there is also the one specified here https://github.com/meteor/promise/blob/master/fiber_pool.js#L123. I think either one could be the culprit.

Contributor

veered commented Oct 26, 2016

@mitar Could you report the output of METEOR_PROFILE=1 with and without the forkJoin patch?

In addition to the fiber pool size you referenced, there is also the one specified here https://github.com/meteor/promise/blob/master/fiber_pool.js#L123. I think either one could be the culprit.

@veered

This comment has been minimized.

Show comment
Hide comment
@veered

veered Oct 26, 2016

Contributor

Changing the fiber pool size from 120 to 1000 here https://github.com/meteor/meteor/blob/devel/tools/fs/files.js#L13 (by adding Fiber.poolSize = 1000;) didn't help in the same way as METEOR_DISABLE_FS_FIBERS=1. In fact, it didn't seem to have any effect on performance.

I think that all the file caching improvements decreases the advantage of running fs stuff in fibers. Maybe the caching layer should be before the fiber is spawned? As in, it checks if the file is in the cache before spawning a fiber to read the file?

Contributor

veered commented Oct 26, 2016

Changing the fiber pool size from 120 to 1000 here https://github.com/meteor/meteor/blob/devel/tools/fs/files.js#L13 (by adding Fiber.poolSize = 1000;) didn't help in the same way as METEOR_DISABLE_FS_FIBERS=1. In fact, it didn't seem to have any effect on performance.

I think that all the file caching improvements decreases the advantage of running fs stuff in fibers. Maybe the caching layer should be before the fiber is spawned? As in, it checks if the file is in the cache before spawning a fiber to read the file?

@zhanbst

This comment has been minimized.

Show comment
Hide comment
@zhanbst

zhanbst Oct 27, 2016

@tdesc In production, meteor doesn't reload. May be, you don't know about how to run in production.

zhanbst commented Oct 27, 2016

@tdesc In production, meteor doesn't reload. May be, you don't know about how to run in production.

@derouck

This comment has been minimized.

Show comment
Hide comment
@derouck

derouck Oct 27, 2016

@veered That METEOR_DISABLE_FS_FIBERS=1 has a huge impact for me as well! For me it seems that the actual 1.4.2 release was quite a bit slower than some of the release candidates.

Thanks everyone for the huge performance improvements!!

derouck commented Oct 27, 2016

@veered That METEOR_DISABLE_FS_FIBERS=1 has a huge impact for me as well! For me it seems that the actual 1.4.2 release was quite a bit slower than some of the release candidates.

Thanks everyone for the huge performance improvements!!

@jchristman

This comment has been minimized.

Show comment
Hide comment
@jchristman

jchristman Oct 27, 2016

Hey guys I love the new release -- super zippy!

I do have a quick question though now that things have changed a bit. I have a library I'm developing that I npm link to a meteor project for the purposes of testing. I have an babel watcher autocompiling changed files, which used to cause server restarts when the linked node_module changed. It seems now that this is no longer the case and I have to actually restart meteor entirely to get changes in my npm linked node_module. Is there a way to specifically enable watchers for a specific node_module? I get that it is hugely expensive to start watchers for every node_module and this is a specific use case, but it's really a pain to restart meteor entirely for any changes to the module, and I don't want to move all of my example/test code into the library.

jchristman commented Oct 27, 2016

Hey guys I love the new release -- super zippy!

I do have a quick question though now that things have changed a bit. I have a library I'm developing that I npm link to a meteor project for the purposes of testing. I have an babel watcher autocompiling changed files, which used to cause server restarts when the linked node_module changed. It seems now that this is no longer the case and I have to actually restart meteor entirely to get changes in my npm linked node_module. Is there a way to specifically enable watchers for a specific node_module? I get that it is hugely expensive to start watchers for every node_module and this is a specific use case, but it's really a pain to restart meteor entirely for any changes to the module, and I don't want to move all of my example/test code into the library.

@benjamn

This comment has been minimized.

Show comment
Hide comment
@benjamn

benjamn Oct 27, 2016

Member

@jchristman you and @damonmaria seem to have the same use case: #7978

Member

benjamn commented Oct 27, 2016

@jchristman you and @damonmaria seem to have the same use case: #7978

@benjamn

This comment has been minimized.

Show comment
Hide comment
@benjamn

benjamn Oct 27, 2016

Member

Let's continue this conversation on the Release 1.4.2.1 pull request, or in individual issues, rather than broadcasting to everyone who ever participated here.

Member

benjamn commented Oct 27, 2016

Let's continue this conversation on the Release 1.4.2.1 pull request, or in individual issues, rather than broadcasting to everyone who ever participated here.

@meteor meteor locked and limited conversation to collaborators Oct 27, 2016

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.