Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve performance of finding files in node_modules #10839

Merged
merged 6 commits into from Jun 24, 2020

Conversation

zodern
Copy link
Member

@zodern zodern commented Dec 23, 2019

The improvements are measured with a smaller medium sized app on windows, with 45 direct npm dependencies. Some of the times are approximate since I didn't record exact before and after times for each change.

Share Cache Keys between Archs

The list of files in node_modules is currently cached separately for each arch. By removing the arch name from the cache key and instead including the options that change between arches, the cache entries can be shared. For most apps, this will allow the legacy architecture to use the cache during the initial build instead of redoing the work done for the modern bundle. Saves >1s during the initial build. I haven't tested it, but the cordova web arch might also be able to use the cache during the initial build.

Reduce node_modules cache entries

Meteor creates a separate cache entry for each folder in node_modules. For this app the cache was using 10mb of memory. Only one entry is used by each architecture during rebuilds. It will now only cache the results that will be used in rebuilds, reducing the cache memory usage to 8kb.

Do not watch node_modules after initial build

During the initial build, Meteor adds files and folders in node_modules to the app's watch set. These are always missing after the first full rebuild. It will now not add node_modules to the app's watch set at all. The files seem to still be watched someplace else, but I haven't looked into why or where.

  • Saves several seconds in setting up watchers after the build finishes, but with the improvements in Meteor 1.8.1 it shouldn't normally be noticeable.
  • Meteor checks all of the folders and files in the server watchset when a change is detected by the client watcher to know if a full rebuild should be done or only a client rebuild. Not watching node_modules will reduce the delay between detecting a change and starting the rebuild for a small app by 0.5 - 1s.

Copy link
Member

@filipenevola filipenevola left a comment

Great!

@filipenevola filipenevola requested a review from benjamn Dec 24, 2019
@brucejo75
Copy link
Contributor

brucejo75 commented Jun 20, 2020

@benjamn, @filipenevola, @zodern, any reason this has not been merged?

@filipenevola filipenevola added this to the Release 1.10.3 milestone Jun 24, 2020
@filipenevola filipenevola changed the base branch from devel to release-1.10.3 Jun 24, 2020
@filipenevola
Copy link
Member

filipenevola commented Jun 24, 2020

Hi @brucejo75 we are going to include this on Meteor 1.10.3

@filipenevola filipenevola merged commit 5ad36c9 into release-1.10.3 Jun 24, 2020
@filipenevola filipenevola deleted the find-node_module-performance branch Jun 24, 2020
@filipenevola
Copy link
Member

filipenevola commented Jun 24, 2020

Thanks @zodern, it's published on 1.10.3-beta.3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants