Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve AST garbage collection #4762

merged 8 commits into from Dec 23, 2022

Improve AST garbage collection #4762

merged 8 commits into from Dec 23, 2022


Copy link

@bluwy bluwy commented Dec 21, 2022

This PR contains:

  • bugfix
  • feature
  • refactor
  • documentation
  • other

Are tests included?

  • yes (bugfixes and features will not be merged without tests)
  • no

Breaking Changes?

  • yes (breaking changes will not be merged unless absolutely necessary)
  • no

List any relevant issue numbers:



This PR does 2 things:

  1. Prevent attaching esTreeNode to BaseNode (except dynamic import case). This means that AST leaf nodes are only copied in the parseNode phase, which also prevents the leaf node being duplicated in both esTreeNode and BaseNode itself. esTreeNode can then be garbage collected once it leaves the constructor call.
  2. Make ModuleInfo.ast lazy and apply LRU cache. Now Rollup doesn't need to keep the AST in memory unless requested. A fixed size LRU cache is added only for perf if someone accesses it repeatedly. This is non-configurable (for now) since it's hard to find the right number. In practice, many plugins don't read ModuleInfo.ast.


If caching is enabled, we keep the old method as it's more efficient to share the AST when building the final cache object. If it's disabled, we use the new lazy method. This also means that this optimization is opt-in only by setting cache: false.


For, I saw peak memory usage from 4.5GB to 3.5GB.

For, it failed with a OOM error, but this PR made it work with peak memory of 3.5GB.

(Note: It's more accurate to say that this PR allows more memory to be freed when the process nears the max heap limit, allowing users to run larger builds)


Thanks Astro for sponsoring the work 馃檪

Copy link

codecov bot commented Dec 21, 2022

Codecov Report

Merging #4762 (575dfa3) into master (1243370) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master    #4762   +/-   ##
  Coverage   99.03%   99.03%           
  Files         217      217           
  Lines        7762     7772   +10     
  Branches     2148     2153    +5     
+ Hits         7687     7697   +10     
  Misses         24       24           
  Partials       51       51           
Impacted Files Coverage 螖
src/ModuleLoader.ts 100.00% <酶> (酶)
src/Graph.ts 100.00% <100.00%> (酶)
src/Module.ts 100.00% <100.00%> (酶)
src/ast/nodes/ImportExpression.ts 100.00% <100.00%> (酶)
src/ast/nodes/shared/Node.ts 100.00% <100.00%> (酶)
src/rollup/rollup.ts 100.00% <100.00%> (酶)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Copy link
Contributor Author

bluwy commented Dec 21, 2022

Added a test and found a bug (fixed). I hope this fixes the coverage. I seem to get tests errors when running test:coverage locally so I'm not sure if it works.

Copy link

@bluwy what method did you use to determine peak memory in your tests? I used /usr/bin/time -v yarn build when testing this out on our project and did not see nearly the same magnitude of benefit. The numbers below are the reported "Maximum resident set size (kbytes)" converted to gb.

rollup@2.79.1 -- 2.11gb
rollup@2.79.1 with build.rollupOptions.cache: false -- 2.19gb
this PR with build.rollupOptions.cache: false -- 2.12gb

Copy link

@lukastaegert lukastaegert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice, this is really top-notch work. Will leave it open for at least a day so that people can do their benchmarking, but it looks very reasonable to me.

src/Graph.ts Outdated Show resolved Hide resolved
Copy link
Contributor Author

bluwy commented Dec 22, 2022

@bluwy what method did you use to determine peak memory in your tests? I used /usr/bin/time -v yarn build when testing this out on our project and did not see nearly the same magnitude of benefit.

@chriscasola Mine was rather unscientific by observing activity monitor 馃槃 Running your command I'm getting similar results too, but I think what's happening is that node will still use and allocate the same amount of memory (which what time shows), but this PR allows more efficient garbage collection, so it's able to drop more memory when it's near the limit, preventing OOM likelihood.

As your example only took 2gb, node doesn't need to free up memory so it shows the same number. I'll update the PR title to better reflect this.

@bluwy bluwy changed the title Reduce AST memory usage Improve AST garbage collection Dec 22, 2022
@lukastaegert lukastaegert enabled auto-merge (squash) December 23, 2022 05:23
@lukastaegert lukastaegert merged commit b179695 into rollup:master Dec 23, 2022
10 checks passed
Copy link

This PR has been released as part of rollup@3.8.1. You can test it via npm install rollup.

@bluwy bluwy deleted the ast-lru branch December 23, 2022 08:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
None yet

Successfully merging this pull request may close these issues.

None yet

4 participants