New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incremental builds #191

Closed
Victorystick opened this Issue Oct 15, 2015 · 12 comments

Comments

Projects
None yet
9 participants
@Victorystick
Member

Victorystick commented Oct 15, 2015

I've started asking myself what incremental builds would look like in Rollup. What data can we keep, and what do we need to change? Consider something like this:

rollup.rollup( options ).then(incrementallyRebuild);

function incrementallyRebuild( bundle ) {
  // Write/generate the current bundle.
  bundle.write( options );

  // If any of the files included by the bundle changes...
  return anyFileChanged( bundle.files ).then( file => {
    // rebuild the bundle with the changes to that file.
    return bundle.invalidateId( file ).rollup( options ).then( incrementallyRebuild );
  });
}

How should we make sure to do as little work as possible for each file change? Generally, only one of the files included by the bundle will change at a time. If we can keep the isolated data for all modules and invalidate the module that changed, perhaps we can speed up the build process.

We would have to:

  1. copy reusable data from some previous Bundle's analysis
  2. invalidate a module id of the changed file
  3. reload that id (presumably a single file)
  4. perform isolated analysis of the resulting module
  5. create a bundle
    1. include/drop modules due to a change to an import/export statement
    2. business as usual

We'd need to separate reusable data from the bundle-specific data. I don't know how much work that'd require, but I imagine quite a lot.

@Rich-Harris

This comment has been minimized.

Contributor

Rich-Harris commented Oct 15, 2015

I was thinking along similar lines – having a persistent bundle object and gradually replacing modules within it ought to be highly efficient.

There are some challenges:

  • references would need to be rebound. That means the declarations in the module being replaced have to know about their references (at the moment, the reference knows about the declaration but not vice versa). The alternative is to rebind all references, which is almost certainly a good deal costlier
  • If a imports c, and b imports c, and b is changed so that it no longer imports c, either c or the bundle needs to understand that it is still depended upon
  • The entry module is a special case because of its export statements

Something that would go hand-in-hand with this: have been thinking about how to eliminate this process of repeatedly marking side-effects until we've covered them all. It would be better if declarations already knew about side-effects that affected them, so that in a situation like this...

var a = {};
a.foo = 'bar';

...as soon as the first statement was included, the second would immediately be included as well rather than having to wait for that later analysis. I have a hunch that this will be key to #179. It all depends on declarations having better knowledge of how they are used in the bundle, which ties it to this issue. (It makes sense in my head, anyway...)

It might also be good to have a separate caching mechanism that allowed you to save analysis to disk, and allowed analysis to be shared between bundles. At the moment, when you build Rollup itself, it has to read every file twice (once for the normal version, once for the browser build), run each file through Babel twice, and conduct the parsing/analysis twice. It would be great if it could read some data from .rollup/cache.json or whatever, and then determine whether to use the cached transformed code and AST/analysis based on whether a checksum matches, or something. Not entirely straightforward since you need to account for different (and changeable) sets of transformations, but nor do I think it's impossible.

One point on the code sample above: I think we should avoid bundle.files, since it's not really filesystem-specific (we could use this stuff for the browser build as well) – bundle.modules should have all the info we need.

@englercj

This comment has been minimized.

englercj commented Nov 26, 2015

Any update on this, the biggest reason not to use this for me right now is that it requires a long time to rebuild; 5-10 seconds.

@Rich-Harris

This comment has been minimized.

Contributor

Rich-Harris commented Nov 27, 2015

@englercj not yet – I've been AFK for a while but this is definitely one of the biggest priorities

@bodokaiser

This comment has been minimized.

bodokaiser commented Mar 22, 2016

Are there only updates on this?

I have react as a dependency and the build time (on each js request) sums up to ~2s.

Are there any ways to speed this up (e.g. just include external libs like react instead of parsing them)?

@Ventajou

This comment has been minimized.

Ventajou commented Mar 31, 2016

I definitely would love this feature as well, I use Gulp to watch the files and build other assets, but any time I change one of my files, all the 3rd party libraries I use also get pulled.

@Rich-Harris

This comment has been minimized.

Contributor

Rich-Harris commented Jun 9, 2016

We can finally close this issue thanks to @TrySound 🎉 . Ended up taking a slightly different route to the one described above, which is a little simpler to implement – rather than invalidating modules in a stateful bundle, we just start from scratch but re-using the transformations, AST and sourcemap chains for any modules that haven't changed since the previous run (using the cache: bundle option, where bundle is the result of the previous build).

This isn't quite as efficient as the stateful approach would have been, but a) it's a hell of a lot easier to reason about, which will mean fewer bugs to contend with, and b) shipping an imperfect implementation beats not shipping a perfect one.

I encourage everyone to try it out! Easiest way is by using the --watch (or -w) flag with the CLI.

@Rich-Harris Rich-Harris closed this Jun 9, 2016

@StreetStrider

This comment has been minimized.

StreetStrider commented Nov 26, 2016

@Rich-Harris, glad this'd been resolved. Should this be reflected in Readme? I think a small section on a hi-level approach of this feature would be good.

@snuggs

This comment has been minimized.

snuggs commented Nov 27, 2016

Possibly the wiki is a better place @StreetStrider @Rich-Harris? My $0.02

@snuggs

This comment has been minimized.

snuggs commented Nov 27, 2016

I'm curious @Rich-Harris @Victorystick what is the difference in this approach and "Hot Module loading"? I'm not sure I even have a grasp on the term itself. I get it ... kinda. I use a tool called browsersync in development that "hot reloads" and injects css diffs. Now starting to hear that in the javascript community. Thanks in advance for the teaching.

@yairEO

This comment has been minimized.

yairEO commented Jan 10, 2017

I can't say the "cache" trick saves much time..
The bundle file is about 8,000 lines of code and takes ~9.5s
that is just the bundle of my tests.. every file save it happens twice, so I "enjoy" waiting about 20s for each save just to see my tests. I might end up splitting the bundle into multiple bundles and concat the result, if rollup can't create the bundle quicker if I change the tiniest thing and then save. it's pretty annoying.

@StreetStrider

This comment has been minimized.

StreetStrider commented Jan 10, 2017

@yairEO, I haven't been using Rollup with big bundles before, I'll post here when I will do.

The more I think of caching for things like Rollup the more I come to the point that we need some form of «virtual file system layer» which will do all the caching. Then, different systems, like Rollup, browserify, postcss could use it transparently. Something similar could be achieved in Gulp now, for instance, when you transform a bunch of PNGs or something. The main pecularity here is «parallel» input — each file can be processed independently, so gulp.src can be cached. Unfortunately, that's not possible for bundlers, where we usually have single entry point and then bundler do the module resolution logic. We can't just swap fs under, because bundles may use it in some different manners.

If anyone knows something like such «vfs layer», or maybe developing one, please, let me know, I will glad to use it or help to cooperate and popularize.

@balazs-endresz balazs-endresz referenced this issue Jan 30, 2017

Merged

ES6 imports #30

17 of 21 tasks complete
@jchook

This comment has been minimized.

jchook commented Jun 8, 2017

I recently converted my project from Webpack to Rollup. Here is what I found..

  1. Initial build time dropped from 60s to 40s
  2. Incremental builds increased from 4s to 11s

Personally, I care a LOT more about incremental build time since it dramatically affects productivity. It seems that no treeshaking, and per-file invalidation combined with a VFS layer would be the ticket to extremely fast incremental builds, giving rollup another advantage over Webpack.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment