Does browserify make use of a cache option?

[dustMason]

Hi,

I've been perusing the source of both browserify and watchify hoping to find a good way to speed up my builds using a cache. I want to take advantage of watchify's caching feature but without having to run a long-running process.

I found a couple things which make it look like browserify might have some related features. For example, https://github.com/substack/node-browserify/blob/47c6933e711567de761c73fe81165223c7873433/test/double_bundle_parallel_cache.js contains a cache option. And https://github.com/substack/watchify/blob/master/index.js shows cache and packageCache as options taken from the browserify instance but I haven't found where those are actually used. Browserify's source doesn't mention a cache option that I can see and watchify doesn't appear to actually make use of the cache that it maintains.

The reason behind this is that my team is migrating our stack to Docker and for us the best workflow would involve watching files on our host machine, then triggering a full rebuild of our assets in a container upon changes.

Thanks!

[ghost]

The caching options are forwarded along to module-deps: https://github.com/substack/module-deps/blob/master/index.js

You could roll your own cache from disk or some such but it would probably be much simpler to just run watchify from inside the docker container. Maybe just use -v to mount the local source directory during development? Otherwise I don't see what's so wrong with running watchify on your own machine instead of in docker. npm is really good for making reproducible builds.

[dustMason]

Using -v was the first approach I tried but since some of us here are on macs (myself included) that means Virtualbox shared folders or NFS shares via Vagrant or boot2docker. Even after trying to configure gaze every which way I could, I've never been able to get watchify to work while watching for edits made on the other end of a virtualbox share. Likewise for NFS shares from my mac which seem to cause the VMs to fire events constantly.

Running watchify locally is, of course, an option but our gulpfile includes several image processing steps that for some reason produce slightly different results on some machines. We like the idea of putting that issue to rest by developing in matching environments.

I tried out https://github.com/benbria/uber-watchify which adds a cache file option, but I worry that it might not be well maintained going forward.

Thanks for your reply. Maybe this issue will gather some input from others in the same situation too.

[startswithaj]

@dustMason,

I'm not really sure if this will help you but I use browserify with a caching mechanism that I stole from watchify.

  # Browserify
  cache = {}
  pkgCache = {}
  b = browserify
    extensions: ".coffee"
    debug: true
    cache: cache
    packageCache: pkgCache
    fullPaths: true #need this for caching
  b.transform coffeeify
  b.add paths.entry

  # browserify submodule module-deps uses b.cache so we create a cache for it
  b.on "dep", (dep) ->
    cache[dep.id] = dep  if typeof dep.id is "string"

  bundleStream = b.bundle (err, src) ->
    if err then console.error "initial browserify bundling error: \n  %s \n  %s %s:%s", err.toString(), err.file, err.location?.first_line, err.location?.first_column, terminalAlertBadge
    b.source = src

    watcher = chokidar.watch paths.client, { ignored: /^\./, persistent: true }
    # custom nodemodules
    watcher.add [
      "node_modules/blah-module-to-watch"
      "node_modules/blah-module-to-watch2"
    ]

    watcher.on 'change', (path) -> 
      console.log "#{path} changed, rebundling..."
      # remove the cache of the file thats changed
      if (cache) 
        delete cache[path]
      debounceBundle()

Watchify uses chokidar underneath so if that doesn't work for watching files on NFS or virtual box shares you will need to substitute it out for watch code that does.

[dustMason]

Thanks @startswithaj. I'm playing with the techniques you've demoed here but still can't get a working cache. Even when I set the cache option to a complete hash of compiled deps captured from a previous build, it seems to rebuild the whole thing anyway. I guess thats because calling add supersedes the cache?

Your example definitely points me in the right direction for implementing watchify to accomplish my goal but I still feel that a persistent on-disk cache is better so I'm going to keep working in that direction.

[startswithaj]

Do you have the fullPaths options set? ...Otherwise it has no way to reference the cache object for the corresponding file. dep.id is the fullpath to the file. If you have cache on and fullpaths set to false it doesn't work.

[dustMason]

I do have fullPaths set. The part I'm not understanding is at what point module-deps makes use of the cache object, and if using the entries option or .add() automatically invalidates that cache. Here's the relevant part of my gulpfile https://gist.github.com/dustMason/258ba9c55050987270b2 . It's incomplete, of course, and you can see an unused hash property on line 20, which is probably the wrong approach but its where i left off.

Even when I change the condition on line 18 to read (typeof dep.id === "string" && !cache[dep.id]) which should never bust the cache, the build doesn't appear to use the cache (or at least takes just as long as without).

[startswithaj]

As @substack mentioned before the cache is used by module-deps more specifically here in the readFile() method. https://github.com/substack/module-deps/blob/master/index.js#L158
Starting line 158:

Deps.prototype.readFile = function (file, id, pkg) {
    var self = this;
    var tr = through();
    if (this.cache && this.cache[file]) {
        tr.push(this.cache[file].source);
        tr.push(null);
        return tr;
    }
    var rs = fs.createReadStream(file);
    rs.on('error', function (err) { self.emit('error', err) });
    this.emit('file', file, id);
    return rs;
};

So basically if there is a cache and the cache has an index for the file its trying to retrieve, it returns the source from the cache object.

Your gist code does not write the cache file in the same way as uber-watchify. uber-watchify does some tricky stuff and I guess he's doing that for a reason.

https://github.com/benbria/uber-watchify/blob/master/index.js#L222

b.write = function(opts) {
        try {
            if (!opts) opts = {};
            fs.writeFileSync(cacheFile, '{');
            var first = true;
            for (var prop in cache) {
                if (cache.hasOwnProperty(prop)) {
                    if (first) first = false;
                    else fs.appendFileSync(cacheFile, ',');
                    fs.appendFileSync(cacheFile, JSON.stringify(prop) + ':' + JSON.stringify(cache[prop]))
                }
            }
            fs.appendFileSync(cacheFile, '}');
        } catch (err) {
            b.emit('log', 'Erroring writing cache file ', + err.message);
        }
    };

[startswithaj]

Also, you might be already be, but I think if you npm install -g node-debug and then run node-debug $(which gulp) task you might get a better look at whats going on with your cache.