require.ensure support

[jstcki]

This adds support for webpack's require.ensure which enables code splitting and async chunk loading.

When the script (not the require.ensure itself) is evaluated, webpack expects a window object; this is solved by supplying an empty object to node-eval's scope.

Note that it's not possible to use require.ensure during the static build (because it expects a DOM) but it's now possible to do code splitting with React Router like it's described in Ryan Florence's Welcome to the Future of Web Application Delivery or in something like componentDidMount. For example:

class MyLazyComponent extends React.Component {
  componentDidMount() {
    require.ensure([], function() {
      // the 'foo' module is split into a separate JS chunk and only loaded when this component is mounted
      var foo = require('foo');
      // use foo
    })
  }
  render() {
    return <div></div>;
  }
}

Potentially breaking if someone checks for window to do the client side render instead of document.

[jstcki]

This should fix #10.

[sergeylukin]

I'm glad I'm not alone trying to get this one sorted out.
I doubt your solution will work with splitting react-routes though as we have to call route require in order to get it's component and successfully generate a page anyways. I think what could work is just removing all instances of require.ensure throughout the files before they are compiled with webpack from within static-site-generator-webpack-plugin with regex similar to:

source.replace(/require\.ensure\(\[\],\s?function\s?\(require\)\s?{([\s\S.]*)}\)/, '$1');

So replace this:

require.ensure([], function (require) {
  callback(null, {
    component: require('./components/Index'),
  })
})

with:

callback(null, {
  component: require('./components/Index'),
})

What I cannot make working is the actual replace before files are compiled from within static-site-generator-webpack-plugin plugin. Any help is appreciated.

[jstcki]

I don't think that would work. Because once you're in the plugin's evaluate, the bundle is already compiled and you can't just require components anymore.

[sergeylukin]

Agree, that won't work.

[jstcki]

Pinging @markdalgleish: While this doesn't enable statically rendering async routes, it provides a fix for cases where require.ensure is used. I've successfully used this already in production, so I think it should be good to merge. Thanks for considering!

[jonathaningram]

FWIW I tried changing the scope to {window: {}} and it didn't work because the webpack require code needed document too, but then also it needed document.getElementsByTagName, etc. etc. But if it works for you, that's interesting.

[jstcki]

@jonathaningram: require.ensure doesn't need document; it just attaches webpackJsonp to window. If this weren't the case, the test would fail.

Probably you did something else like ReactDOM.render()? Or maybe you used webpack@2 and that has a different behavior?

[jonathaningram]

@herrstucki I am using webpack 1.12.13 and webpack-dev-server 1.14.1. Below is the JS file that was generated for my really basic index.js with nothing but an on-demand require in it.

require(['./SimpleComponent'], function(Com) {
  console.log(Com)
})

export default (locals, callback) => {
  callback(null, 'todo')
}

You can see it's doing the hot module thing and using document so that could be where mine fails, but it's also calling __webpack_require__.e /* require */ at the end of the file and that function uses document too.

(function webpackUniversalModuleDefinition(root, factory) {
  if (typeof exports === 'object' && typeof module === 'object')
    module.exports = factory()
  else if (typeof define === 'function' && define.amd)
    define([], factory)
  else {
    var a = factory()
    for (var i in a)(typeof exports === 'object' ? exports : root)[i] = a[i]
  }
})(this, function() {
  return (function(modules) { // webpackBootstrap // install a JSONP callback for chunk loading
      var parentJsonpFunction = window['webpackJsonp']
      window['webpackJsonp'] = function webpackJsonpCallback(chunkIds, moreModules) { // add 'moreModules' to the modules object, // then flag all 'chunkIds' as loaded and fire callback
        var moduleId, chunkId, i = 0,
          callbacks = []
        for (; i < chunkIds.length; i++) {
          chunkId = chunkIds[i]
          if (installedChunks[chunkId])
            callbacks.push.apply(callbacks, installedChunks[chunkId])
          installedChunks[chunkId] = 0
        }
        for (moduleId in moreModules) {
          modules[moduleId] = moreModules[moduleId]
        }
        if (parentJsonpFunction) parentJsonpFunction(chunkIds, moreModules)
        while (callbacks.length)
          callbacks.shift().call(null, __webpack_require__)
      }
      var parentHotUpdateCallback = this['webpackHotUpdate']
      this['webpackHotUpdate'] =
        function webpackHotUpdateCallback(chunkId, moreModules) { // eslint-disable-line no-unused-vars
          hotAddUpdateChunk(chunkId, moreModules)
          if (parentHotUpdateCallback) parentHotUpdateCallback(chunkId, moreModules)
        }

      function hotDownloadUpdateChunk(chunkId) { // eslint-disable-line no-unused-vars
        var head = document.getElementsByTagName('head')[0]
        var script = document.createElement('script')
        script.type = 'text/javascript'
        script.charset = 'utf-8'
        script.src = __webpack_require__.p + '' + chunkId + '.' + hotCurrentHash + '.hot-update.js'
        head.appendChild(script)
      }

      function hotDownloadManifest(callback) { // eslint-disable-line no-unused-vars
        if (typeof XMLHttpRequest === 'undefined')
          return callback(new Error('No browser support'))
        try {
          var request = new XMLHttpRequest()
          var requestPath = __webpack_require__.p + '' + hotCurrentHash + '.hot-update.json'
          request.open('GET', requestPath, true)
          request.timeout = 10000
          request.send(null)
        } catch (err) {
          return callback(err)
        }
        request.onreadystatechange = function() {
          if (request.readyState !== 4) return
          if (request.status === 0) { // timeout
            callback(new Error('Manifest request to ' + requestPath + ' timed out.'))
          } else if (request.status === 404) { // no update available
            callback()
          } else if (request.status !== 200 && request.status !== 304) { // other failure
            callback(new Error('Manifest request to ' + requestPath + ' failed.'))
          } else { // success
            try {
              var update = JSON.parse(request.responseText)
            } catch (e) {
              callback(e)
              return
            }
            callback(null, update)
          }
        }
      }
      var hotApplyOnUpdate = true
      var hotCurrentHash = '85e82da68c44b4323f09' // eslint-disable-line no-unused-vars
      var hotCurrentModuleData = {}
      var hotCurrentParents = [] // eslint-disable-line no-unused-vars
      function hotCreateRequire(moduleId) { // eslint-disable-line no-unused-vars
        var me = installedModules[moduleId]
        if (!me) return __webpack_require__
        var fn = function(request) {
          if (me.hot.active) {
            if (installedModules[request]) {
              if (installedModules[request].parents.indexOf(moduleId) < 0)
                installedModules[request].parents.push(moduleId)
              if (me.children.indexOf(request) < 0)
                me.children.push(request)
            } else hotCurrentParents = [moduleId]
          } else {
            console.warn('[HMR] unexpected require(' + request + ') from disposed module ' + moduleId)
            hotCurrentParents = []
          }
          return __webpack_require__(request)
        }
        for (var name in __webpack_require__) {
          if (Object.prototype.hasOwnProperty.call(__webpack_require__, name)) {
            Object.defineProperty(fn, name, (function(name) {
              return {
                configurable: true,
                enumerable: true,
                get: function() {
                  return __webpack_require__[name]
                },
                set: function(value) {
                  __webpack_require__[name] = value
                }
              }
            }(name)))
          }
        }
        Object.defineProperty(fn, 'e', {
          enumerable: true,
          value: function(chunkId, callback) {
            if (hotStatus === 'ready')
              hotSetStatus('prepare')
            hotChunksLoading++
            __webpack_require__.e(chunkId, function() {
              try {
                callback.call(null, fn)
              } finally {
                finishChunkLoading()
              }

              function finishChunkLoading() {
                hotChunksLoading--
                if (hotStatus === 'prepare') {
                  if (!hotWaitingFilesMap[chunkId]) {
                    hotEnsureUpdateChunk(chunkId)
                  }
                  if (hotChunksLoading === 0 && hotWaitingFiles === 0) {
                    hotUpdateDownloaded()
                  }
                }
              }
            })
          }
        })
        return fn
      }

      function hotCreateModule(moduleId) { // eslint-disable-line no-unused-vars
        var hot = { // private stuff
          _acceptedDependencies: {},
          _declinedDependencies: {},
          _selfAccepted: false,
          _selfDeclined: false,
          _disposeHandlers: [], // Module API
          active: true,
          accept: function(dep, callback) {
            if (typeof dep === 'undefined')
              hot._selfAccepted = true
            else if (typeof dep === 'function')
              hot._selfAccepted = dep
            else if (typeof dep === 'object')
              for (var i = 0; i < dep.length; i++)
                hot._acceptedDependencies[dep[i]] = callback
            else
              hot._acceptedDependencies[dep] = callback
          },
          decline: function(dep) {
            if (typeof dep === 'undefined')
              hot._selfDeclined = true
            else if (typeof dep === 'number')
              hot._declinedDependencies[dep] = true
            else
              for (var i = 0; i < dep.length; i++)
                hot._declinedDependencies[dep[i]] = true
          },
          dispose: function(callback) {
            hot._disposeHandlers.push(callback)
          },
          addDisposeHandler: function(callback) {
            hot._disposeHandlers.push(callback)
          },
          removeDisposeHandler: function(callback) {
            var idx = hot._disposeHandlers.indexOf(callback)
            if (idx >= 0) hot._disposeHandlers.splice(idx, 1)
          }, // Management API
          check: hotCheck,
          apply: hotApply,
          status: function(l) {
            if (!l) return hotStatus
            hotStatusHandlers.push(l)
          },
          addStatusHandler: function(l) {
            hotStatusHandlers.push(l)
          },
          removeStatusHandler: function(l) {
            var idx = hotStatusHandlers.indexOf(l)
            if (idx >= 0) hotStatusHandlers.splice(idx, 1)
          }, //inherit from previous dispose call
          data: hotCurrentModuleData[moduleId]
        }
        return hot
      }
      var hotStatusHandlers = []
      var hotStatus = 'idle'

      function hotSetStatus(newStatus) {
        hotStatus = newStatus
        for (var i = 0; i < hotStatusHandlers.length; i++)
          hotStatusHandlers[i].call(null, newStatus)
      } // while downloading
      var hotWaitingFiles = 0
      var hotChunksLoading = 0
      var hotWaitingFilesMap = {}
      var hotRequestedFilesMap = {}
      var hotAvailibleFilesMap = {}
      var hotCallback // The update info
      var hotUpdate, hotUpdateNewHash

      function toModuleId(id) {
        var isNumber = (+id) + '' === id
        return isNumber ? +id : id
      }

      function hotCheck(apply, callback) {
        if (hotStatus !== 'idle') throw new Error('check() is only allowed in idle status')
        if (typeof apply === 'function') {
          hotApplyOnUpdate = false
          callback = apply
        } else {
          hotApplyOnUpdate = apply
          callback = callback || function(err) {
            if (err) throw err
          }
        }
        hotSetStatus('check')
        hotDownloadManifest(function(err, update) {
          if (err) return callback(err)
          if (!update) {
            hotSetStatus('idle')
            callback(null, null)
            return
          }
          hotRequestedFilesMap = {}
          hotAvailibleFilesMap = {}
          hotWaitingFilesMap = {}
          for (var i = 0; i < update.c.length; i++)
            hotAvailibleFilesMap[update.c[i]] = true
          hotUpdateNewHash = update.h
          hotSetStatus('prepare')
          hotCallback = callback
          hotUpdate = {}
          for (var chunkId in installedChunks) { // eslint-disable-line no-lone-blocks
            /*globals chunkId */
            hotEnsureUpdateChunk(chunkId)
          }
          if (hotStatus === 'prepare' && hotChunksLoading === 0 && hotWaitingFiles === 0) {
            hotUpdateDownloaded()
          }
        })
      }

      function hotAddUpdateChunk(chunkId, moreModules) { // eslint-disable-line no-unused-vars
        if (!hotAvailibleFilesMap[chunkId] || !hotRequestedFilesMap[chunkId])
          return
        hotRequestedFilesMap[chunkId] = false
        for (var moduleId in moreModules) {
          if (Object.prototype.hasOwnProperty.call(moreModules, moduleId)) {
            hotUpdate[moduleId] = moreModules[moduleId]
          }
        }
        if (--hotWaitingFiles === 0 && hotChunksLoading === 0) {
          hotUpdateDownloaded()
        }
      }

      function hotEnsureUpdateChunk(chunkId) {
        if (!hotAvailibleFilesMap[chunkId]) {
          hotWaitingFilesMap[chunkId] = true
        } else {
          hotRequestedFilesMap[chunkId] = true
          hotWaitingFiles++
          hotDownloadUpdateChunk(chunkId)
        }
      }

      function hotUpdateDownloaded() {
        hotSetStatus('ready')
        var callback = hotCallback
        hotCallback = null
        if (!callback) return
        if (hotApplyOnUpdate) {
          hotApply(hotApplyOnUpdate, callback)
        } else {
          var outdatedModules = []
          for (var id in hotUpdate) {
            if (Object.prototype.hasOwnProperty.call(hotUpdate, id)) {
              outdatedModules.push(toModuleId(id))
            }
          }
          callback(null, outdatedModules)
        }
      }

      function hotApply(options, callback) {
        if (hotStatus !== 'ready') throw new Error('apply() is only allowed in ready status')
        if (typeof options === 'function') {
          callback = options
          options = {}
        } else if (options && typeof options === 'object') {
          callback = callback || function(err) {
            if (err) throw err
          }
        } else {
          options = {}
          callback = callback || function(err) {
            if (err) throw err
          }
        }

        function getAffectedStuff(module) {
          var outdatedModules = [module]
          var outdatedDependencies = {}
          var queue = outdatedModules.slice()
          while (queue.length > 0) {
            var moduleId = queue.pop()
            var module = installedModules[moduleId]
            if (!module || module.hot._selfAccepted)
              continue
            if (module.hot._selfDeclined) {
              return new Error('Aborted because of self decline: ' + moduleId)
            }
            if (moduleId === 0) {
              return
            }
            for (var i = 0; i < module.parents.length; i++) {
              var parentId = module.parents[i]
              var parent = installedModules[parentId]
              if (parent.hot._declinedDependencies[moduleId]) {
                return new Error('Aborted because of declined dependency: ' + moduleId + ' in ' + parentId)
              }
              if (outdatedModules.indexOf(parentId) >= 0) continue
              if (parent.hot._acceptedDependencies[moduleId]) {
                if (!outdatedDependencies[parentId])
                  outdatedDependencies[parentId] = []
                addAllToSet(outdatedDependencies[parentId], [moduleId])
                continue
              }
              delete outdatedDependencies[parentId]
              outdatedModules.push(parentId)
              queue.push(parentId)
            }
          }
          return [outdatedModules, outdatedDependencies]
        }

        function addAllToSet(a, b) {
          for (var i = 0; i < b.length; i++) {
            var item = b[i]
            if (a.indexOf(item) < 0)
              a.push(item)
          }
        } // at begin all updates modules are outdated // the 'outdated' status can propagate to parents if they don't accept the children
        var outdatedDependencies = {}
        var outdatedModules = []
        var appliedUpdate = {}
        for (var id in hotUpdate) {
          if (Object.prototype.hasOwnProperty.call(hotUpdate, id)) {
            var moduleId = toModuleId(id)
            var result = getAffectedStuff(moduleId)
            if (!result) {
              if (options.ignoreUnaccepted)
                continue
              hotSetStatus('abort')
              return callback(new Error('Aborted because ' + moduleId + ' is not accepted'))
            }
            if (result instanceof Error) {
              hotSetStatus('abort')
              return callback(result)
            }
            appliedUpdate[moduleId] = hotUpdate[moduleId]
            addAllToSet(outdatedModules, result[0])
            for (var moduleId in result[1]) {
              if (Object.prototype.hasOwnProperty.call(result[1], moduleId)) {
                if (!outdatedDependencies[moduleId])
                  outdatedDependencies[moduleId] = []
                addAllToSet(outdatedDependencies[moduleId], result[1][moduleId])
              }
            }
          }
        } // Store self accepted outdated modules to require them later by the module system
        var outdatedSelfAcceptedModules = []
        for (var i = 0; i < outdatedModules.length; i++) {
          var moduleId = outdatedModules[i]
          if (installedModules[moduleId] && installedModules[moduleId].hot._selfAccepted)
            outdatedSelfAcceptedModules.push({
              module: moduleId,
              errorHandler: installedModules[moduleId].hot._selfAccepted
            })
        } // Now in 'dispose' phase
        hotSetStatus('dispose')
        var queue = outdatedModules.slice()
        while (queue.length > 0) {
          var moduleId = queue.pop()
          var module = installedModules[moduleId]
          if (!module) continue
          var data = {} // Call dispose handlers
          var disposeHandlers = module.hot._disposeHandlers
          for (var j = 0; j < disposeHandlers.length; j++) {
            var cb = disposeHandlers[j]
            cb(data)
          }
          hotCurrentModuleData[moduleId] = data // disable module (this disables requires from this module)
          module.hot.active = false // remove module from cache
          delete installedModules[moduleId] // remove 'parents' references from all children
          for (var j = 0; j < module.children.length; j++) {
            var child = installedModules[module.children[j]]
            if (!child) continue
            var idx = child.parents.indexOf(moduleId)
            if (idx >= 0) {
              child.parents.splice(idx, 1)
            }
          }
        } // remove outdated dependency from module children
        for (var moduleId in outdatedDependencies) {
          if (Object.prototype.hasOwnProperty.call(outdatedDependencies, moduleId)) {
            var module = installedModules[moduleId]
            var moduleOutdatedDependencies = outdatedDependencies[moduleId]
            for (var j = 0; j < moduleOutdatedDependencies.length; j++) {
              var dependency = moduleOutdatedDependencies[j]
              var idx = module.children.indexOf(dependency)
              if (idx >= 0) module.children.splice(idx, 1)
            }
          }
        } // Not in 'apply' phase
        hotSetStatus('apply')
        hotCurrentHash = hotUpdateNewHash // insert new code
        for (var moduleId in appliedUpdate) {
          if (Object.prototype.hasOwnProperty.call(appliedUpdate, moduleId)) {
            modules[moduleId] = appliedUpdate[moduleId]
          }
        } // call accept handlers
        var error = null
        for (var moduleId in outdatedDependencies) {
          if (Object.prototype.hasOwnProperty.call(outdatedDependencies, moduleId)) {
            var module = installedModules[moduleId]
            var moduleOutdatedDependencies = outdatedDependencies[moduleId]
            var callbacks = []
            for (var i = 0; i < moduleOutdatedDependencies.length; i++) {
              var dependency = moduleOutdatedDependencies[i]
              var cb = module.hot._acceptedDependencies[dependency]
              if (callbacks.indexOf(cb) >= 0) continue
              callbacks.push(cb)
            }
            for (var i = 0; i < callbacks.length; i++) {
              var cb = callbacks[i]
              try {
                cb(outdatedDependencies)
              } catch (err) {
                if (!error)
                  error = err
              }
            }
          }
        } // Load self accepted modules
        for (var i = 0; i < outdatedSelfAcceptedModules.length; i++) {
          var item = outdatedSelfAcceptedModules[i]
          var moduleId = item.module
          hotCurrentParents = [moduleId]
          try {
            __webpack_require__(moduleId)
          } catch (err) {
            if (typeof item.errorHandler === 'function') {
              try {
                item.errorHandler(err)
              } catch (err) {
                if (!error)
                  error = err
              }
            } else if (!error)
              error = err
          }
        } // handle errors in accept handlers and self accepted module load
        if (error) {
          hotSetStatus('fail')
          return callback(error)
        }
        hotSetStatus('idle')
        callback(null, outdatedModules)
      } // The module cache
      var installedModules = {} // object to store loaded and loading chunks // '0' means 'already loaded' // Array means 'loading', array contains callbacks
      var installedChunks = {
          0: 0
        } // The require function
      function __webpack_require__(moduleId) { // Check if module is in cache
        if (installedModules[moduleId])
          return installedModules[moduleId].exports // Create a new module (and put it into the cache)
        var module = installedModules[moduleId] = {
            exports: {},
            id: moduleId,
            loaded: false,
            hot: hotCreateModule(moduleId),
            parents: hotCurrentParents,
            children: []
          } // Execute the module function
        modules[moduleId].call(module.exports, module, module.exports, hotCreateRequire(moduleId)) // Flag the module as loaded
        module.loaded = true // Return the exports of the module
        return module.exports
      } // This file contains only the entry chunk. // The chunk loading function for additional chunks
      __webpack_require__.e = function requireEnsure(chunkId, callback) { // '0' is the signal for 'already loaded'
          if (installedChunks[chunkId] === 0)
            return callback.call(null, __webpack_require__) // an array means 'currently loading'.
          if (installedChunks[chunkId] !== undefined) {
            installedChunks[chunkId].push(callback)
          } else { // start chunk loading
            installedChunks[chunkId] = [callback]
            var head = document.getElementsByTagName('head')[0]
            var script = document.createElement('script')
            script.type = 'text/javascript'
            script.charset = 'utf-8'
            script.async = true
            script.src = __webpack_require__.p + '' + chunkId + '.assets/' + hotCurrentHash + '.js'
            head.appendChild(script)
          }
        } // expose the modules object (__webpack_modules__)
      __webpack_require__.m = modules // expose the module cache
      __webpack_require__.c = installedModules // __webpack_public_path__
      __webpack_require__.p = '' // __webpack_hash__
      __webpack_require__.h = function() {
          return hotCurrentHash
        } // Load entry module and return exports
      return hotCreateRequire(0)(0)
    })
    ([
      /* 0 */
      function(module, exports, __webpack_require__) {
        Object.defineProperty(exports, '__esModule', {
          value: true
        })
        if (typeof window !== 'undefined') {
          __webpack_require__.e /* require */ (1, function(__webpack_require__) {
            var __WEBPACK_AMD_REQUIRE_ARRAY__ = [__webpack_require__(1)]
              (function(Com) {
                console.log(Com)
              }.apply(null, __WEBPACK_AMD_REQUIRE_ARRAY__))
          })
        }

        exports.default = function(locals, callback) {
          callback(null, 'todo')
        }
      }
    ])
})

[jstcki]

@jonathaningram I'm sorry, my previous statement wasn't accurate: require.ensure itself does use document. But in the case where it's not called, adding window: {} to the context is sufficient because that's where webpack puts the webpackJsonp function.

So, a simple React case where this works:

class Foo extends React.Component {
  componentDidMount() {
    // Does not get called during static site build
    require.ensure([], () => {
      // This chunk will only be loaded when this component is mounted
      const foo = require('./foo');
      // ...
    })
  }
  render() {
    return <div />;
  }
}

// In the static route handler …
ReactDOM.renderToString(<Foo />);

Now, it probably won't work with React Router's getComponent route prop because that gets called during the static build.

I think it could work if webpack allowed a separate entry point for the static build with target: 'node' but as far as I know, it's not possible to have different targets for different entry points.

[jonathaningram]

@herrstucki thanks for the clarification. Correct, in my tests I was putting the require.ensure directly at the top of my index.js entry.

[markdalgleish]

@herrstucki this looks like a nice tidy PR. My only concern is that this could be a breaking chage if you have any code in your bundle that detects whether you're in a node context or not by evaluating typeof window !== 'undefined'. What do you think?

[jstcki]

Yes, that's certainly a concern.

The example you provide in the README uses typeof document === 'undefined', so probably most people will just copy that 😁 But to be safe, this should be treated as a breaking change, so people don't get caught by surprise.

Or we could allow to define the eval scope via the plugin's options, so it's opt-in.

[markdalgleish]

I think allowing a scope object to be passed in as an option is probably the safest and most flexible way forward here, but it will probably require an API change since this plugin arguably has too many arguments already. I'll need to think about this a little.

[markdalgleish]

@herrstucki how about we deprecate the current API in favour of something more flexible like this, which allows for a scope parameter?

new StaticSiteGeneratorPlugin({
  input: 'main',
  paths: [
    ...
  ],
  locals: {
    ...
  },
  scope: {
    window: {}
  }
})

[markdalgleish]

@herrstucki maybe in this PR it'd be a good idea to add an extra scope parameter to the plugin's constructor, so that you can pass in { window: {} }, and then I'll work on a new config-based API. I'll maintain backwards compatibility with the current API, though, since it'll be easy to tell them apart. What do you think?

[markdalgleish]

Actually, your commits are good enough to merge, I'll make the changes myself. Thanks for the really great PR 😄

[markdalgleish]

Just published v2.1.0 which allows a custom scope object to be provided, which means you'll need to manually provide { window: {} } as the last argument.