New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: Dll-like Plugin for an external context - external manifest, dynamic module-map injection and loading #2592

Open
niieani opened this Issue Jun 2, 2016 · 21 comments

Comments

Projects
None yet
@niieani
Contributor

niieani commented Jun 2, 2016

Currently, the DllPlugin and DllReferencePlugin allows loading external modules from separately generated bundles. The problem is that we need to provide a manifest with all the modules being bundled at the time of bundling. This is limiting in certain scenarios, where we would like to dynamically load modules which are not known at the time of build (user input, external files).

Example use case:

Imagine an docs-browser app, that dynamically loads the documentation text and executable examples, based on a JSON file that lists where they are contained. The documentation and examples are stored and built separately. They both can grow (more files are added) and we shouldn't need to republish the entire documentation website after every new addition to the documentation.

What I propose is that we delegate the manifest to the Dll bundle itself. This way the following could be possible:

require('https://domain.com/dll-bundle.js#jquery');

Assuming the dll-bundle.js file is a bundle that contains the module jquery, the above require would download the file (proper CORS required), load the manifest from it (listing all the mappings of modules and merging them with the preloaded ones) and finally load the code from the bundled module jquery.

This would require the module IDs to be long and random (e.g. UUIDs) so that collisions do not occur, but they could technically also be generated based on prefixed some seeded value.

This would solve questions like #1421 or #150.

EDIT:
A helper method for requiring from an external bundle might make more sense, something like:

require.ensureExternal('https://domain.com/dll-bundle.js', function(require) {
  var $ = require('jquery');
})

// or even:
require.ensureExternal(dynamicallyGeneratedUrl, function(require) {
  var someModule = require(totallyDynamicName);
})

@bebraw bebraw added the enhancement label Jun 3, 2016

@sokra

This comment has been minimized.

Show comment
Hide comment
@sokra

sokra Jun 4, 2016

Member

webpack doesn't take care of the module loading. You need to use another library for the loading part. But you can use webpack to bundle the main file and the loaded files. For your use case you don't need complex constructs like DLLs. Just define a interface for the loaded scripts, expose it with the output.library function and load them with a script loader like scriptjs.

Example:

function loadDocumentation(name) {
  return new Promise(function(resolve, reject) {
    scriptjs("https://domain.com/documentation-" + name + ".js", function() {
      var doc = window["documentation-" + name];
      resolve(doc);
    });
  });
}

documentation part build with output.library = "documenation-" + name

Member

sokra commented Jun 4, 2016

webpack doesn't take care of the module loading. You need to use another library for the loading part. But you can use webpack to bundle the main file and the loaded files. For your use case you don't need complex constructs like DLLs. Just define a interface for the loaded scripts, expose it with the output.library function and load them with a script loader like scriptjs.

Example:

function loadDocumentation(name) {
  return new Promise(function(resolve, reject) {
    scriptjs("https://domain.com/documentation-" + name + ".js", function() {
      var doc = window["documentation-" + name];
      resolve(doc);
    });
  });
}

documentation part build with output.library = "documenation-" + name

@niieani

This comment has been minimized.

Show comment
Hide comment
@niieani

niieani Jun 4, 2016

Contributor

@sokra Indeed it doesn't take care of module loading, but it does take care of bundle/chunk loading. What I'm after is lazy-loading an external bundle without knowing what modules it specifically contains (at least not at the time of building) and then handling the newly loaded modules as if they're originally compiled in. I feel it is very similar to the DLL construct, with the small difference of not mapping the modules during build-time, but during run-time.

Contributor

niieani commented Jun 4, 2016

@sokra Indeed it doesn't take care of module loading, but it does take care of bundle/chunk loading. What I'm after is lazy-loading an external bundle without knowing what modules it specifically contains (at least not at the time of building) and then handling the newly loaded modules as if they're originally compiled in. I feel it is very similar to the DLL construct, with the small difference of not mapping the modules during build-time, but during run-time.

@niieani

This comment has been minimized.

Show comment
Hide comment
@niieani

niieani Jun 4, 2016

Contributor

@sokra Webpack wouldn't need to take care of loading anything from HTTP if the user had to provide a Promise or a Thunk that resolves to the contents of the bundle. This would also make it more flexible as to the source of the bundle.

Example:

var bundleContentsPromise = fetch('https://domain.com/dll-bundle.js');

require.ensureExternal(bundleContentsPromise, function(require) {
  var $ = require('jquery');
})
Contributor

niieani commented Jun 4, 2016

@sokra Webpack wouldn't need to take care of loading anything from HTTP if the user had to provide a Promise or a Thunk that resolves to the contents of the bundle. This would also make it more flexible as to the source of the bundle.

Example:

var bundleContentsPromise = fetch('https://domain.com/dll-bundle.js');

require.ensureExternal(bundleContentsPromise, function(require) {
  var $ = require('jquery');
})
@sokra

This comment has been minimized.

Show comment
Hide comment
@sokra

sokra Jun 5, 2016

Member
// dll-bundle-entry.js
module.exports = function(request) {
  switch(request) {
    case "jquery": return require("jquery");
    // ...
  }
};
// or: module.exports = require.context(...)
var bundleContentsPromise = fetch('https://domain.com/dll-bundle.js');
var requireInBundlePromise = bundleContentsPromise.then(evalContent);

requireInBundlePromise.then(function(require) {
  require("jquery");
});

You don't need any special support for the runtime. You dll-bundle is just a library which exposes a require-like API, but you could use any API.

If writing the entry point manually is too much work, you could generate it with a loader.

Member

sokra commented Jun 5, 2016

// dll-bundle-entry.js
module.exports = function(request) {
  switch(request) {
    case "jquery": return require("jquery");
    // ...
  }
};
// or: module.exports = require.context(...)
var bundleContentsPromise = fetch('https://domain.com/dll-bundle.js');
var requireInBundlePromise = bundleContentsPromise.then(evalContent);

requireInBundlePromise.then(function(require) {
  require("jquery");
});

You don't need any special support for the runtime. You dll-bundle is just a library which exposes a require-like API, but you could use any API.

If writing the entry point manually is too much work, you could generate it with a loader.

@niieani

This comment has been minimized.

Show comment
Hide comment
@niieani

niieani Jun 5, 2016

Contributor

@sokra That looks good, thanks. But won't the module IDs start to conflict when I import one Webpack project into another? They are stored in the global context, isn't that right?

Contributor

niieani commented Jun 5, 2016

@sokra That looks good, thanks. But won't the module IDs start to conflict when I import one Webpack project into another? They are stored in the global context, isn't that right?

@sokra

This comment has been minimized.

Show comment
Hide comment
@sokra

sokra Jun 5, 2016

Member

They are stored in the global context, isn't that right?

nope. They are stored inside the webpack runtime, which is in the entry chunk. bundles doesn't leak anything into the global scope except the jsonp function when using code splitting (you may rename it with a config option).

Member

sokra commented Jun 5, 2016

They are stored in the global context, isn't that right?

nope. They are stored inside the webpack runtime, which is in the entry chunk. bundles doesn't leak anything into the global scope except the jsonp function when using code splitting (you may rename it with a config option).

@mnzaki

This comment has been minimized.

Show comment
Hide comment
@mnzaki

mnzaki Dec 1, 2016

But require.ensure automatically loads bundles, and it would be really nice to have that functionality for DllReferencePlugin as well. Something like this:

require.ensure([], () => {
  require('somelibrary');
});

And in webpack config:

plugins: [
  new DllReferencePlugin({
    context: '.',
    manifest: require('../dll-manifest.json')
  })
]

Where the dll-manifest.json file contains a manifest generated by DllPlugin for a build that contains the somelibrary module.

If the DllReferencePlugin can cause this to autoload the DLL bundle, that would be awesome.

mnzaki commented Dec 1, 2016

But require.ensure automatically loads bundles, and it would be really nice to have that functionality for DllReferencePlugin as well. Something like this:

require.ensure([], () => {
  require('somelibrary');
});

And in webpack config:

plugins: [
  new DllReferencePlugin({
    context: '.',
    manifest: require('../dll-manifest.json')
  })
]

Where the dll-manifest.json file contains a manifest generated by DllPlugin for a build that contains the somelibrary module.

If the DllReferencePlugin can cause this to autoload the DLL bundle, that would be awesome.

@mnzaki

This comment has been minimized.

Show comment
Hide comment
@mnzaki

mnzaki Dec 1, 2016

Actually, require.ensure is unrelated. It would be nice if DllReferencePlugin, or perhaps a wrapper around it, can auto-load DLL bundles when modules are require()ed from them.

mnzaki commented Dec 1, 2016

Actually, require.ensure is unrelated. It would be nice if DllReferencePlugin, or perhaps a wrapper around it, can auto-load DLL bundles when modules are require()ed from them.

@senritsu

This comment has been minimized.

Show comment
Hide comment
@senritsu

senritsu Feb 9, 2017

@sokra are there any plans of supporting async loading in the DllReferencePlugin as outlined by @mnzaki at the moment?

As both external dependencies and async dependencies (via split points) are supported, the infrastructure, in theory at least, seems to support integrating a script loader like scriptjs to the point that a regular require.ensure call without any further adjustments to the rest of the pipeline would just work (tm), is that correct?

After reading the sources for the DllReferencePlugin and related classes i have an inkling of how it works, but am still not really at a point where i understand it enough to be able to really judge the complexity of something like this, so any thoughts on that matter from someone who knows webpack in and out would be much appreciated.

senritsu commented Feb 9, 2017

@sokra are there any plans of supporting async loading in the DllReferencePlugin as outlined by @mnzaki at the moment?

As both external dependencies and async dependencies (via split points) are supported, the infrastructure, in theory at least, seems to support integrating a script loader like scriptjs to the point that a regular require.ensure call without any further adjustments to the rest of the pipeline would just work (tm), is that correct?

After reading the sources for the DllReferencePlugin and related classes i have an inkling of how it works, but am still not really at a point where i understand it enough to be able to really judge the complexity of something like this, so any thoughts on that matter from someone who knows webpack in and out would be much appreciated.

@githoniel

This comment has been minimized.

Show comment
Hide comment
@githoniel

githoniel Feb 10, 2017

I need to share very large amount of code across several apps, and it's impossible to build them together in one project(they will be developed by different people and different time). In order to use the browser cache, reduce the file size in server and client cache,DllReferencePlugin is a very good idea.

But since shared code are in big size, load them with script tag in html will make first load very slow. supporting async loading DllReferencePlugin will resolve this.

I know use scriptjs or other module loader can do this, even use requirejs with 'requirejs(['module'], a => { //todo })'. but those loaders cannot read dependency,alias and path settings from webpack,a duplicate setting is need. it is so ugly and inconvenient

githoniel commented Feb 10, 2017

I need to share very large amount of code across several apps, and it's impossible to build them together in one project(they will be developed by different people and different time). In order to use the browser cache, reduce the file size in server and client cache,DllReferencePlugin is a very good idea.

But since shared code are in big size, load them with script tag in html will make first load very slow. supporting async loading DllReferencePlugin will resolve this.

I know use scriptjs or other module loader can do this, even use requirejs with 'requirejs(['module'], a => { //todo })'. but those loaders cannot read dependency,alias and path settings from webpack,a duplicate setting is need. it is so ugly and inconvenient

@shaikhspear16

This comment has been minimized.

Show comment
Hide comment
@shaikhspear16

shaikhspear16 Feb 23, 2017

@githoniel I am looking for a solution for the same. Did you find a way to achieve that?

For me, we create an angular 2 library such as a "Menu" that exposes a menu directive and host to to an internal CDN. We need to utilize this library from the CDN to all our other SPAs for the same reasons you mentioned above. It would be great to have some direction.

shaikhspear16 commented Feb 23, 2017

@githoniel I am looking for a solution for the same. Did you find a way to achieve that?

For me, we create an angular 2 library such as a "Menu" that exposes a menu directive and host to to an internal CDN. We need to utilize this library from the CDN to all our other SPAs for the same reasons you mentioned above. It would be great to have some direction.

@githoniel

This comment has been minimized.

Show comment
Hide comment
@githoniel

githoniel Feb 23, 2017

@shaikhspear16 as I metioned,let webpack export AMD Libaray and use requirejs to load them is the only way to do this as i know。

githoniel commented Feb 23, 2017

@shaikhspear16 as I metioned,let webpack export AMD Libaray and use requirejs to load them is the only way to do this as i know。

@CameronPlimsoll

This comment has been minimized.

Show comment
Hide comment
@CameronPlimsoll

CameronPlimsoll Mar 24, 2017

@githoniel do you have any sample code ?

CameronPlimsoll commented Mar 24, 2017

@githoniel do you have any sample code ?

@githoniel

This comment has been minimized.

Show comment
Hide comment
@githoniel

githoniel Mar 24, 2017

@CameronPlimsoll It's my pleasure.

1、In you dll generator webpack config,use assets-webpack-plugin to outp a JSON object to descript you output libaray, like

module.exports = merge(baseWebpackConfig, {
     entry: utils.getLibaryEntry(),
     output: {
         path: config.libraryPath,
         jsonpFunction: `webpackJsonp${Date.now()}`,  //!! to avoid webpackJsonp conflict with reference webpack 
         filename: 'bundle.[id].js',
         chunkFilename: 'chunk.[id].[chunkhash].js',
         publicPath: '../../',   //you public Path
         libraryTarget: 'commonjs2'  //output  cmd 
     },
    plugins: [
         new AssetsPlugin({
             filename: 'assets.json',
             fullPath: false,
             path: config.libraryPath,
             metadata: {
                 version: (new Date()).getTime()
             }
         }),
  1. In my project , I use SystemJS to load js from dll project, I wrote a simple plugin to insert SystemJS code to webpack bootstrap, but insert SystemJS to html template maybe easier. After that, you need to set map for Systemjs, like
return fetch(mapJsonLocation)
        .then(response => response.json())
        .then((asset) => {
            const map = {}
            if (asset.metadata) {
                delete asset.metadata
            }
            Object.keys(asset).forEach((key) => {
                const aliasKey = `${packageName}/${key}`
                const aliasPath = `${prefixPath}/${asset[key].js}`
                map[aliasKey] = aliasPath
            })
            SystemJS.config({
                map
            })
        })
  1. Now you can write in your file, Notice that async load is the only option, sync load nope, it is a little verbose
SystemJS.import('packageName/moduleName')
  .then((module) => {
     //todo 
  })
  1. If you use requirejs, Output AMD format libary and use
 requirejs(['module'], (module) => {})

UPDATE: If you use commnChunkPlugin to get a vendor output,you MUST load it before load any dll js file ! or you will get Error: (SystemJS) webpackJsonp1490526215268 is not defined LIKE @CameronPlimsoll

githoniel commented Mar 24, 2017

@CameronPlimsoll It's my pleasure.

1、In you dll generator webpack config,use assets-webpack-plugin to outp a JSON object to descript you output libaray, like

module.exports = merge(baseWebpackConfig, {
     entry: utils.getLibaryEntry(),
     output: {
         path: config.libraryPath,
         jsonpFunction: `webpackJsonp${Date.now()}`,  //!! to avoid webpackJsonp conflict with reference webpack 
         filename: 'bundle.[id].js',
         chunkFilename: 'chunk.[id].[chunkhash].js',
         publicPath: '../../',   //you public Path
         libraryTarget: 'commonjs2'  //output  cmd 
     },
    plugins: [
         new AssetsPlugin({
             filename: 'assets.json',
             fullPath: false,
             path: config.libraryPath,
             metadata: {
                 version: (new Date()).getTime()
             }
         }),
  1. In my project , I use SystemJS to load js from dll project, I wrote a simple plugin to insert SystemJS code to webpack bootstrap, but insert SystemJS to html template maybe easier. After that, you need to set map for Systemjs, like
return fetch(mapJsonLocation)
        .then(response => response.json())
        .then((asset) => {
            const map = {}
            if (asset.metadata) {
                delete asset.metadata
            }
            Object.keys(asset).forEach((key) => {
                const aliasKey = `${packageName}/${key}`
                const aliasPath = `${prefixPath}/${asset[key].js}`
                map[aliasKey] = aliasPath
            })
            SystemJS.config({
                map
            })
        })
  1. Now you can write in your file, Notice that async load is the only option, sync load nope, it is a little verbose
SystemJS.import('packageName/moduleName')
  .then((module) => {
     //todo 
  })
  1. If you use requirejs, Output AMD format libary and use
 requirejs(['module'], (module) => {})

UPDATE: If you use commnChunkPlugin to get a vendor output,you MUST load it before load any dll js file ! or you will get Error: (SystemJS) webpackJsonp1490526215268 is not defined LIKE @CameronPlimsoll

@CameronPlimsoll

This comment has been minimized.

Show comment
Hide comment
@CameronPlimsoll

CameronPlimsoll Mar 26, 2017

@githoniel Thanks , appreciate the code snippet.
However i have run into some problems
when i try an import the module it seems to ignore the systemJS config ?

    let packageName = 'myAwesomePlugin';
    let packageMainPath = 'http://localhost/plugin/main.bundle.js'
    let prefixPath = 'http://localhost/plugin';
    let moduleName = 'AppModule';
    let mapLocation = 'http://localhost/plugin/helios-service-assets.json'

    this.http.get(mapLocation)
      .subscribe(res => {
        let asset = res.json()
        const map = {}
        if (asset.metadata) {
          delete asset.metadata
        }
        map[packageName] = prefixPath;
        Object.keys(asset).forEach((key) => {
          const aliasKey = `${packageName}/${key}`
          const aliasPath = `${prefixPath}/${asset[key].js}`
          map[aliasKey] = aliasPath

        })
        let packages = {}
        packages[`${packageName}`] = { main: 'main.bundle.js', defaultExtension: 'js' };
        SystemJS.config({
          baseURL: prefixPath,
          map,
          packages
        })
      }, (err) => {
      }, () => {
        //systemjs config has been set , try and import the module 
        SystemJS.import(`${packageName}/${moduleName}`).then((module) => {
          console.log(module);
        }).catch(err => {
          console.error(err);
        })
      });
  }

the above throws the following error

Error: (SystemJS) XHR error (404 Not Found) loading http://localhost/plugin/AppModule.js

when i try import the main.bundle created from output library i get the following

Error: (SystemJS) webpackJsonp1490526215268 is not defined

this is the directory structure of the library

|-- plugin
        |-- favicon.ico
        |-- helios-service-assets.json
        |-- index.html
        |-- inline.bundle.js
        |-- main.bundle.js
        |-- polyfills.bundle.js
        |-- styles.bundle.js
        |-- vendor.bundle.js

any advice would be greatly appreciated

CameronPlimsoll commented Mar 26, 2017

@githoniel Thanks , appreciate the code snippet.
However i have run into some problems
when i try an import the module it seems to ignore the systemJS config ?

    let packageName = 'myAwesomePlugin';
    let packageMainPath = 'http://localhost/plugin/main.bundle.js'
    let prefixPath = 'http://localhost/plugin';
    let moduleName = 'AppModule';
    let mapLocation = 'http://localhost/plugin/helios-service-assets.json'

    this.http.get(mapLocation)
      .subscribe(res => {
        let asset = res.json()
        const map = {}
        if (asset.metadata) {
          delete asset.metadata
        }
        map[packageName] = prefixPath;
        Object.keys(asset).forEach((key) => {
          const aliasKey = `${packageName}/${key}`
          const aliasPath = `${prefixPath}/${asset[key].js}`
          map[aliasKey] = aliasPath

        })
        let packages = {}
        packages[`${packageName}`] = { main: 'main.bundle.js', defaultExtension: 'js' };
        SystemJS.config({
          baseURL: prefixPath,
          map,
          packages
        })
      }, (err) => {
      }, () => {
        //systemjs config has been set , try and import the module 
        SystemJS.import(`${packageName}/${moduleName}`).then((module) => {
          console.log(module);
        }).catch(err => {
          console.error(err);
        })
      });
  }

the above throws the following error

Error: (SystemJS) XHR error (404 Not Found) loading http://localhost/plugin/AppModule.js

when i try import the main.bundle created from output library i get the following

Error: (SystemJS) webpackJsonp1490526215268 is not defined

this is the directory structure of the library

|-- plugin
        |-- favicon.ico
        |-- helios-service-assets.json
        |-- index.html
        |-- inline.bundle.js
        |-- main.bundle.js
        |-- polyfills.bundle.js
        |-- styles.bundle.js
        |-- vendor.bundle.js

any advice would be greatly appreciated

@githoniel

This comment has been minimized.

Show comment
Hide comment
@githoniel

githoniel Mar 26, 2017

It seem SystemJS config's problem, i think baseURL is not needed

You can only load main.bundle by SystemJS from you entries in your dll webpack config.

githoniel commented Mar 26, 2017

It seem SystemJS config's problem, i think baseURL is not needed

You can only load main.bundle by SystemJS from you entries in your dll webpack config.

@mastilver

This comment has been minimized.

Show comment
Hide comment
@mastilver

mastilver Jul 3, 2017

It's not related to DllPlugin but I think this plugin I wrote solves most of your issues (https://github.com/mastilver/modules-cdn-webpack-plugin):

  • no need to specify library at build time
  • dependencies are shared across build / apps

mastilver commented Jul 3, 2017

It's not related to DllPlugin but I think this plugin I wrote solves most of your issues (https://github.com/mastilver/modules-cdn-webpack-plugin):

  • no need to specify library at build time
  • dependencies are shared across build / apps
@webpack-bot

This comment has been minimized.

Show comment
Hide comment
@webpack-bot

webpack-bot Jan 2, 2018

This issue had no activity for at least half a year.

It's subject to automatic issue closing if there is no activity in the next 15 days.

webpack-bot commented Jan 2, 2018

This issue had no activity for at least half a year.

It's subject to automatic issue closing if there is no activity in the next 15 days.

@webpack-bot webpack-bot added the inactive label Jan 2, 2018

@niieani

This comment has been minimized.

Show comment
Hide comment
@niieani

niieani Jan 3, 2018

Contributor

@webpack-bot fist-bump

Contributor

niieani commented Jan 3, 2018

@webpack-bot fist-bump

@webpack-bot webpack-bot removed the inactive label Jan 3, 2018

@stevefan1999

This comment has been minimized.

Show comment
Hide comment
@stevefan1999

stevefan1999 Feb 20, 2018

I was also looking for such solution. I wanted to develop external vendor JS plugin that was loaded at the same time with the main bundle. I wanted to particularly export the vendor bundle so the external JS plugin could relieve from loading duplicated code, and I'm sure this is something webpack DLL cannot do.

webpack should allow a global require like Browserify. I wonder why webpack is doing such a setback.

stevefan1999 commented Feb 20, 2018

I was also looking for such solution. I wanted to develop external vendor JS plugin that was loaded at the same time with the main bundle. I wanted to particularly export the vendor bundle so the external JS plugin could relieve from loading duplicated code, and I'm sure this is something webpack DLL cannot do.

webpack should allow a global require like Browserify. I wonder why webpack is doing such a setback.

@seajean

This comment has been minimized.

Show comment
Hide comment
@seajean

seajean May 28, 2018

I am looking for the same solution too, Some of my SPA module was written by others in other project, and I want to load it asynchronously and render it. Still its requires my modules as dependencies, If was in my project, it would be very easy to do by using webpack's import, but I can not find a way to load a module externally.

seajean commented May 28, 2018

I am looking for the same solution too, Some of my SPA module was written by others in other project, and I want to load it asynchronously and render it. Still its requires my modules as dependencies, If was in my project, it would be very easy to do by using webpack's import, but I can not find a way to load a module externally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment