Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

The case for Workers #6472

Open
surma opened this issue Feb 12, 2018 · 13 comments

Comments

@surma
Copy link

commented Feb 12, 2018

Do you want to request a feature or report a bug?

Feature! 馃帀

What is the current behavior?

(This is gonna be a bit longer.)

Let鈥檚 imagine working on a project using WebWorkers. Let鈥檚 say I am using this great new shiny library Comlink. It鈥檚 needed by both main.js and worker.js as Comlink is an abstraction over postMessage. Of course both main.js and worker.js have other dependencies besides Comlink.

dependencies

(You might already be able to tell where I鈥檓 going with this)

Attempt 1: Entry points

Listing both worker.js and main.js as entry points doesn鈥檛 yield the expected results as all static imports will be inlined. Comlink will be (down)loaded twice. The semantics of entry points are that only one entry point will be loaded at a time. In the context of my example, though, two entry points are loaded in parallel.

suboptimalchunk

Attempt 2: Dynamic loading

I tried some hackery with import(), but the problem is that WebPack (rightly) assumes that there鈥檚 is a global registry of loaded modules. worker.js, however, runs in a different realm and has a separate global scope to main.js

Workaround: Force a common chunk

My current workaround is to use the entry point approach and manually maintain a list of shared modules that need to be put in a separate chunk using CommonChunksPlugin:

module.exports = {
  entry: {
    main: './src/main.js',
    worker: './src/worker.js',
  },
  output: {
    filename: '[name].js',
  },
  plugins: [
    // Force comlink into its own bundle
    new webpack.optimize.CommonsChunkPlugin({
      name: 'comlink',
      minChunks: 2,
    }),
  ],
};

If the current behavior is a bug, please provide the steps to reproduce.

What is the expected behavior?

Under the assumption that there鈥檚 good caching headers, the optimal behavior is to put Comlink in it鈥檚 own chunk (or a chunk for shared modules) so it鈥檚 only downloaded once. The second request by the worker will hit the HTTP cache and re-use the already downloaded shared chunk.

optimalchunk

API

I鈥檓 not sure what the API for this would look like. This API would not only be useful for workers, but also for pages that load another entry point in an iframe (although, tbh, I struggle to find a good example for this 馃し鈥嶁檪锔 )

I think the most straight-forward way would to add a realmEndpoints (name pending lol) to the webpack config which lists entry points that can be loaded in parallel to one of the main entry points.

What do y鈥檃ll think? API suggestions/corrections? Am I just holding it wrong?

If this is a feature request, what is motivation or use case for changing the behavior?

Please mention other relevant information such as the browser version, Node.js version, webpack version and Operating System.

@sokra

This comment has been minimized.

Copy link
Member

commented Feb 13, 2018

Currently a target: "web" compilation and a target: "webworker" compilation can't share chunks, because they are using different ways to lazy-load chunks. web uses JSONP, webworker uses importScripts.

@sokra

This comment has been minimized.

Copy link
Member

commented Feb 13, 2018

It may be possible to create a new agnostic target in which chunks have the same format, than something like this could work.


For on-demand-loading:

While in web target loading chunk in parallel is easily possible via <script> tags, importScripts in webworker context is sync and don't allow parallel loading. We would need to use fetch instead, which has cross-origin limitation, but I assume this won't be a problem in webworkers.

For initial loading:

web target support multiple initial chunks via multiple <script> tags in HTML. new Worker() only supports a single argument. So it's difficult to startup a worker with multiple parallel initial chunks.

It may be possible to solve these challenges with a ServiceWorker doing the parallel requests and concatenation the responses...

@surma

This comment has been minimized.

Copy link
Author

commented Feb 13, 2018

Currently a target: "web" compilation and a target: "webworker" compilation can't share chunks, because they are using different ways to lazy-load chunks. web uses JSONP, webworker uses importScripts.

I forgot about that bit! In my workaround (see below) I currently also hand-load the chunks using importScript. I think an agnostic format that can be loaded by both workers and web would be great to see.

const base = `${location.protocol}//${location.host}${location.pathname.split('/').slice(0, -1).join('/')}`;
new Worker(`data:application/javascript,window=self;importScripts('${base}/comlink.js');importScripts('${base}/worker.js');`));
@surma

This comment has been minimized.

Copy link
Author

commented Feb 13, 2018

It may be possible to solve these challenges with a ServiceWorker doing the parallel requests and concatenation the responses...

This feels like early optimization/over-engineering to me. I see where you are coming from, but just supporting WebWorkers with sequential importScript() for now seems totally sufficient. Maybe allow to inject another function than importScript so developers can write their own loader if they really need parallel fetching. I think in the long run it鈥檚 easier to bet on ES6 module loading to land in workers and use that.

@sokra

This comment has been minimized.

Copy link
Member

commented Feb 14, 2018

WebWorkers with sequential importScript()

fetch + eval is a good option too.

@neoncom

This comment has been minimized.

Copy link

commented Mar 19, 2018

a combined "web+webworker" target really is nesseccary.
I think that web+webworker code is so common that this target is a must have. Else you cannot create common chunks between the two.
last but not least i have 5+ mb of duplicate code because of this issue because chunks cannot be shared between web+webworker entry points:
#microsoft/monaco-editor#776

@TCMiranda

This comment has been minimized.

Copy link

commented Mar 31, 2018

importScripts('foo.js', 'bar.js'); /* imports two scripts */

Note: Scripts may be downloaded in any order, but will be executed in the order in which you pass the filenames into importScripts() . ...

From MDN Using Web Workers:

Seems that it doesn't means that 'foo.js' and 'bar.js' could be downloaded in parallel.
In fact they are downloaded and executed synchronously.

prefetch as script also works. Even using importScript, and the file is loaded from disk cache.

@TCMiranda

This comment has been minimized.

Copy link

commented Mar 31, 2018

Isn't it possible to infer a "web+webworker" target by seeing importScripts being used? Or the target is to keep using import / require inside workers?

@mixtur

This comment has been minimized.

Copy link

commented Oct 22, 2018

It seems to be the same as #6908. And both issues seems dead. Why is that? Sounds like a very useful feature to have

@webpack-bot

This comment has been minimized.

Copy link
Contributor

commented Jul 15, 2019

This issue had no activity for at least half a year.

It's subject to automatic issue closing if there is no activity in the next 15 days.

@evilebottnawi

This comment has been minimized.

Copy link
Member

commented Jul 15, 2019

bump

@webpack-bot webpack-bot removed the inactive label Jul 15, 2019

@cacheflow

This comment has been minimized.

Copy link
Contributor

commented Aug 16, 2019

Has this work been done already? Could this possibly be in Webpack 5? @evilebottnawi @sokra

@evilebottnawi

This comment has been minimized.

Copy link
Member

commented Aug 16, 2019

@cacheflow i think no, it is not high priority, but you can send a PR and help us 馃憤

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
9 participants
You can鈥檛 perform that action at this time.