Vite duplicates code across main bundle and web worker bundle

[gabrielecirulli]

Describe the bug

Please refer to this repository for a way to reproduce the issue: https://github.com/gabrielecirulli/vite-pixi-bundling

When bundling Web Worker code with Vite, some of the dependencies that are used
by both the worker and the page context are duplicated in the final bundle, even
though they should be shared because they are virtually the same code.

In the provided repository, pixi.js is used to render to two instances of canvas, once within the page context and once within a Web Worker.

To do this, we have a file canvas.ts which imports pixi.js and exports a class called Canvas. We also have a worker.ts script that will run as a web worker. Both the worker script and main.ts import the Canvas class and set it up. Other than this they virtually do the same thing.

Intuitively, all of pixi.js should just be bundled once and shared across the two page context and web worker scripts which import it. However, the code ends up duplicated and bloating the bundle size (see build logs)

As far as I can tell, there is nothing in the Vite or Rollup docs that helps with this. I have tried various combinations of manualChunks and other options to no avail.

Reproduction

https://github.com/gabrielecirulli/vite-pixi-bundling

Steps to reproduce

• Clone this repository
• Run pnpm install
• Run pnpm run build
• Check the generated modules in dist/assets

Comparing the build output

The repository includes a branch called with-manual-chunks. This branch forces Rollup to bundle all code belonging to pixi.js into a manual chunk called pixi.js. This way, you can compare the two bundles to see the (lack of) differences:

• Switch to branch with-manual-chunks
• Run pnpm install
• Run pnpm run build
• Run git diff --no-index dist/assets/pixi.js*

System Info

System:
    OS: macOS 14.0
    CPU: (8) arm64 Apple M2
    Memory: 236.97 MB / 24.00 GB
    Shell: 3.6.1 - /opt/homebrew/bin/fish
  Binaries:
    Node: 21.5.0 - ~/.local/share/nvm/v21.5.0/bin/node
    Yarn: 1.22.21 - /opt/homebrew/bin/yarn
    npm: 10.2.4 - ~/.local/share/nvm/v21.5.0/bin/npm
    pnpm: 8.14.0 - ~/Library/pnpm/pnpm
  Browsers:
    Chrome: 124.0.6367.60
    Chrome Canary: 125.0.6422.0
    Firefox Nightly: 128.0a1
    Safari: 17.0
    Safari Technology Preview: 17.4
  npmPackages:
    vite: ^5.2.11 => 5.2.11

Used Package Manager

npm

Logs

`vite build --debug`

```shell 2024-05-19T19:25:38.250Z vite:config bundled config file loaded in 13.52ms 2024-05-19T19:25:38.266Z vite:config using resolved config: { server: { preTransformRequests: true, host: true, sourcemapIgnoreList: [Function: isInNodeModules$1], middlewareMode: false, fs: { strict: true, allow: [Array], deny: [Array], cachedChecks: undefined } }, worker: { format: 'es', plugins: '() => plugins', rollupOptions: {} }, build: { target: [ 'es2020', 'edge88', 'firefox78', 'chrome87', 'safari14' ], cssTarget: [ 'es2020', 'edge88', 'firefox78', 'chrome87', 'safari14' ], outDir: 'dist', assetsDir: 'assets', assetsInlineLimit: 4096, cssCodeSplit: true, sourcemap: false, rollupOptions: {}, minify: 'esbuild', terserOptions: {}, write: true, emptyOutDir: null, copyPublicDir: true, manifest: false, lib: false, ssr: false, ssrManifest: false, ssrEmitAssets: false, reportCompressedSize: true, chunkSizeWarningLimit: 500, watch: null, commonjsOptions: { include: [Array], extensions: [Array] }, dynamicImportVarsOptions: { warnOnError: true, exclude: [Array] }, modulePreload: { polyfill: true }, cssMinify: true }, configFile: '/dir/to/vite-pixi-bundling/vite.config.ts', configFileDependencies: [ '/dir/to/vite-pixi-bundling/vite.config.ts' ], inlineConfig: { root: undefined, base: undefined, mode: undefined, configFile: undefined, logLevel: undefined, clearScreen: undefined, build: {} }, root: '/dir/to/vite-pixi-bundling', base: '/', rawBase: '/', resolve: { mainFields: [ 'browser', 'module', 'jsnext:main', 'jsnext' ], conditions: [], extensions: [ '.mjs', '.js', '.mts', '.ts', '.jsx', '.tsx', '.json' ], dedupe: [], preserveSymlinks: false, alias: [ [Object], [Object] ] }, publicDir: '/dir/to/vite-pixi-bundling/public', cacheDir: '/dir/to/vite-pixi-bundling/node_modules/.vite', command: 'build', mode: 'production', ssr: { target: 'node', optimizeDeps: { noDiscovery: true, esbuildOptions: [Object] } }, isWorker: false, mainConfig: null, bundleChain: [], isProduction: true, plugins: [ 'vite:build-metadata', 'vite:watch-package-data', 'vite:pre-alias', 'alias', 'vite:modulepreload-polyfill', 'vite:resolve', 'vite:html-inline-proxy', 'vite:css', 'vite:esbuild', 'vite:json', 'vite:wasm-helper', 'vite:worker', 'vite:asset', 'vite:wasm-fallback', 'vite:define', 'vite:css-post', 'vite:build-html', 'vite:worker-import-meta-url', 'vite:asset-import-meta-url', 'vite:force-systemjs-wrap-complete', 'commonjs', 'vite:data-uri', 'vite:dynamic-import-vars', 'vite:import-glob', 'vite:build-import-analysis', 'vite:esbuild-transpile', 'vite:terser', 'vite:reporter', 'vite:load-fallback' ], css: { lightningcss: undefined }, esbuild: { jsxDev: false }, preview: { port: undefined, strictPort: undefined, host: true, https: undefined, open: undefined, proxy: undefined, cors: undefined, headers: undefined }, envDir: '/dir/to/vite-pixi-bundling', env: { BASE_URL: '/', MODE: 'production', DEV: false, PROD: true }, assetsInclude: [Function: assetsInclude], logger: { hasWarned: false, info: [Function: info], warn: [Function: warn], warnOnce: [Function: warnOnce], error: [Function: error], clearScreen: [Function: clearScreen], hasErrorLogged: [Function: hasErrorLogged] }, packageCache: Map(1) { 'fnpd_/dir/to/vite-pixi-bundling' => { dir: '/dir/to/vite-pixi-bundling', data: [Object], hasSideEffects: [Function: hasSideEffects], webResolvedImports: {}, nodeResolvedImports: {}, setResolvedCache: [Function: setResolvedCache], getResolvedCache: [Function: getResolvedCache] }, set: [Function (anonymous)] }, createResolver: [Function: createResolver], optimizeDeps: { holdUntilCrawlEnd: true, esbuildOptions: { preserveSymlinks: false } }, appType: 'spa', experimental: { importGlobRestoreExtension: false, hmrPartialAccept: false }, getSortedPlugins: [Function: getSortedPlugins], getSortedPluginHooks: [Function: getSortedPluginHooks] } vite v5.2.11 building for production... transforming... ✓ 616 modules transformed. rendering chunks... computing gzip size... dist/assets/batchSamplersUniformGroup-esTw4NqZ.js 0.23 kB dist/index.html 0.38 kB │ gzip: 0.27 kB dist/assets/CanvasPool-DAAltsjf.js 0.69 kB dist/assets/colorToUniform-PMjxk6y4.js 24.37 kB dist/assets/WebGPURenderer-C4GTSOc9.js 35.19 kB dist/assets/browserAll-s_Odv2Zz.js 37.24 kB dist/assets/SharedSystems-Dmf2ic5s.js 43.80 kB dist/assets/WebGLRenderer-DSs9dsMu.js 59.35 kB dist/assets/webworkerAll-CBdqNP-U.js 77.34 kB dist/assets/worker-wXglRUk2.js 228.69 kB dist/assets/batchSamplersUniformGroup-BJBcGol3.js 0.23 kB │ gzip: 0.21 kB dist/assets/CanvasPool-P9anc0rq.js 0.69 kB │ gzip: 0.38 kB dist/assets/colorToUniform-DdNNl6bf.js 24.37 kB │ gzip: 7.84 kB dist/assets/WebGPURenderer-D2rAnSo4.js 35.19 kB │ gzip: 9.73 kB dist/assets/browserAll-Qg6UHUQs.js 37.24 kB │ gzip: 9.96 kB dist/assets/SharedSystems-DviRLSqg.js 43.79 kB │ gzip: 12.25 kB dist/assets/WebGLRenderer-D69Oodqk.js 59.35 kB │ gzip: 16.33 kB dist/assets/webworkerAll-B3O_29JZ.js 77.34 kB │ gzip: 22.81 kB dist/assets/index-DaWpJUK9.js 174.57 kB │ gzip: 55.57 kB ✓ built in 3.09s ```

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Make sure this is a Vite issue and not a framework-specific issue. For example, if it's a Vue SFC related bug, it should likely be reported to vuejs/core instead.
• Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
• The provided reproduction is a minimal reproducible example of the bug.

[cyan-2048]

the plot thickens, workers with the same module import would also make duplicates

also why is it only es or iife? systemjs and amd has solutions for workers as well
(as a workaround I would set format to systemjs even though it would make the typescript emit an error, however with this issue open I would rather use iife anyways)

edit: i guess to give it more details(?, although this should be a different issue really) there's 2 web workers which both use comlink as that's a common thing to use, making 2 comlink bundles should not be necessary.

[gabrielecirulli]

@sapphi-red @hi-ogawa I was wondering if you have any thoughts please. This issue is still affecting us and doubling our bundle size (a significant burden on bandwidth required by users and loading speed)

[hi-ogawa]

@gabrielecirulli Current behavior is by design and Vite build triggers a dedicated rollup build for a worker entry (even with a separate set of plugins https://vitejs.dev/config/worker-options.html#worker-plugins).

A few reasons are explained in #16554 (comment)

because worker builds have different resolving and runtime behaviour than normal browser builds, e.g. access to ESM, worker-specific globals etc.

If worker entries were treated simply as a part of main client build (just like dynamic import), then Vite cannot guarantee it won't access things like document which is not available inside worker.

If you know this risk, then it's possible to write a plugin using emitFile({ type: "chunk" }) to dynamically include worker entries as a part of main build. Not fully tested but a plugin like below seems to work for your reproduction:

Plugin code

import { defineConfig, Plugin } from "vite";

// https://vitejs.dev/config/
export default defineConfig({
  server: {
    host: true,
  },
  worker: {
    format: "es",
  },
  build: {
    minify: false,
    // need to eliminate `document` reference
    modulePreload: false,
  },
  plugins: [workerChunkPlugin()],
});

// use emitFile to create a worker entry as a chunk of main client build.
function workerChunkPlugin(): Plugin {
  return {
    name: workerChunkPlugin.name,
    apply: "build",
    enforce: "pre",
    async resolveId(source, importer, _options) {
      // intercept "xxx?worker"
      if (source.endsWith("?worker")) {
        const resolved = await this.resolve(source.split("?")[0], importer);
        return "\0" + resolved?.id + "?worker-chunk";
      }
    },
    load(id, _options) {
      if (id.startsWith("\0") && id.endsWith("?worker-chunk")) {
        const referenceId = this.emitFile({
          type: "chunk",
          id: id.slice(1).split("?")[0],
        });
        return `
          export default function WorkerWrapper() {
            return new Worker(
              import.meta.ROLLUP_FILE_URL_${referenceId},
              { type: "module" }
            );
          }
        `;
      }
    },
  };
}

$ pnpm build

vite v5.2.11 building for production...
✓ 616 modules transformed.
dist/index.html                                      0.38 kB │ gzip:   0.26 kB
dist/assets/batchSamplersUniformGroup-BZfrq0yw.js    0.43 kB │ gzip:   0.28 kB
dist/assets/index-BsivXy7a.js                        0.76 kB │ gzip:   0.35 kB
dist/assets/CanvasPool-HXi5M9ss.js                   2.01 kB │ gzip:   0.71 kB
dist/assets/colorToUniform-BBLU0xpW.js              49.00 kB │ gzip:  11.74 kB
dist/assets/WebGPURenderer-D3B6HnuZ.js              63.89 kB │ gzip:  13.54 kB
dist/assets/SharedSystems-CtlWS3fy.js               87.52 kB │ gzip:  18.96 kB
dist/assets/browserAll-yDBsCdS0.js                  97.48 kB │ gzip:  19.72 kB
dist/assets/WebGLRenderer-C4toZjTE.js              112.72 kB │ gzip:  24.12 kB
dist/assets/worker-CKKmHEZU.js                     113.95 kB │ gzip:  30.33 kB
dist/assets/webworkerAll-DLeBSlo-.js               161.31 kB │ gzip:  35.30 kB
dist/assets/canvas-D3GKaOyS.js                     466.66 kB │ gzip: 104.15 kB
✓ built in 1.78s

[gabrielecirulli]

Thank you very much! At a basic level, this appears to work in our scenario.