Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Webpack doesn’t work well with wasm modules created with Emscripten #7352

Open
surma opened this issue May 20, 2018 · 4 comments

Comments

@surma
Copy link

commented May 20, 2018

Feature request

What is the current behavior?
The modularized JS emitted by Emscripten registers a global with a given name that loads the wasm file on invocation, initializes the wasm runtime and returns a Module.

Making it work with Webpack is quite hard as there seems to be interference with Webpack 4 defaults.
This is the webpack.config.js that I came up with:

const webpack = require("webpack");
const path = require("path");

module.exports = {
  mode: "development",
  entry: "./index.js",
  output: {
    path: path.resolve(__dirname, "dist"),
    filename: "bundle.js"
  },
  module: {
    defaultRules: [
      {
        type: "javascript/auto",
        resolve: {}
      }
    ],
    rules: [
      {
        test: /fibonacci\.js$/,
        loader: "exports-loader"
      },
      {
        test: /fibonacci\.wasm$/,
        loader: "file-loader",
        options: {
          publicPath: "dist/"
        }
      }
    ]
  },
  // This is necessary due to the fact that emscripten puts both Node and
  // web code into one file. The node part uses Node’s `fs` module to load
  // the wasm file.
  // Issue: https://github.com/kripken/emscripten/issues/6542.
  plugins: [new webpack.IgnorePlugin(/(fs)/)]
};

(Here is a minimal test project in a gist that you can clone and build with npm start. Docker required!)

edit:
In the meantime, @sokra informed that that I can simplify the configuration a bit (and make it less like a sledgehammer):

module.exports = {
  /* ... */
  browser: { 
    "fs": false // ← !!
  },
  module: {
    rules: [
      /* ... */
      {
        test: /fibonacci\.wasm$/,
        type: "javascript/auto", // ← !!
        loader: "file-loader",
        options: {
          publicPath: "dist/"
        }
      }
    ]
  },
};

Unexpected things I had to do

  • I needed to overwrite defaultRules as otherwise some sort of default rule will run in addition to the ones I specified and making webpack error “Module parse failed: magic header not detected” (try it!)
  • I needed to specify file-loader for the wasm file as otherwise webpack tries to resolve the names of the wasm module’s import object like env, which are provided by the JS file.
  • I needed to set a locateFile() function as webpack changes the file (and potentially path) of the wasm file and Emscripten hardcodes that name (not visible here but in the gist)

I am not sure what the right course of action here is, but considering that most wasm projects are going to be built with Emscripten, I feel like it’s worth making it easier.

Happy to answer Qs to give y’all a clearer picture.

What is the expected behavior?

Ideally, Webpack would recognize the typical Emscripten JS files and automatically bundle the accomodating wasm module and make paths work.

Other relevant information:
webpack version: 4.8.3
Node.js version: 10
Operating System: Mac OS 10.13.4
Additional tools:

@xtuc

This comment has been minimized.

Copy link
Member

commented May 20, 2018

Here's my suggestion: #7264 (comment)

We don't need to explicitly support emscripten, that would be generic enough.

Note that wasn-bindgen already does that.

@sokra

This comment has been minimized.

Copy link
Member

commented May 22, 2018

It would be great to see a ESM target for Emscripten. This is how webpack's wasm support works. It expects WASM to be imported via import { ... } from "./something.wasm". WASM can get access to JS functions via (import "./something.js" "exportName") (import section).

wasm-bindgen creates two files abc.wasm and abc.js. The abc.js exports all exposed functions (public api) and all internal functions. It also imports the abc.wasm file. The abc.wasm imports all internal functions from abc.js.

That works great but exposes all internal functions too. I think a 3 file output would be better:

  • abc.js exports all public functions, imports abc_internal.js and abc_internal.js
  • abc_internal.js exports all internal functions
  • abc_internal.wasm imports abc_internal.js

Even better if repeated runtime code could be moved into a npm package: This way the wasm could import it directly from this package. This could get handy if you got many wasm modules in an application.

@xtuc

This comment has been minimized.

Copy link
Member

commented May 22, 2018

For ref emscripten-core/emscripten#6483

Also note that the same works for a module on npm. I think it would be more practical.

@gogins gogins referenced this issue May 28, 2018
Closed
jnferner added a commit to myelin-ai/myelin that referenced this issue Sep 13, 2018
Fix webpack issues by using a workaround
The workaround is described in webpack/webpack#7352
Note that now the webserver cannot load our wasm:
"TypeError: _myelin_visualization_bg__WEBPACK_IMPORTED_MODULE_0__.init is not a function"
This might be caused by a transcient dependency on stdweb:
https://github.com/sebcrozet/nphysics/blob/master/build/nphysics2d/Cargo.toml#L31
Which is incompatible with wasm-bindgen:
koute/stdweb#263
agilgur5 added a commit to agilgur5/physijs-webpack that referenced this issue Sep 16, 2018
(fix): slightly modify ammo so it won't try to require stuff
- this is what I was having trouble with when I first created this repo
  - c.f. kripken/ammo.js#109 (comment) ,
    webpack/webpack#7352 ,
    and emscripten-core/emscripten#6542
- can't just upgrade to a new ammo as that might very well break
  physijs, so just make the Node environment check false
  - in this case I changed `var qa="object"===typeof process,` to
    `var qa=false,` in the minified code
    - could also remove the `if(qa){`... parts now, though that'll
      probably get re-minified too and nbd
agilgur5 added a commit to agilgur5/physijs-webpack that referenced this issue Sep 16, 2018
(fix): slightly modify ammo so it won't try to require stuff
- this is what I was having trouble with when I first created this repo
  - c.f. kripken/ammo.js#109 (comment) ,
    webpack/webpack#7352 ,
    and emscripten-core/emscripten#6542
- can't just upgrade to a new ammo as that might very well break
  physijs, so just make the Node environment check false
  - in this case I changed `var qa="object"===typeof process,` to
    `var qa=false,` in the minified code
  - and also remove thed `if(qa){`...`}` parts
    - this was the part that had `require` statements, so now webpack
      etc should be able to parse it without a problem
      - there was a `require("fs")` and a `require("path")` in there
  - alternatively, could build and replace stuff with webpack, but I'd
    need to provide that anyway for auto-config
    - downside is it might no longer work on Node, but not the target
      audience so w/e
agilgur5 added a commit to VerifiableRobotics/LTLMoPWeb3D that referenced this issue Oct 3, 2018
(deps): require Three + Physijs via Webpack + NPM
- use physijs-webpack, a fork I created a few years ago and finally
  took some time to debug and get working
  - use via github since someone took the NPM name a few years ago and
    referenced my repo in it even though it never worked until today...
    - maybe will update once I get that resolved
  - now we don't have to vendor in all those scripts anymore and don't
    need to manually configure Physijs either!
    - remove the vendored physijs and three scripts
  - had to debug and modify ammo to get that version working with
    webpack / bundlers in general
    - newer versions of Emscripten can target specific envs
    - c.f. kripken/ammo.js#109 (comment) ,
      webpack/webpack#7352 ,
      and emscripten-core/emscripten#6542
  - and then made the worker config more flexible
  - add worker-loader as a devDep per physijs-webpack instructions
    - fix publicPath location as this is now actually used
      - the worker is loaded based on the publicPath

- add three-window-resize and three-trackballcontrols as deps as well
  - since three's examples/js/ folder doesn't quite work with bundlers
  - c.f. mrdoob/three.js#9562
  - maybe if I made some modifications, updated to newer Three
    revision, and used imports-loader it might work :shrug: TBD

- upgrade physijs to latest and Three from r60 to r73
  - latest physijs uses r73, so remain consistent
  - also physijs-webpack has a peerDep for that specific version
  - three-trackballcontrols@0.0.3 also requires r73 as dep
- had to refactor a bit due to upgrade
  - WebGLRenderer() -> WebGLRenderer({alpha: true})
    - the canvas now defaults to black without this, which was extremely
      disorienting
  - shadowMapEnabled -> shadowMap.enabled
  - CubeGeometry -> BoxGeometry
  - quaternion._euler -> rotation
    - I probably could've just used this before, couldn't have I...?

.
.

- TODO: improve webpack perf (CPU + Memory) and build speed
- this change slows down initial build times quite a bit (~20s), since
  Three, Physijs, and Ammo are all parsed by Webpack now
  - will want to update webpack to get a dev-server (wepback-serve)
    running
    - webpack itself is faster in later versions as well
    - and perhaps add HardSource for caching otherwise or split out
      vendored stuff into a DLL
    - probably can't update until decaffeinate'd since I believe the
      loaders used here are no longer maintained :/
      - and then would need to figure out the literate programming
        and its sourcemaps
  - will want to output a separate vendor bundle per best practice
    - may also want to output HTML via webpack too while at it
      - the templates don't do any templating so would be nbd
      - and would allow for hashing and therefore cache-busting
        - no need to manually clean cache then
  - and prod build / uglification is even _slower_ (~80s)
    - may want to exclude some files from Uglify
      - i.e. Ammo and use three.min so it can be excluded too
cbreak-black added a commit to disneyresearch/jeri that referenced this issue Dec 14, 2018
Webpack 4 Hack Test
Trying to get webpack 4 to work by hacking on the code until it stops
complaining. Done with input from:
webpack/webpack#8412
webpack/webpack#7352
@mikeheddes

This comment has been minimized.

Copy link

commented Mar 18, 2019

I also came across this problem when using a JS library I made, that uses WebAssembly, in a React web project.

I solved it by adding some hacky pre.js code to basically hijack the wasm module instantiation from the Emscripten created glue code. This way I have all the control over initialising the WASM module. The pre.js file overrides the createWasm function and adds a custom module initialisation function which it exports using ES6. Note that I compile with the default JS glue code settings (see cpp build script) so the EXPORT_ES6 and modularize options are not set but because the pre.js code has an ES6 export statement the generated glue code is an ES6 module.

After that I combine the generated JS glue code with the JS API interface code bundled with Rollup. The entry of this can be found in index.js.

Then finally when using the library I need to provide it an arrayBuffer (node.js) or a fetch instance (browsers) of the generated dcgp.wasm file. With webpack that means I needed the following rule:

{
  test: /\.wasm$/,
  type: 'javascript/auto',
  loader: 'file-loader',
}

Some implementation references:

So my conclusion being...
I agree with @surma that Webpack and Emscripten don't work well together but I think the main solution should be found in the way Emscripten generates the JS glue code and not how Webpack should handle them. I'm also not sure if @surma's suggestion in #6542 to make separate platform outputs will fix the root cause of this problem. For me the following implementation would make more sense in the JS ecosystem:

// node.js
import { initializer } from './wasm-glue-code'
import { fileToArrayBuffer } from 'emscripten-helpers'

const myLibrary = initializer(fileToArrayBuffer('./wasm-file.wasm'))

// web
import { initializer } from './wasm-glue-code'

const myLibrary = initializer(fetch('./wasm-file.wasm'))

Hope this was helpful in some way.

r3dDoX added a commit to r3dDoX/eaSiDiWidget that referenced this issue Aug 15, 2019
try arcgis webpack bundling
* currently webpack throws error because of missing 'env'
* see: webpack/webpack#7352
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.