How to load onigasm.wasm via webpack

[epatpol]

What's the best way to do it? Usually I would load it with an asm loader for webpack and then use the script from the assembly file, but now I only have to supply it to loadWASM which comes from onigasm, any ideas on how to do this? Thanks

[zikaari]

My original vision was to have it loaded through file-loader like so:

await loadWASM(require('file-loader!onigasm/lib/onigasm.wasm'))

But please, I beg you, to show me the alternative method(asm loader). I won't think twice if some other method makes the developer experience more smoother.

[epatpol]

I'm actually not that familiar with webpack, I just know the bare minimum to make it work in theia, so I can try the file-loader and see if it works for now 😄

[epatpol]

Actually I'm getting

ERROR in /home/epatpol/Projects/Github/theia/node_modules/onigasm/lib/onigasm.wasm (/home/epatpol/Projects/Github/theia/node_modules/file-loader/dist/cjs.js!/home/epatpol/Projects/Github/theia/node_modules/onigasm/lib/onigasm.wasm)
Module parse failed: Wasm decoding failedResult = expected magic word 00 61 73 6d, found 6d 6f 64 75 @+0

You may need an appropriate loader to handle this file type.
CompileError: Wasm decoding failedResult = expected magic word 00 61 73 6d, found 6d 6f 64 75 @+0

using

            {
                test: /.(jpg|png|gif|wasm)$/,
                loader: 'file-loader',
                options: {
                    name: '[path][name].[hash].[ext]',
                }
            },

Let me tinker a bit more with wasm and webpack...

[zikaari]

Totally agree and pissed at the fact that I can't find a way myself to disable Webpack 4's WASM parser. The problem is whenever Webpack 4 encounters a require that refers to a wasm file Webpack's WebAssembly parser kicks in and messes up everything, even when not invited to the party.

The culprit in question - webpack/lib/wasm/WebAssemblyModulesPlugin.js

I have tried module.noParse option set to /onigasm.wasm/ in webpack config, the error on CLI disappears but the emitted JS file still doesn't look like as it should; i.e loadWASM being called with a URL to wasm file that file-loader created for us. But nope.

I am sticking to Webpack 3 for now, until the WASM dust settles a bit. (await loadWASM(require('file-loader!onigasm/lib/onigasm.wasm')) works with Webpack 3)

[zikaari]

It's critical to note that Webpack 4 supports importing wasm files out of the box. You might see examples like this around the web:

import('./add.wasm').then(({ add }) => {
    console.log(add(22, 2200))
})

Example above will work out of the box when compiled with Webpack 4. Now you might ask, then why can we not import onigasm.wasm like that? You see oniguruma is massive beast, it's compiled with emscripten in such a way that we can read/write from/to runtime memory, call malloc and free from JavaScript side. It's not just we can, it's that we must. We have to have direct access to memory on JavaScript side so we can write strings to memory, have oniguruma perform regex operation, let it write result to memory and then read that back on JavaScript side. The problem? emscripten's glue code allows that but Webpack's does not. Very unfortunate.

Please read Background section of this comment to see the complexity.

[epatpol]

Thank you for the in-depth explanation. So from what I understand is that we're stuck with using webpack3 for now. Although I'd usually be fine with it, we've been using webpack4 for a while so I should check to make sure it doesn't break anything else. Otherwise it's a bummer for us :(

Indeed it looks like noParse option not working is probably a bug in webpack (I saw that you opened an issue for that), hopefully something can be done about this.

[zikaari]

I'll strongly advise you to continue using Webpack 4. While webpack issue is being fixed, use CopyWebpackPlugin to manually copy onigasm.wasm from node_modules to build output directory and not use require at all.

PS: Since Webpack 4 now resolves .wasm files before .js, there's an import in onigasmH.ts that needs to made explicit .js. I'm away RN and will patch in t-minus 6 hours.

[zikaari]

Since Webpack 4 now resolves .wasm files before .js, there's an import in onigasmH.ts that needs to made explicit .js. I'm away RN and will patch in t-minus 6 hours.

Fixed in 6787446 and available in onigasm@1.3.1. Run npm i onigasm again to update.

---

Let me know if you got through the day using CopyWebpackPlugin or similar hack.

[epatpol]

Thanks! I'll report back as soon as I test this out.

[zikaari]

Please ignore all the methods we've discovered so far (most of which are ugly hacks), thanks to Florent and Tobias comment, there's a better way and we get to stay with Webpack 4 😘

// index.js

import {
    loadWASM,
    OnigRegExp
} from 'onigasm'

(async () => {
    await loadWASM(require('onigasm/lib/onigasm.wasm'))
    const reg = new OnigRegExp('[a-z]+')
    console.log(reg.searchSync('nice'))
})()

// webpack.config.js

module.exports = {
    mode: 'development',
    entry: __dirname + '/index.js',
    output: {
        filename: 'bundle.js',
        publicPath: 'dist/'
    },
    module: {
        rules: [{
            test: /\.wasm$/,
            loader: "file-loader",
            type: "javascript/auto",
        }]
    }
}

[epatpol]

This works fine with me. We can close the issue if you consider this okay :)

[Menci]

@NeekSandhu @epatpol This method doesn't work for me. I got this from require:

Using its default property's value works.

reproduce.zip

[mofux]

If I recall correctly there was a change in the file-loader module a couple of months back that introduced that changed behaviour in order to support the es module „import“ statement.

[zm-cttae-archive]

I was able to use:

• typescript config: commonjs and no ESM
• Dependency: encoded-uint8array-loader
• import * as bytes from 'onigasm.wasm'
• Webpack config: { test: /\.wasm$/, type: 'javascript/auto', loader: 'encoded-uint8array-loader' }
• and the following reference file:

// wiring from webpack `encoded-uint8array-loader` to inline WASM buffer view
declare module '*.wasm' { const bytes: Uint8Array; export = bytes; }

[maxdekrieger]

@zm-cttae Where do you put the declare module '*.wasm' { const bytes: Uint8Array; export = bytes; } line?

I get the "Cannot find module 'onigasm.wasm' or its corresponding type declarations." error on the import * as bytes from 'onigasm.wasm' line in my source file.

[zm-cttae]

Hi you need to make a ref.d.ts preferably in your root folder or your source folder (the first entry in tsconfig includes rules)

https://github.com/vsce-toolroom/vscode-textmate-languageservice/blob/b46dfe055db08af8cd96ac16c6c793fcac0b01a6/src/webpack.d.ts

[maxdekrieger]

Thanks for your suggestions. With a webpack.d.ts in my root folder that has the declare module '*.wasm' { const bytes: Uint8Array; export = bytes; } line, TypeScript thinks it's all good.

However, I'm still missing something. Webpack can't find the onigasm.wasm file in the folders where it's looking (src/ and under node_modules) which is correct because it's under node_modules/onigasm/lib/onigasm.wasm. Do I need to include this somehow in my Webpack config?

[zm-cttae]

You can import it using a relative import where .. = parent folder

[maxdekrieger]

Perfect, that seems to get me to the next step. So far:

import * as bytes from '../node_modules/onigasm/lib/onigasm.wasm'
// ...
  await loadWASM(bytes);

With the following Webpack config:

      {
        test: /\.wasm$/,
        loader: "encoded-uint8array-loader",
        type: "javascript/auto",
      },

This compiles but but gives me the following error in the browser:
Uncaught (in promise) TypeError: Expected a string (URL of .wasm file) or ArrayBuffer (.wasm file itself) as first parameter.

[zm-cttae]

I think the issue is that you got a UInt8Array aka ArrayBufferView but you want an ArrayBuffer.
The two are convertible and the answer for it should be on a StackOverflow!

EDIT its the bytes.buffer property lol!