Does not work with Vite/Browser

[stefnotch]

I tried out the example in the Readme, and after fixing it #7 and tweaking it to use TextEncoder/TextDecoder, it refused to work. At this point, I decided to inspect the imported object

import * as brotliPromise from "brotli-wasm";
console.log(brotliPromise);

To my surprise, it's basically empty.

Then I decided to cut to the chase and directly imported the relevant bits and pieces.

import { compress, decompress } from "brotli-wasm/pkg.bundler/brotli_wasm";

However, this fails since the WASM object doesn't seem to have been loaded yet.

For what it's worth, a different project of mine used Brotli-Wasm. Back then, I couldn't figure it out either and just copy-pasted the code + license. Then, I fixed it up a bit... https://github.com/stefnotch/starboard-editor/blob/d51bc17673385ad192417592add0092881379bfb/src/useCompression.ts#L1

[stefnotch]

Basically, I think the import * as wasm from './brotli_wasm_bg.wasm'; line in brotli_wasm_bg.js doesn't work in browsers.

[pimterry]

Hmmmmm, I don't know anything about Vite I'm afraid!

This project definitely does work in browsers in some configurations (I'm using it with Webpack 4 in multiple projects, and the test suite in this repo builds and tests it using Webpack 5). I'm not sure what the difference is in the Vite case though! What should happen is that:

• Bundlers look at the browser field in package.json to find the main file for browsers
• That points to index.browser.js
• That exports a promise for an import of /pkg.bundler/brotli_wasm.js which just re-exports /pkg.bundler/brotli_wasm_bg.js
• brotli_wasm_bg loads the wasm, and exports a bunch of function wrappers around it.

Any idea where that's going wrong? When you say the import * as wasm doesn't work - are there any errors from there? How does Vite do bundling?

For reference the working webpack 5 config is here (there's a asyncWebAssembly feature you have to enable) and the webpack 4 is here (no WASM-specific config at all I think, it just works out of the box).

[stefnotch]

Basically, Vite uses Esbuild + some extra magic for dependencies during development. Webpack's performance is atrocious after all.
For production, Vite uses Rollup.

With the magic stuff enabled, I can't quite figure out what exactly Vite does.

With it disabled, it pretty much uses normal ES module imports.
So import * as brotliPromise from "brotli-wasm"; will simply import the index.browser.js file as an ES module.
However, index.browser.js uses the other way of exporting modules

module.exports = import("./pkg.bundler/brotli_wasm.js");

at which point the web browser goes "Uncaught ReferenceError: module is not defined" and fails.

[stefnotch]

So my next step is trying out

import * as brotliPromise from "brotli-wasm/pkg.bundler/brotli_wasm.js";
console.log(brotliPromise);

This actually correctly imports the module.

However, it then fails, since import * as wasm from './brotli_wasm_bg.wasm'; is basically magic. Webpack can handle it, since being able to handle anything, no matter how arcane, is Webpack's specialty.

Vite on the other hand can't handle it as nicely and ends up doing the following:
https://vitejs.dev/guide/features.html#webassembly
Essentially, wasm ends up being an asynchronous initialization function, which also mirrors how browsers handle WebAssembly. It's asynchronous. https://developer.mozilla.org/en-US/docs/WebAssembly/Loading_and_running

I can see a few different options:

• Every consumer of the library has to use Webpack
• A Vite specific hack is added
• The browser variant uses the browser native way of loading WebAssembly https://developer.mozilla.org/en-US/docs/WebAssembly/Loading_and_running
  • I think the downside here might be that some tools will struggle to include the .wasm file in a production release. After all fetch("./brotli_wasm_bg.wasm") is slightly harder to statically analyze than the imports at the top of the file.

[pimterry]

Everything inside the pkg.bundler is boilerplate wrapper output generated by https://github.com/rustwasm/wasm-pack, the standard tool for building WASM from Rust, building for a 'bundler' target (built here).

That same code is used by more or less 100% of Rust+WASM packages that exist for JS.

If that part isn't working, then it's a Vite and/or wasm-pack bug, there's not much to do here. The only custom code in this repo is the promise wrapper in index.browser.js, the Rust code, the tiny build script, and tests.

[stefnotch]

I see, thank you very much.
Apparently there is an issue over there already rustwasm/wasm-pack#1106

As for the future, apparently there is an active proposal to properly fix this at a language level
https://github.com/WebAssembly/esm-integration/tree/main/proposals/esm-integration

[stefnotch]

I guess this can then be closed as a combination of "wontfix" and "upstream issue 🐟".

[pimterry]

Ok! Hope that points you towards a solution eventually. If you do find a fix and there are small changes that can be made here to support Vite without breaking the existing setup then PRs are very welcome 😄.

In the meantime I'm actually going to leave this open - it's still a real issue regardless, and maybe this will eventually point towards some useful info for any other Vite users hitting the same problem. Even if it is an upstream issue that wasm-pack can fix, we'll need to pull through an update to actually sort it here afterwards too.

[kyr0]

@stefnotch Have you found a solution? I just bumped in the same issue(s) in the same order ;)

[stefnotch]

@kyr0 Sadly not really. My workaround ended up being copying a lot of brotli-wasm's code into my project (don't forget about the licensing part), and only importing the .wasm part.

https://stefnotch.github.io/url-catpressor/#𝓒𝗮t𝓷𝝸𝕡ƽ_ℰ𝘨ｙｐ𝐭𝙞𝕒n-𐊰𝑎ꭒ_𝐋ⅇ𝛐ρɑ𝕣𝐝𝛖𝒔_𝙰𝕤𝒊а𝗇-S𝓮𝑚𝘪-׀۵ոℊ𝒽𝕒𝑖𝐫_ℙe𝙧ꮪ𝐢𝑎𝕟_ᗰeഠ𝑤𝒔_𝛣𝒓𝖺𝐳ɩℓ𝘪𝖺𝙣-Ѕ𝚑𝞼ꭇt𝒽𝓪ｉ𝗋_Ꮤh𝞲𝒕𝘦-С𝐚𝐭𝓈

[pimterry]

I don't have a good answer for you either @kyr0 but I think this is a general problem with wasm-pack, and you should ping them about it, e.g. over here: rustwasm/wasm-pack#1106. I expect this probably affects all Vite users for all Wasm-Pack projects (i.e. basically every Rust-via-WASM library anywhere) so if you're keen on Vite it's well worth getting this fixed more generally. Might also be worth opening an issue with Vite too to see if there's solutions on that side.

If you do get any information on ways to work around this, or if there's any improvements on the Wasm-Pack side then do share that here, I'd be happy to fix this in brotli-wasm if possible.

[stefnotch]

Update from the future: It now almost works out of the box.

https://github.com/stefnotch/url-catpressor/blob/a1ecbe032711363f64811c10f5bef2913d3d8f42/vite.config.ts#L9

https://github.com/stefnotch/url-catpressor/blob/main/src/useCompression.ts

[justin-schroeder]

☝️ this is the way

[miroljub1995]

We came to the same issue. Looks like there is a way to workaround it without changing vite config:

import init, * as brotli from "../../node_modules/brotli-wasm/pkg.web/brotli_wasm";
import wasmUrl from "../../node_modules/brotli-wasm/pkg.web/brotli_wasm_bg.wasm?url";

const brotliPromise = init(wasmUrl).then(() => brotli);

?url did the trick and .wasm gets copied to output with hash, which is cool.

The only problem here is that I need to specify relative path to module. When I try as brotli-wasm/pkg.web/brotli_wasm_bg.wasm?url, I get error from vite Internal server error: Missing "./pkg.web/brotli_wasm_bg.wasm" specifier in "brotli-wasm" package

@stefnotch @pimterry I'm not bundle expert, but I think some additional items should be added to exports along with the current?

"exports": {
    ".": {
      "import": "./index.web.js",
      "browser": "./index.browser.js",
      "require": "./index.node.js",
      "default": "./index.web.js"
    }
  }

After that it would be nice to use it like, or something better:

import init, * as brotli from "brotli-wasm/brotli_wasm";
import wasmUrl from "brotli-wasm/brotli_wasm_bg.wasm?url";

const brotliPromise = init(wasmUrl).then(() => brotli);