plugin-wasm: Unexpected errors after compilation

[nzakas]

• Rollup Plugin Name: rollup/plugin-wasm
• Rollup Plugin Version: 6.0.1
• Rollup Version: 3.3.0
• Operating System (or Browser): Windows
• Node Version: 18.10.0
• Link to reproduction (⚠️ read below):

https://github.com/humanwhocodes/momoa/tree/rust

Note: This requires the Rust toolchain to be installed. You can find the installation directions here:
https://github.com/humanwhocodes/momoa/tree/rust#development

• The main entrypoint is src/index.js
• The main function exported from wasm is tokenize_js().
• The output from wasm-bindgen ends up in the build directory.
• This should build upon running npm install, but you can also manually build using npm run build.

Expected Behavior

Project should compile to api.js in the root of the project, and I should be able to load that into Node.js using require() and then execute tokenize_js("true");

Actual Behavior

I get a lot of warnings during the build (these are incorrect, the file does export all of these symbols):

Output log

memory is not exported by build/momoa_bg.wasm
40: function getUint8Memory0() {
41:     if (cachedUint8Memory0.byteLength === 0) {
42:         cachedUint8Memory0 = new Uint8Array(wasm.memory.buffer);
                                                     ^
43:     }
44:     return cachedUint8Memory0;
build/momoa_bg.js
memory is not exported by build/momoa_bg.wasm
175: function getInt32Memory0() {
176:     if (cachedInt32Memory0.byteLength === 0) {
177:         cachedInt32Memory0 = new Int32Array(wasm.memory.buffer);
                                                      ^
178:     }
179:     return cachedInt32Memory0;
build/momoa_bg.js
__wbindgen_malloc is not exported by build/momoa_bg.wasm
184: */
185: export function tokenize_js(input) {
186:     const ptr0 = passStringToWasm0(input, wasm.__wbindgen_malloc, wasm.__wbindgen_realloc);
                                                    ^
187:     const len0 = WASM_VECTOR_LEN;
188:     const ret = wasm.tokenize_js(ptr0, len0);
build/momoa_bg.js
__wbindgen_realloc is not exported by build/momoa_bg.wasm
184: */
185: export function tokenize_js(input) {
186:     const ptr0 = passStringToWasm0(input, wasm.__wbindgen_malloc, wasm.__wbindgen_realloc);
                                                                            ^
187:     const len0 = WASM_VECTOR_LEN;
188:     const ret = wasm.tokenize_js(ptr0, len0);
build/momoa_bg.js
tokenize_js is not exported by build/momoa_bg.wasm
186:     const ptr0 = passStringToWasm0(input, wasm.__wbindgen_malloc, wasm.__wbindgen_realloc);
187:     const len0 = WASM_VECTOR_LEN;
188:     const ret = wasm.tokenize_js(ptr0, len0);
                          ^
189:     return takeObject(ret);
190: }
build/momoa_bg.js
__wbindgen_malloc is not exported by build/momoa_bg.wasm
243: export function __wbindgen_debug_string(arg0, arg1) {
244:     const ret = debugString(getObject(arg1));
245:     const ptr0 = passStringToWasm0(ret, wasm.__wbindgen_malloc, wasm.__wbindgen_realloc);
                                                  ^
246:     const len0 = WASM_VECTOR_LEN;
247:     getInt32Memory0()[arg0 / 4 + 1] = len0;
build/momoa_bg.js
__wbindgen_realloc is not exported by build/momoa_bg.wasm
243: export function __wbindgen_debug_string(arg0, arg1) {
244:     const ret = debugString(getObject(arg1));
245:     const ptr0 = passStringToWasm0(ret, wasm.__wbindgen_malloc, wasm.__wbindgen_realloc);
                                                                          ^
246:     const len0 = WASM_VECTOR_LEN;
247:     getInt32Memory0()[arg0 / 4 + 1] = len0;

Additionally, when I require('./api.js') with Node.js and run tokenize_js("true"), I get an error:

$ node
Welcome to Node.js v18.10.0.
Type ".help" for more information.
> const momoa = require("./api");
undefined
> momoa.tokenize_js("true")
Uncaught TypeError: malloc is not a function
    at passStringToWasm0 (C:\Users\nzaka\projects\humanwhocodes\momoa\api.js:1189:21)   
    at Object.tokenize_js (C:\Users\nzaka\projects\humanwhocodes\momoa\api.js:1227:18)  
>

So while the file api.js is built, it appears to be built in such a way that this function can't actually be executed.

All of the other JavaScript-based functions work correctly, it's just the wasm-based function that breaks.

Additional Information

[shellscape]

@nzakas before our script closes this for an invalid reproduction, please re-read the template and add a valid repro. we just don't have the maintainers or time available to parse your project, and we have the requirements in that template for very good reason

[nzakas]

My repo is pretty minimal as a test case, but if that's not enough, I'd at least appreciate some pointers as to what might cause the type of warning I'm seeing and why some symbols aren't being generated. I'm happy to dig in myself, but I need to know where to start.

[nzakas]

I've verified that all of the exports Rollup complains about not being present are, in fact, present in the wasm file. This seems to be the main issue: that Rollup isn't finding these exports and so the generated file is substituting undefined where those functions should be.

[shellscape]

@nickbabcock was our last contributor for the wasm plugin, so they might be able to help out here. unfortunately I'm not sure if any of our maintainers have taken ownership on this plugin. It'll be next week before I'll have the opportunity to dabble.

[nickbabcock]

I see the provided project invokes wasm-pack without --target web, which is what I and others use even when not building for the web (the rollup plugin will emit node specific code as appropriate).

So try switching over to the web target and adjust code accordingly. I find web target much better regardless of bundler or environment, as it increases flexibility for when the wasm is compiled / instantiated.

---

As for background, the crux of the problem, is that the bundler output from wasm-pack / wasm-bindgen contains:

import * as wasm from "./momoa_bg.wasm";

And the output is assuming that this wasm is compiled and instantiated. However, what rollup gives you is a function that can fetch, base64 decode, or read the wasm file off the file system.

[nzakas]

Ah interesting. So I tried wasm-pack --target web and that means I need to load the .wasm file with fetch(), which is what I was trying to avoid.

I'm assuming that Rollup is targeting files that were made by something other than wasm-bindgen? Would it make sense to make a wasm-bindgen compatible mode too? (Again, happy to contribute if someone can point me in the right direction.)

However, what rollup gives you is a function that can fetch, base64 decode, or read the wasm file off the file system.

I think this tidbit, plus some examples of how to set up your files to work with the Rollup wasm plugin, would by very helpful to add to the README.

[nickbabcock]

I'm AFK at the moment, so my answers may be a bit too vague. My apologies and let me know what I should follow up on.

that means I need to load the .wasm file with fetch(), which is what I was trying to avoid

The entrypoint of the emitted wasm loader from bindgen doesn't require fetch(). You can pass other parameters to it.

Would it make sense to make a wasm-bindgen compatible mode too

This idea definitely has merit for non-rust users, as I'd essentially embed what the web target from wasm-bindgen emits. It's a surprisingly complex topic to setup a library to cater to all of these use cases:

• node.js users that read off the file system
• electron users that use webpack
• Users who just want a single file off a CDN without any configurations (base64 inlined)
• Browser users who want a more efficient way of instantiation (fetch wasm separately)
• Users who don't want the base64 payload from the resulting bundle at all
• Users who pass in a precompiled WebAssembly.Module

The web target + the wasm plugin as is, allows for all these use cases. I'm not familiar with the other rollup plugins, but I don't think they are intrusive as far as manipulating package.json. It seems like it would be better for a higher-level tool to generate everything required to support these use cases.

I have published several npm packages that do all the rigmarole required:

• jomini
• highwayhasher

I also have published an article Recommendations when publishing a Wasm library that dives into the solution (if I wasn't AFK, I'd update the article as the suggestion resolving around updating the wasm plugin has been merged). Lots to think about, and I'm definitely interested in simplifying the process as I want to make it as easy as possible for js users to leverage wasm for every situation.

[nzakas]

The entrypoint of the emitted wasm loader from bindgen doesn't require fetch(). You can pass other parameters to it.

This is the bit that is really hard for me to grok — this doesn’t seem to be my experience at all, and I haven’t been able to find anything online actually showing a complete, nontrivial example.

I’ll dig through your blog posts to see if I can piece things together without bothering you.

[nickbabcock]

Yeah it's pretty bleeding edge stuff, so no worries.

Another way to look at it. The initialization function emitted from the wasm-bindgen web target takes the following parameter type (if supplied):

type InitInput = RequestInfo | URL | Response | BufferSource | WebAssembly.Module

The WebAssembly.Module parameter type is important as that allows this plugin to read off the file system or base64 decode an inlined payload (or fetch, etc), and pass in the already compiled WebAssembly.Module to the web target initialization function.

[nzakas]

Okay, so I've been digging around and I think I'm starting to get somewhere.

I set wasm-pack --target browser and tried this:

// wasm-bindgen outputted JS file
import { initSync, tokenize_js } from "../build/momoa.js";

// wasm-bindgen outputted wasm file
import wasm from "../build/momoa_bg.wasm";

// this doesn't work!!!
initSync(wasm);

// export the function
export {tokenize_js};

With the Rollup wasm plugin, wasm becomes a function that expects imports to be passed in, which is a problem because I don't have access to the imports necessary to initiate the module -- this is determined by the getImports() method inside of build/momoa.js, which is not exported. Further the base64-encoded version of the module is inside of the wasm() function, so I don't have access to it.

What is the intended way for me to get the correct imports to instantiate the module?

[nickbabcock]

imports is an optional argument that you should leave off or give undefined, otherwise the rollup plugin will give you WebAssembly.Instance instead of an WebAssembly.Module, which is what the wasm-bindgen initialization function wants to turn that into an WebAssembly.Instance for you.

In short, try this:

-initSync(wasm);
+initSync(wasm());

I'm not familiar with the browser target, I assumed you meant web.

[nzakas]

Yes, sorry, I meant --target web.

Okay, so when I tried what you suggested, I got an error. wasm() returns a Promise by default, so passing that directly into initSync() threw an error. However, if I updated my config to specify that the wasm file should be loaded synchronously, then it works. Here's my final config:

const {wasm} = require("@rollup/plugin-wasm");

module.exports = {
    input: 'src/index.js',
    output: {
        file: 'api.mjs',
        format: 'esm'
    },
    plugins: [
        wasm({
            targetEnv: "auto-inline",
            maxFileSize: 0,
            sync: ["build/momoa_bg.wasm"]
        })]
};

Does that seem correct? If so, I'd be happy to update the README with some specific instructions for wasm-bindgen users.

[nickbabcock]

Nice, glad you got up and running. Though I strongly caution against writing any instructions that use sync compilation as the only use case for that are wasm modules that are either trivial or destined exclusively for web workers where synchronous compilations make sense.

When packaging a wasm library for wider use, the shortest configuration that can check all boxes is something like:

    env != "slim" &&
      wasm(
        env == "node"
          ? { maxFileSize: 0, targetEnv: "node" }
          : { targetEnv: "auto-inline" }
      ),

• When distributing a slim build, we don’t worry about the Wasm at all as the downstream developer will be handling that. Make sure to include the wasm in the package exports for the downstream developer to instantiate as they please. This creates the ultimate flexibility for environments where either Wasm compilation is forbidden or the cost of the inlined base64 Wasm is too great (bandwidth / compute / caching (as base64 wasm can't be cached)). It does require the library in question to expose an entrypoint to receive Wasm (I tend to use the same the InitInput argument as wasm-bindgen).
• If we’re targeting a node environment, copy the Wasm to a separate file and omit browser specific code (like fetch) from the Wasm helper module.
• Else we’re inlining the Wasm as base64, so decode it using environment specific code (which doesn’t use Node.js builtin modules). I called this method auto-inline, as it AUTOmatically decodes INLINE Wasm based on if we’re in a node environment or not (since it should still be possible to run the UMD or any other importable build with Wasm inlined in node).

All the nuance is frustrating, and there's more that I've omitted for the sake of time. I want there to be better docs, but seems easy to overwhelm newcomers as just working with Wasm can be cumbersome, as you've seen.

[nzakas]

Ooph my head hurts. 😅 I need to spend some time digesting all of this. I’m trying to create a package that works in both Node and in the browser, so I was wanting to avoid both bundling a separate wasm file and also async initialization so I could still use CommonJS. But maybe I just need to bite the bullet and go ESM so o can use top-level await to init the module. All the methods the package exposes are sync and I don’t want consumers to have to change their code.

Lots more to think about.

[nickbabcock]

You and me both. It can be awkward to saddle the downstream developer with coordinating an async initialization. Though hopefully the pain can be mitigated if the initialization returns an API that users are familiar with:

import { initParser, type Parser } from "my-parser";

// function that uses an API with all expected sync methods and
// keeps backwards compatibility with previous API
function parseObject(parser: Parser) {
    const out = parser.lex("{}");
    // ...
}

/// the async initialization
initParser().then(parseObject);

I think it would be prudent to start mentally preparing for the breaking change that async initialization will bring.