Format of the custom section

[alexcrichton]

The general idea behind this tool is that source languages will encode data into a custom section of the wasm binary which the wasm-bindgen tool will decode and remove, using it to generate appropriate JS bindings.

Currently, however, this isn't what happens! Instead we shove data into a static as rustc doesn't expose a way to emit data into a custom section as-is. Additionally the static is just a totally random encoding that I got working, it's not principled at all. We should fix that!

I think it's relatively uncontroversial to use a custom section to communicate this information, and we'll continue to work around rustc's lack of ability to emit a custom section in the meantime and will otherwise take advantage of the ability to do this in rustc as soon as it becomes available! In the meantime, though, I think it may be useful to discuss the custom format as well.

Right now the wasm-bindgen-cli-shared crate shows off effectively what's encoded right now, the top-level entry being a Program and a wasm binary has multiple Program instances. That is, however, only what the bindgen tool currently uses and will likely change over time both in requirements and breadth. In that sense, does it makes sense to lock down a binary format just yet? Or maybe we can lock down an extensible format like JSON which you can easily add keys to over time?

One thing I definitely want to do in the near term is to version the wasm-bindgen information in the sense that if the CLI installed at a particular git rev finds information produced by a different git rev then it should immediately generate an error saying they're out of sync (as it's so unstable). Eventually this'll probably get removed and we'll want to have the producer and consumer versioned separately, but that may be awhile!

In any case, this is all just some off-the-cuff thinking, I'd love to hear others' thoughts on this too!

[alexcrichton]

cc @ashleygwilliams @lukewagner

[lukewagner]

Just from a quick glance at wasm-bindgen-cli-shared, it looks like you're in the right space. Walking over this carefully to see if we could map each bindgen type to a C++ signature would be useful at some point, but probably later. It sounds like, especially with versioning, you'll be able to iterate on the details of this custom section for while to come.

[alexcrichton]

As of today the custom section is a bunch of JSON blobs of this schema prefixed with a binary 4 byte length so they can be concatenated

[FreeMasen]

As an example here is what the output of the comments example looks like

gist

[afdw]

There is another problem with this approach. Consider this code:

#[wam_bindgen]
pub fn main() {
    web_sys::Window::alert_with_message("Hello world! (alert)");
    web_sys::console::log_1(&"Hello world! (console.log)".into());
}

It works fine. But, if you try to remove out call to alert, like so:

#[wam_bindgen]
pub fn main() {
    web_sys::console::log_1(&"Hello world! (console.log)".into());
}

Then it will compile, CLI will not produce any errors, but it will throw exception like these in runtime:

LinkError: import object field '__widl_f_log_1_' is not a Function

Uncaught (in promise) LinkError: WebAssembly Instantiation: Import #0 module="./wasmtest" function="__widl_f_log_1_" error: function import requires a callable

It turns out that there are no mentions of the __widl_f_log_1_ function (it meant to act like a proxy: it is called from WASM, and then it calls the target function, console.log in this case) in the JS file produced by CLI at all, that is why browsers fail to find the import.

CLI uses a custom section stored in a WASM file (that is an only input of it) to store a list of functions for which such proxy functions in JS should be generated. The custom section is emitted while expanding macros in the Rust compiler. It is done here: https://github.com/rustwasm/wasm-bindgen/blob/15654591074b3895c61401e638ef26d41d3066a9/crates/backend/src/codegen.rs#L111-L118

But, if you try the check WASM file, you will see that there is no __wasm_bindgen_unstable section for web-sys in it, only ones for js-sys, while in the WASM binary for the first example both will present.

I think that this happens when you do not use anything that is in web-sys itself, only in methods from submodules. To confirm that this is true, I have done some testing.

Unzip the test.zip file. It contains two crates: test123 and test456, the former depends on the later. Try to build test123 for the WASM target architecture (cd test123 and cargo build --target wasm32-unknown-unknown). Then do wasm2wat -v target/wasm32-unknown-unknown/debug/test123.wasm | grep -A 1 __wasm_bindgen_unstable. As you can see, the binary actually has the __wasm_bindgen_unstable section:

  BeginCustomSection('__wasm_bindgen_unstable', size: 27)
  EndCustomSection

Then comment out the test456::m(); line and repeat last steps. The output of the grep command will be empty.

I believe that it means that 2 examples in the begging correspond to these ones for this simple test:

#[no_mangle]
pub extern fn main() {
    test456::n::m();
    test456::m();
}

#[no_mangle]
pub extern fn main() {
    test456::n::m();
}

[alexcrichton]

Thanks for the investigation there @afdw! I don't think it's necessarily related to the format of the custom section though (this issue) and should be fixed by #846 in any case!

[FreeMasen]

I am curious of there would be any interest in moving to formalize this against something like Estree?

[alexcrichton]

Perhaps! Right now though it's pretty ad-hoc so it may not be worth formalizing too much. If we were to actually formalize it, though, that sounds good to me!

[alexcrichton]

I'm gonna go ahead and close this as the current encoding is "good enough" for now and we can always revisit it if/when we start coordinating with other producers!