Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow returning Vec<T> #111

Closed
rookboom opened this issue Apr 9, 2018 · 60 comments · Fixed by #3554
Closed

Allow returning Vec<T> #111

rookboom opened this issue Apr 9, 2018 · 60 comments · Fixed by #3554
Labels
more-types Adding support for more Rust types to cross the boundary

Comments

@rookboom
Copy link

rookboom commented Apr 9, 2018

It would really be great to be able to return a vector of structs or tuples:

For example. I have the following type:

#[wasm_bindgen]
struct Range {
    offset: u32,
    length: u32
}

that I want to return in my function

#[wasm_bindgen]
pub fn token_ranges(text: &str) -> Vec<Range> 

I am getting this error:

   Compiling picl_wasm_runtime v0.1.0 (file:///A:/Repos/picl_native_runtime/bindings/typescript)
error[E0277]: the trait bound `std::boxed::Box<[Range]>: wasm_bindgen::convert::WasmBoundary` is not satisfied
  --> src\lib.rs:15:1
   |
15 | #[wasm_bindgen]
   | ^^^^^^^^^^^^^^^ the trait `wasm_bindgen::convert::WasmBoundary` is not implemented for `std::boxed::Box<[Range]>`
   |
   = help: the following implementations were found:
             <std::boxed::Box<[u16]> as wasm_bindgen::convert::WasmBoundary>
             <std::boxed::Box<[i16]> as wasm_bindgen::convert::WasmBoundary>
             <std::boxed::Box<[f32]> as wasm_bindgen::convert::WasmBoundary>
             <std::boxed::Box<[i32]> as wasm_bindgen::convert::WasmBoundary>
           and 5 others
   = note: required because of the requirements on the impl of `wasm_bindgen::convert::WasmBoundary` for `std::vec::Vec<Range>`

My workaround is to flatten my data and return a Vec and then splice my token ranges on the JS side. This is unfortunate...

How can I add a WasmBoundary trait for a custom type?
Is there a better way?

@alexcrichton
Copy link
Contributor

Thanks for the report!

Right now there's not a great way to do this in wasm-bindgen itself in terms of the conversion here has to be also accompanied with a conversion on the JS side which may be somewhat tricky. It should certainly be possible though given enough support!

@Hywan
Copy link
Contributor

Hywan commented Apr 11, 2018

Same issue here :-).

What kind of support do you need?

Hywan added a commit to Hywan/gutenberg-parser-rs that referenced this issue Apr 11, 2018
So, `wasm-bindgen` does not support `Vec<T>` (see
rustwasm/wasm-bindgen#111), so my quick and
dirty solution right now is to serialize the vector to JSON, and parse
it from the JS land.
@alexcrichton
Copy link
Contributor

@rookboom @Hywan do y'all basically need Vec<T> where T has #[wasm_bindgen] on it?

@Hywan
Copy link
Contributor

Hywan commented Apr 11, 2018

@alexcrichton Exactly, that's precisely my usecase.

@rookboom
Copy link
Author

Yes, that would be fantastic.

@alexcrichton
Copy link
Contributor

Ok this is then I think pretty similar to #104. We somehow need a way to communicate this to the CLI tool but currently the strategy for doing that is quite limited.

alexcrichton added a commit that referenced this issue Apr 13, 2018
This commit is a complete overhaul of how the `#[wasm_bindgen]` macro
communicates type information to the CLI tool, and it's done in a somewhat...
unconventional fashion.

Today we've got a problem where the generated JS needs to understand the types
of each function exported or imported. This understanding is what enables it to
generate the appropriate JS wrappers and such. We want to, however, be quite
flexible and extensible in types that are supported across the boundary, which
means that internally we rely on the trait system to resolve what's what.

Communicating the type information historically was done by creating a four byte
"descriptor" and using associated type projections to communicate that to the
CLI tool. Unfortunately four bytes isn't a lot of space to cram information like
arguments to a generic function, tuple types, etc. In general this just wasn't
flexible enough and the way custom references were treated was also already a
bit of a hack.

This commit takes a radical step of creating a **descriptor function** for each
function imported/exported. The really crazy part is that the `wasm-bindgen` CLI
tool now embeds a wasm interpreter and executes these functions when the CLI
tool is invoked. By allowing arbitrary functions to get executed it's now *much*
easier to inform `wasm-bindgen` about complicated structures of types. Rest
assured though that all these descriptor functions are automatically unexported
and gc'd away, so this should not have any impact on binary sizes

A new internal trait, `WasmDescribe`, is added to represent a description of all
types, sort of like a serialization of the structure of a type that
`wasm-bindgen` can understand. This works by calling a special exported function
with a `u32` value a bunch of times. This means that when we run a descriptor we
effectively get a `Vec<u32>` in the `wasm-bindgen` CLI tool. This list of
integers can then be parsed into a rich `enum` for the JS generation to work
with.

This commit currently only retains feature parity with the previous
implementation. I hope to soon solve issues like #123, #104, and #111 with this
support.
alexcrichton added a commit that referenced this issue Apr 14, 2018
This commit is a complete overhaul of how the `#[wasm_bindgen]` macro
communicates type information to the CLI tool, and it's done in a somewhat...
unconventional fashion.

Today we've got a problem where the generated JS needs to understand the types
of each function exported or imported. This understanding is what enables it to
generate the appropriate JS wrappers and such. We want to, however, be quite
flexible and extensible in types that are supported across the boundary, which
means that internally we rely on the trait system to resolve what's what.

Communicating the type information historically was done by creating a four byte
"descriptor" and using associated type projections to communicate that to the
CLI tool. Unfortunately four bytes isn't a lot of space to cram information like
arguments to a generic function, tuple types, etc. In general this just wasn't
flexible enough and the way custom references were treated was also already a
bit of a hack.

This commit takes a radical step of creating a **descriptor function** for each
function imported/exported. The really crazy part is that the `wasm-bindgen` CLI
tool now embeds a wasm interpreter and executes these functions when the CLI
tool is invoked. By allowing arbitrary functions to get executed it's now *much*
easier to inform `wasm-bindgen` about complicated structures of types. Rest
assured though that all these descriptor functions are automatically unexported
and gc'd away, so this should not have any impact on binary sizes

A new internal trait, `WasmDescribe`, is added to represent a description of all
types, sort of like a serialization of the structure of a type that
`wasm-bindgen` can understand. This works by calling a special exported function
with a `u32` value a bunch of times. This means that when we run a descriptor we
effectively get a `Vec<u32>` in the `wasm-bindgen` CLI tool. This list of
integers can then be parsed into a rich `enum` for the JS generation to work
with.

This commit currently only retains feature parity with the previous
implementation. I hope to soon solve issues like #123, #104, and #111 with this
support.
@alexcrichton alexcrichton added the more-types Adding support for more Rust types to cross the boundary label Apr 16, 2018
@gabrielcarneiro97
Copy link

gabrielcarneiro97 commented Apr 23, 2018

I'm with a similar issue here, but with a complex struct.

code:

#[derive(Clone)]
#[wasm_bindgen]
pub struct Coords {
    x: usize,
    y: usize
}

#[wasm_bindgen]
pub struct Cell {
    state: State,
    position: Coords,
    neighboors: Vec<Coords>,
    neighboors_alive: i32
}

#[wasm_bindgen]
impl Cell {
    pub fn new(state: State, position: Coords, neighboors: Vec<Coords>) -> Cell {
        Cell {
            state,
            position,
            neighboors,
            neighboors_alive: 0
        }
    }
}

error:

error[E0277]: the trait bound `std::boxed::Box<[Coords]>: wasm_bindgen::convert::FromWasmAbi` is not satisfied
  --> src\lib.rs:36:1
   |
36 | #[wasm_bindgen]
   | ^^^^^^^^^^^^^^^ the trait `wasm_bindgen::convert::FromWasmAbi` is not implemented for `std::boxed::Box<[Coords]>`
   |
   = help: the following implementations were found:
             <std::boxed::Box<[u16]> as wasm_bindgen::convert::FromWasmAbi>
             <std::boxed::Box<[wasm_bindgen::JsValue]> as wasm_bindgen::convert::FromWasmAbi>
             <std::boxed::Box<[u8]> as wasm_bindgen::convert::FromWasmAbi>
             <std::boxed::Box<[f32]> as wasm_bindgen::convert::FromWasmAbi>
           and 5 others
   = note: required because of the requirements on the impl of `wasm_bindgen::convert::FromWasmAbi` for `std::vec::Vec<Coords>`

@David-OConnor
Copy link
Member

David-OConnor commented May 22, 2018

Gabriel and rook - have y'all found a workaround? Solving or working around this would add much flexibility to WASM in Rust.

Eventually, being able to use HashMaps, or structs from other packages (Like ndarrays) would be nice, but having some type of collection that maps to JS arrays would be wonderful; not sure if I can continue my project without this.

@rookboom
Copy link
Author

My workaround is to pass JSON over the wasm boundary... Not ideal but works for now.

@Gisleburt
Copy link

Gisleburt commented Jan 17, 2019

For those looking for a workaround on this, if you can turn your data into a Vec<u8> or &[u8]

#[wasm_bindgen]
pub struct ByteStream {
    offset: *const u8,
    size: usize,
}

#[wasm_bindgen]
impl ByteStream {
    pub fn new(bytes: &[u8]) -> ByteStream {
        ByteStream {
            offset: bytes.as_ptr(),
            size: bytes.len(),
        }
    }

    pub fn offset(&self) -> *const u8 {
        self.offset
    }

    pub fn size(&self) -> usize {
        self.size
    }
}

A good example of how to use this is creating a texture in Rust to render in Javascript, so for example:

#[wasm_bindgen]
pub fn render() -> ByteStream {
  let texture = Vec::new();
  // ...
  ByteStream::new(&texture);
}
const texture = render();
const textureRaw = new Uint8ClampedArray(memory.buffer, texture.offset(), texture.size());
const image = new ImageData(textureRaw, width, height);

@fitzgen
Copy link
Member

fitzgen commented Jan 17, 2019

Rather than returning pointers and lengths manually, you can use this, which should be slightly less error prone: https://docs.rs/js-sys/0.3.9/js_sys/struct.Uint8Array.html#method.view

@stefan2718
Copy link

stefan2718 commented Jun 28, 2019

If anyone is still looking at this, I was able to work around this using Serde to serialize/deserialize the data. This was the guide I used: https://rustwasm.github.io/docs/wasm-bindgen/reference/arbitrary-data-with-serde.html

Edit: For those wanting to avoid JSON serialization, the guide above also includes a link to serde-wasm-bindgen which "leverages direct APIs for JavaScript value manipulation instead of passing data in a JSON format."

@theashguy
Copy link

theashguy commented Jul 20, 2019

I'm wanting to take this one step further and return Vec<js_sys::Function> which means no serde serialization for me (Maybe there is some kind of ref I could serialize if I dug into the internals?). The workaround I'm kicking about at the moment is to create the collection on the JS side and expose some methods for appending and cleaning up.

Something like...

window.vecCacheOfAnything = {
  append: function (key, item) {
    if(!window.vecCache.cache[key]) {
      window.vecCache.cache[key] = []
    }

    window.vecCache[key].push(item)
  },
  clear: function (key) {
    delete window.vecCache.cache[key]
  },
  cache: {}
}

Not suitable for prod work and by no means ideal or ergonomic but I'm really just thrashing around to see how far I can take rustwasm at the moment :D. Possibly an idea for anyone blocked by this issue and wanting to do things more complex. Also an extra AC for the team to add to the backlog!

@Zireael07
Copy link

This issue is kind of a blocker for pretty much any sort of a bigger project using rustwasm.

@Pauan
Copy link
Contributor

Pauan commented Sep 3, 2019

It's not a complete solution, but I created #1749 which adds in FromIterator for Array:

use js_sys::Array;

#[wasm_bindgen]
pub fn token_ranges(text: &str) -> Array {
    get_vec_somehow().into_iter().collect()
}

This means that now you can send Vec<T> to JS, you just have to return an Array and use .into_iter().collect() to convert the Vec<T> into an Array.

@dasoncheng
Copy link

Same issue here :-).

#[wasm_bindgen]
#[derive(Debug)]
pub struct HeartBeat {
    pub template: u8,
    pub classify: u8,
    pub index: u32,
    pub tr: u16,
    pub hr: u16,
    pub feature: [[f32; 20]; 3],
}
the trait bound `[[f32; 20]; 3]: wasm_bindgen::convert::traits::IntoWasmAbi` is not satisfied

the trait `wasm_bindgen::convert::traits::IntoWasmAbi` is not implemented for `[[f32; 20]; 3]`

help: the following implementations were found:
        <&'a [f32] as wasm_bindgen::convert::traits::IntoWasmAbi>
        <&'a [f64] as wasm_bindgen::convert::traits::IntoWasmAbi>
        <&'a [i16] as wasm_bindgen::convert::traits::IntoWasmAbi>
        <&'a [i32] as wasm_bindgen::convert::traits::IntoWasmAbi>
      and 20 othersrustc(E0277)
lib.rs(38, 1): the trait `wasm_bindgen::convert::traits::IntoWasmAbi` is not implemented for `[[f32; 20]; 3]`

@dragly
Copy link

dragly commented Oct 16, 2019

@Pauan Great! Thanks for looking into this.

However, I am encountering the following error now.

the trait `std::convert::AsRef<wasm_bindgen::JsValue>` is not implemented for `MyObject`
note: required because of the requirements on the impl of `std::iter::FromIterator<MyObject>` for `js_sys::Array`

from the following code:

#[wasm_bindgen]
struct MyObject {
    a: f32,
    b: f32,
}

#[wasm_bindgen]
pub fn test() -> Array {
    let objects = vec![
        MyObject {
            a: 123.0,
            b: 1024.1,
        },
        MyObject {
            a: 456.7,
            b: 1024.8,
        },
    ];
    objects.into_iter().collect()
}

Am I missing something?

This is using wasm-bindgen = "0.2.51". Maybe your change is not in yet? The error is different from before, however, so it seems like something changed.

@Pauan
Copy link
Contributor

Pauan commented Oct 17, 2019

@dragly As explained in the PR, you need to use .map(JsValue::from), like this:

objects.into_iter().map(JsValue::from).collect()

This is because structs are a Rust data type, and so you have to manually use JsValue::from to convert them into a JS data type (the same is true for other Rust data types like &str, i32, etc.).

alvarosan added a commit to alvarosan/saturno that referenced this issue Jan 25, 2020
@Kinrany
Copy link

Kinrany commented May 8, 2020

As a workaround, is it possible to also specify the TypeScript type returned by the function?

By default fn foo() -> Array is compiled into () => any[] instead of something like () => string[].

@Pauan
Copy link
Contributor

Pauan commented May 8, 2020

@Kinrany Yes, but it requires a bit of a hack:

#[wasm_bindgen]
extern "C" {
    #[wasm_bindgen(typescript_type = "Array<string>")]
    type MyArray;
}

(The typescript_type attribute can specify any TypeScript type, even complex types like |)

Now you just do fn foo() -> MyArray and use .unchecked_into::<MyArray>() to cast the Array into MyArray.

@ZhaoXiangXML
Copy link

@ZhaoXiangXML https://rustwasm.github.io/wasm-bindgen/reference/attributes/on-rust-exports/typescript_type.html

I end up using a macro to impl the slice types, with #[wasm_bindgen] and member functions length(), get() and set().

ben-xD added a commit to ben-xD/bounding_box_annotation_flutter that referenced this issue Dec 14, 2022
refactor: lots of refactoring to backend and client

WIP:
```
// TODO read rustwasm/wasm-bindgen#111 and use some workaround to send ImagesBytes to server.
```
@bushrat011899
Copy link

Is there any work on this issue still? I saw #2734 had a solution to this issue but needed additional help with passing tests. If a full resolution cannot be implemented without a breaking change, could the #[wasm_bindgen] macro be extended with an attribute similar to getter_with_clone which uses serialization to coerce function outputs into something JS compatible?

/// Current Workaround
#[wasm_bindgen]
pub fn manual_serialization() -> JsValue {
    serde_wasm_bindgen::to_value::<Vec<String>>(&vec![]).unwrap()
}

/// Proposed Alternative
#[wasm_bindgen(serialize_output)]
pub fn automatic_serialization() -> Vec<String> {
    vec![]
}

This appears to be the workaround I've seen the most commonly used, but has a massive flaw in that all type information is lost for the developer and wasm-bindgen itself for type annotations. I saw that #2649 was closed because of the potential hidden performance impact from a solution like this. I would think that adding this as a macro option would allow the ergonomics without hiding the issues.

@sinking-point
Copy link
Contributor

@bushrat011899

I have had no response from @chinedufn @alexcrichton regarding the failing test. I spent a lot of time on this PR and this is discouraging to say the least. I wonder if they just never look at PRs unless they pass all tests. In which case, I could assume an interpretation of the comment in the test, and get it to pass. That way I'd get their attention, and if I assumed the wrong interpretation they will tell me to change it. However the PR is quite old now and probably has merge conflicts with master to sort out. If I'm going to do more work on this I want some assurance that I'm not wasting my time.

@bushrat011899
Copy link

@bushrat011899

I have had no response from @chinedufn @alexcrichton regarding the failing test. I spent a lot of time on this PR and this is discouraging to say the least. I wonder if they just never look at PRs unless they pass all tests. In which case, I could assume an interpretation of the comment in the test, and get it to pass. That way I'd get their attention, and if I assumed the wrong interpretation they will tell me to change it. However the PR is quite old now and probably has merge conflicts with master to sort out. If I'm going to do more work on this I want some assurance that I'm not wasting my time.

That really is unfortunate, this seems like a really obvious feature to work on in my opinion. My first thought for something useful with WASM was to write a REST API client. That way, I could have type guarantees and unit tests that, for example, my Axum server was interacting with a web UI correctly. Kinda hard to do that if something simple like "give me a list of data" requires type erasure.

@bushrat011899
Copy link

bushrat011899 commented Feb 23, 2023

Because this issue still doesn't have a good workaround IMO, I've tried to make my own. Below is a single-file procedural attribute macro which modifies a function definition in-place to:

  1. Return a JsValue instead of whatever type it is supposed to return.
  2. Evaluate the function body (async or sync) and store in a temporary variable __result.
  3. Pass the __result to serde_wasm_bindgen to create a JsValue to return.

This doesn't fix the issue entirely. For example, wasm-bindgen will mark the TypeScript definition of this function as returning any (understandable, it is!), and serialization is not always required to cross the FFI boundary.

I haven't published a crate before, and this is small enough that I don't know if it really needs to be, but this is how I've worked around this pretty massive functionality gap in wasm-bindgen. Hope someone finds it useful!

Proc-Macro Crate lib.rs

use proc_macro::TokenStream;
use proc_macro2::TokenStream as TokenStream2;
use quote::quote;
use syn::{parse::Nothing, parse2, parse_quote, ItemFn, ReturnType, Type};

#[proc_macro_attribute]
pub fn wrap_jsvalue(args: TokenStream, input: TokenStream) -> TokenStream {
    let input: TokenStream2 = input.into();

    let function = parse2::<Nothing>(args.into()).and(parse2::<ItemFn>(input.clone()));

    let expanded = match function {
        Ok(function) => expand_macro(function),
        Err(parse_error) => {
            let compile_error = parse_error.to_compile_error();
            quote!(#compile_error #input)
        }
    };

    expanded.into()
}

fn expand_macro(mut function: ItemFn) -> TokenStream2 {
    let ReturnType::Type(arrow, return_type) = function.sig.output else {
        return quote!(#function);
    };

    let stmts = function.block.stmts;

    let result = if function.sig.asyncness.is_some() {
        quote!((move || async move { #(#stmts)* })().await)
    } else {
        quote!((move || { #(#stmts)* })())
    };

    function.block = Box::new(parse_quote!({
        let __result: #return_type = #result;

        serde_wasm_bindgen::to_value(&__result).unwrap()
    }));

    function.sig.output = ReturnType::Type(arrow, Box::new(Type::Verbatim(quote!(JsValue))));

    quote!(#function)
}

Primary Crate lib.rs

use wasm_bindgen::prelude::*;
use my_macros_crate::wrap_jsvalue;

#[wrap_jsvalue]
#[wasm_bindgen]
pub async fn test() -> Vec<String> {
    vec!["Hello".to_string(), "World".to_string()]
}

@fjarri
Copy link

fjarri commented Feb 23, 2023

@bushrat011899: I hit this issue (and similar ones with argument types) before, and tried to generalize it as https://docs.rs/wasm-bindgen-derive/latest/wasm_bindgen_derive/ . Would be interesting to include your macro there as well.

@bushrat011899
Copy link

@fjarri I'm not surprised, it seems like a pretty easy issue to hit! Feel free to take the code I wrote and include it if you like. It's straightforward enough that I'm sure any reasonable person would end up converging on this independently anyway.

@rossng
Copy link

rossng commented Sep 4, 2023

Great work @Liamolucko! I look forward to using this once released.

@gabrielDevlog
Copy link

Hi guys, great to see a PR has been merged.
Is there any ETA for this feature ? Or is it possible for you to trigger a new release, so we can have this feature using cargo add.

Thanks

@daxpedda
Copy link
Collaborator

daxpedda commented Oct 5, 2023

See #3530.

@samuelOsborne
Copy link

@daxpedda I checked that issue, still not released right ?

@daxpedda
Copy link
Collaborator

daxpedda commented Oct 9, 2023

No.
You can find all our releases in the release section.

@Teebor-Choka
Copy link

Teebor-Choka commented Oct 18, 2023

When's the next release? The last one was in June.

@daxpedda
Copy link
Collaborator

We don't have a timetable for a next release.
See #3530.

@mnjdhl
Copy link

mnjdhl commented Feb 2, 2024

I am also stuck with this issue:
image

Has the fix been released already?

@mnjdhl
Copy link

mnjdhl commented Feb 2, 2024

Even it doesn't work with vector of basic types such as Vec:
image

@daxpedda
Copy link
Collaborator

daxpedda commented Feb 2, 2024

Has the fix been released already?

Yes it has been.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
more-types Adding support for more Rust types to cross the boundary
Projects
None yet