Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow returning Vec<T> #111

Open
rookboom opened this issue Apr 9, 2018 · 26 comments
Open

Allow returning Vec<T> #111

rookboom opened this issue Apr 9, 2018 · 26 comments
Labels

Comments

@rookboom
Copy link

@rookboom rookboom commented Apr 9, 2018

It would really be great to be able to return a vector of structs or tuples:

For example. I have the following type:

#[wasm_bindgen]
struct Range {
    offset: u32,
    length: u32
}

that I want to return in my function

#[wasm_bindgen]
pub fn token_ranges(text: &str) -> Vec<Range> 

I am getting this error:

   Compiling picl_wasm_runtime v0.1.0 (file:///A:/Repos/picl_native_runtime/bindings/typescript)
error[E0277]: the trait bound `std::boxed::Box<[Range]>: wasm_bindgen::convert::WasmBoundary` is not satisfied
  --> src\lib.rs:15:1
   |
15 | #[wasm_bindgen]
   | ^^^^^^^^^^^^^^^ the trait `wasm_bindgen::convert::WasmBoundary` is not implemented for `std::boxed::Box<[Range]>`
   |
   = help: the following implementations were found:
             <std::boxed::Box<[u16]> as wasm_bindgen::convert::WasmBoundary>
             <std::boxed::Box<[i16]> as wasm_bindgen::convert::WasmBoundary>
             <std::boxed::Box<[f32]> as wasm_bindgen::convert::WasmBoundary>
             <std::boxed::Box<[i32]> as wasm_bindgen::convert::WasmBoundary>
           and 5 others
   = note: required because of the requirements on the impl of `wasm_bindgen::convert::WasmBoundary` for `std::vec::Vec<Range>`

My workaround is to flatten my data and return a Vec and then splice my token ranges on the JS side. This is unfortunate...

How can I add a WasmBoundary trait for a custom type?
Is there a better way?

@alexcrichton
Copy link
Contributor

@alexcrichton alexcrichton commented Apr 9, 2018

Thanks for the report!

Right now there's not a great way to do this in wasm-bindgen itself in terms of the conversion here has to be also accompanied with a conversion on the JS side which may be somewhat tricky. It should certainly be possible though given enough support!

@Hywan
Copy link
Contributor

@Hywan Hywan commented Apr 11, 2018

Same issue here :-).

What kind of support do you need?

Hywan added a commit to Hywan/gutenberg-parser-rs that referenced this issue Apr 11, 2018
So, `wasm-bindgen` does not support `Vec<T>` (see
rustwasm/wasm-bindgen#111), so my quick and
dirty solution right now is to serialize the vector to JSON, and parse
it from the JS land.
@alexcrichton
Copy link
Contributor

@alexcrichton alexcrichton commented Apr 11, 2018

@rookboom @Hywan do y'all basically need Vec<T> where T has #[wasm_bindgen] on it?

@Hywan
Copy link
Contributor

@Hywan Hywan commented Apr 11, 2018

@alexcrichton Exactly, that's precisely my usecase.

@rookboom
Copy link
Author

@rookboom rookboom commented Apr 11, 2018

Yes, that would be fantastic.

@alexcrichton
Copy link
Contributor

@alexcrichton alexcrichton commented Apr 12, 2018

Ok this is then I think pretty similar to #104. We somehow need a way to communicate this to the CLI tool but currently the strategy for doing that is quite limited.

alexcrichton added a commit that referenced this issue Apr 13, 2018
This commit is a complete overhaul of how the `#[wasm_bindgen]` macro
communicates type information to the CLI tool, and it's done in a somewhat...
unconventional fashion.

Today we've got a problem where the generated JS needs to understand the types
of each function exported or imported. This understanding is what enables it to
generate the appropriate JS wrappers and such. We want to, however, be quite
flexible and extensible in types that are supported across the boundary, which
means that internally we rely on the trait system to resolve what's what.

Communicating the type information historically was done by creating a four byte
"descriptor" and using associated type projections to communicate that to the
CLI tool. Unfortunately four bytes isn't a lot of space to cram information like
arguments to a generic function, tuple types, etc. In general this just wasn't
flexible enough and the way custom references were treated was also already a
bit of a hack.

This commit takes a radical step of creating a **descriptor function** for each
function imported/exported. The really crazy part is that the `wasm-bindgen` CLI
tool now embeds a wasm interpreter and executes these functions when the CLI
tool is invoked. By allowing arbitrary functions to get executed it's now *much*
easier to inform `wasm-bindgen` about complicated structures of types. Rest
assured though that all these descriptor functions are automatically unexported
and gc'd away, so this should not have any impact on binary sizes

A new internal trait, `WasmDescribe`, is added to represent a description of all
types, sort of like a serialization of the structure of a type that
`wasm-bindgen` can understand. This works by calling a special exported function
with a `u32` value a bunch of times. This means that when we run a descriptor we
effectively get a `Vec<u32>` in the `wasm-bindgen` CLI tool. This list of
integers can then be parsed into a rich `enum` for the JS generation to work
with.

This commit currently only retains feature parity with the previous
implementation. I hope to soon solve issues like #123, #104, and #111 with this
support.
alexcrichton added a commit that referenced this issue Apr 14, 2018
This commit is a complete overhaul of how the `#[wasm_bindgen]` macro
communicates type information to the CLI tool, and it's done in a somewhat...
unconventional fashion.

Today we've got a problem where the generated JS needs to understand the types
of each function exported or imported. This understanding is what enables it to
generate the appropriate JS wrappers and such. We want to, however, be quite
flexible and extensible in types that are supported across the boundary, which
means that internally we rely on the trait system to resolve what's what.

Communicating the type information historically was done by creating a four byte
"descriptor" and using associated type projections to communicate that to the
CLI tool. Unfortunately four bytes isn't a lot of space to cram information like
arguments to a generic function, tuple types, etc. In general this just wasn't
flexible enough and the way custom references were treated was also already a
bit of a hack.

This commit takes a radical step of creating a **descriptor function** for each
function imported/exported. The really crazy part is that the `wasm-bindgen` CLI
tool now embeds a wasm interpreter and executes these functions when the CLI
tool is invoked. By allowing arbitrary functions to get executed it's now *much*
easier to inform `wasm-bindgen` about complicated structures of types. Rest
assured though that all these descriptor functions are automatically unexported
and gc'd away, so this should not have any impact on binary sizes

A new internal trait, `WasmDescribe`, is added to represent a description of all
types, sort of like a serialization of the structure of a type that
`wasm-bindgen` can understand. This works by calling a special exported function
with a `u32` value a bunch of times. This means that when we run a descriptor we
effectively get a `Vec<u32>` in the `wasm-bindgen` CLI tool. This list of
integers can then be parsed into a rich `enum` for the JS generation to work
with.

This commit currently only retains feature parity with the previous
implementation. I hope to soon solve issues like #123, #104, and #111 with this
support.
@gabrielcarneiro97
Copy link

@gabrielcarneiro97 gabrielcarneiro97 commented Apr 23, 2018

I'm with a similar issue here, but with a complex struct.

code:

#[derive(Clone)]
#[wasm_bindgen]
pub struct Coords {
    x: usize,
    y: usize
}

#[wasm_bindgen]
pub struct Cell {
    state: State,
    position: Coords,
    neighboors: Vec<Coords>,
    neighboors_alive: i32
}

#[wasm_bindgen]
impl Cell {
    pub fn new(state: State, position: Coords, neighboors: Vec<Coords>) -> Cell {
        Cell {
            state,
            position,
            neighboors,
            neighboors_alive: 0
        }
    }
}

error:

error[E0277]: the trait bound `std::boxed::Box<[Coords]>: wasm_bindgen::convert::FromWasmAbi` is not satisfied
  --> src\lib.rs:36:1
   |
36 | #[wasm_bindgen]
   | ^^^^^^^^^^^^^^^ the trait `wasm_bindgen::convert::FromWasmAbi` is not implemented for `std::boxed::Box<[Coords]>`
   |
   = help: the following implementations were found:
             <std::boxed::Box<[u16]> as wasm_bindgen::convert::FromWasmAbi>
             <std::boxed::Box<[wasm_bindgen::JsValue]> as wasm_bindgen::convert::FromWasmAbi>
             <std::boxed::Box<[u8]> as wasm_bindgen::convert::FromWasmAbi>
             <std::boxed::Box<[f32]> as wasm_bindgen::convert::FromWasmAbi>
           and 5 others
   = note: required because of the requirements on the impl of `wasm_bindgen::convert::FromWasmAbi` for `std::vec::Vec<Coords>`
@David-OConnor
Copy link
Member

@David-OConnor David-OConnor commented May 22, 2018

Gabriel and rook - have y'all found a workaround? Solving or working around this would add much flexibility to WASM in Rust.

Eventually, being able to use HashMaps, or structs from other packages (Like ndarrays) would be nice, but having some type of collection that maps to JS arrays would be wonderful; not sure if I can continue my project without this.

@rookboom
Copy link
Author

@rookboom rookboom commented May 23, 2018

My workaround is to pass JSON over the wasm boundary... Not ideal but works for now.

@Gisleburt
Copy link

@Gisleburt Gisleburt commented Jan 17, 2019

For those looking for a workaround on this, if you can turn your data into a Vec<u8> or &[u8]

#[wasm_bindgen]
pub struct ByteStream {
    offset: *const u8,
    size: usize,
}

#[wasm_bindgen]
impl ByteStream {
    pub fn new(bytes: &[u8]) -> ByteStream {
        ByteStream {
            offset: bytes.as_ptr(),
            size: bytes.len(),
        }
    }

    pub fn offset(&self) -> *const u8 {
        self.offset
    }

    pub fn size(&self) -> usize {
        self.size
    }
}

A good example of how to use this is creating a texture in Rust to render in Javascript, so for example:

#[wasm_bindgen]
pub fn render() -> ByteStream {
  let texture = Vec::new();
  // ...
  ByteStream::new(&texture);
}
const texture = render();
const textureRaw = new Uint8ClampedArray(memory.buffer, texture.offset(), texture.size());
const image = new ImageData(textureRaw, width, height);
@fitzgen
Copy link
Member

@fitzgen fitzgen commented Jan 17, 2019

Rather than returning pointers and lengths manually, you can use this, which should be slightly less error prone: https://docs.rs/js-sys/0.3.9/js_sys/struct.Uint8Array.html#method.view

@stefan2718
Copy link

@stefan2718 stefan2718 commented Jun 28, 2019

If anyone is still looking at this, I was able to work around this using Serde to serialize/deserialize the data. This was the guide I used: https://rustwasm.github.io/docs/wasm-bindgen/reference/arbitrary-data-with-serde.html

Edit: For those wanting to avoid JSON serialization, the guide above also includes a link to serde-wasm-bindgen which "leverages direct APIs for JavaScript value manipulation instead of passing data in a JSON format."

@theashguy
Copy link

@theashguy theashguy commented Jul 20, 2019

I'm wanting to take this one step further and return Vec<js_sys::Function> which means no serde serialization for me (Maybe there is some kind of ref I could serialize if I dug into the internals?). The workaround I'm kicking about at the moment is to create the collection on the JS side and expose some methods for appending and cleaning up.

Something like...

window.vecCacheOfAnything = {
  append: function (key, item) {
    if(!window.vecCache.cache[key]) {
      window.vecCache.cache[key] = []
    }

    window.vecCache[key].push(item)
  },
  clear: function (key) {
    delete window.vecCache.cache[key]
  },
  cache: {}
}

Not suitable for prod work and by no means ideal or ergonomic but I'm really just thrashing around to see how far I can take rustwasm at the moment :D. Possibly an idea for anyone blocked by this issue and wanting to do things more complex. Also an extra AC for the team to add to the backlog!

@Zireael07
Copy link

@Zireael07 Zireael07 commented Aug 15, 2019

This issue is kind of a blocker for pretty much any sort of a bigger project using rustwasm.

@Pauan
Copy link
Contributor

@Pauan Pauan commented Sep 3, 2019

It's not a complete solution, but I created #1749 which adds in FromIterator for Array:

use js_sys::Array;

#[wasm_bindgen]
pub fn token_ranges(text: &str) -> Array {
    get_vec_somehow().into_iter().collect()
}

This means that now you can send Vec<T> to JS, you just have to return an Array and use .into_iter().collect() to convert the Vec<T> into an Array.

@dasoncheng
Copy link

@dasoncheng dasoncheng commented Sep 19, 2019

Same issue here :-).

#[wasm_bindgen]
#[derive(Debug)]
pub struct HeartBeat {
    pub template: u8,
    pub classify: u8,
    pub index: u32,
    pub tr: u16,
    pub hr: u16,
    pub feature: [[f32; 20]; 3],
}
the trait bound `[[f32; 20]; 3]: wasm_bindgen::convert::traits::IntoWasmAbi` is not satisfied

the trait `wasm_bindgen::convert::traits::IntoWasmAbi` is not implemented for `[[f32; 20]; 3]`

help: the following implementations were found:
        <&'a [f32] as wasm_bindgen::convert::traits::IntoWasmAbi>
        <&'a [f64] as wasm_bindgen::convert::traits::IntoWasmAbi>
        <&'a [i16] as wasm_bindgen::convert::traits::IntoWasmAbi>
        <&'a [i32] as wasm_bindgen::convert::traits::IntoWasmAbi>
      and 20 othersrustc(E0277)
lib.rs(38, 1): the trait `wasm_bindgen::convert::traits::IntoWasmAbi` is not implemented for `[[f32; 20]; 3]`
@dragly
Copy link

@dragly dragly commented Oct 16, 2019

@Pauan Great! Thanks for looking into this.

However, I am encountering the following error now.

the trait `std::convert::AsRef<wasm_bindgen::JsValue>` is not implemented for `MyObject`
note: required because of the requirements on the impl of `std::iter::FromIterator<MyObject>` for `js_sys::Array`

from the following code:

#[wasm_bindgen]
struct MyObject {
    a: f32,
    b: f32,
}

#[wasm_bindgen]
pub fn test() -> Array {
    let objects = vec![
        MyObject {
            a: 123.0,
            b: 1024.1,
        },
        MyObject {
            a: 456.7,
            b: 1024.8,
        },
    ];
    objects.into_iter().collect()
}

Am I missing something?

This is using wasm-bindgen = "0.2.51". Maybe your change is not in yet? The error is different from before, however, so it seems like something changed.

@Pauan
Copy link
Contributor

@Pauan Pauan commented Oct 17, 2019

@dragly As explained in the PR, you need to use .map(JsValue::from), like this:

objects.into_iter().map(JsValue::from).collect()

This is because structs are a Rust data type, and so you have to manually use JsValue::from to convert them into a JS data type (the same is true for other Rust data types like &str, i32, etc.).

alvarosan added a commit to alvarosan/saturno that referenced this issue Jan 25, 2020
@Kinrany
Copy link

@Kinrany Kinrany commented May 8, 2020

As a workaround, is it possible to also specify the TypeScript type returned by the function?

By default fn foo() -> Array is compiled into () => any[] instead of something like () => string[].

@Pauan
Copy link
Contributor

@Pauan Pauan commented May 8, 2020

@Kinrany Yes, but it requires a bit of a hack:

#[wasm_bindgen]
extern "C" {
    #[wasm_bindgen(typescript_type = "Array<string>")]
    type MyArray;
}

(The typescript_type attribute can specify any TypeScript type, even complex types like |)

Now you just do fn foo() -> MyArray and use .unchecked_into::<MyArray>() to cast the Array into MyArray.

Kinrany added a commit to Kinrany/likelike-online-rs that referenced this issue May 8, 2020
@tlaukkan
Copy link

@tlaukkan tlaukkan commented May 12, 2020

It would still be awesome to have builtin support for something like this which is quite normal case:

#[wasm_bindgen]
pub struct IntersectResult {
    pub hit: bool,
    pub triangle_index: u32,
    pub u: f32,
    pub v: f32,
    pub distance: f32,
}

#[wasm_bindgen]
pub struct IntersectResultArray {
    pub intersects: Vec<IntersectResult>,
}
@i-schuetz
Copy link

@i-schuetz i-schuetz commented Oct 24, 2020

What do we have to do currently to achieve this? E.g to pass Vec<Foo> to JS:

struct Foo {
    field: i32,
    str: String,
}
@fjarri
Copy link

@fjarri fjarri commented Nov 25, 2020

@i-schuetz : the workaround that I use is having a return type of Vec<JsValue> and converting your vector to it on return: myvec.iter().map(JsValue::from).collect().

@fjarri
Copy link

@fjarri fjarri commented Nov 25, 2020

Does this issue also cover the support for Vec<T> function arguments? Because that's also impossible at the moment, and much harder to work around.

sliminality added a commit to sliminality/raidtomi that referenced this issue Jan 5, 2021
Due to wasm-bindgen's limitations [1], we need a constructor that
takes a `Vec<u32>` since we cannot take a `Vec<Nature>`. As a result, in
Rust-land we go through annoying acrobatics to convert a `Vec<Nature>`
into a `Vec<u32>` everywhere.

Instead, we simply implement a `NatureFilter::from_natures` method that
is not exposed to JS. This allows our tests to pass a `Vec<Nature>`,
which is much more ergonomic.

[1] rustwasm/wasm-bindgen#111
@ppershing
Copy link

@ppershing ppershing commented Apr 3, 2021

Unfortunately, solutions with js_sys have huge performance overheads.
You can also return Box<[T]> with T being basic numeric type which is efficient on the JS side (it just slices relevant wasm memory). This however needs a copy on the Rust side. In general, it is faster than js_sys though.

What I come up with (and seems to be faster by a margin) is to have the following wrapper type:

pub struct MySlice<T> {
    phantom: std::marker::PhantomData::<T>,
    _ptr: u32,
    _len: u32,
}

impl<T: wasm_bindgen::describe::WasmDescribe> wasm_bindgen::describe::WasmDescribe for MySlice<T> {
    fn describe() {
        wasm_bindgen::describe::inform(wasm_bindgen::describe::REF);
        wasm_bindgen::describe::inform(wasm_bindgen::describe::SLICE);
        T::describe();
    }
}

impl<T: wasm_bindgen::describe::WasmDescribe> wasm_bindgen::convert::IntoWasmAbi for MySlice<T> {
    type Abi=wasm_bindgen::convert::WasmSlice;

    #[inline]
    fn into_abi(self) -> wasm_bindgen::convert::WasmSlice {
        wasm_bindgen::convert::WasmSlice {
            ptr: self._ptr,
            len: self._len,
        }
    }
}

impl<T> std::convert::From<&Vec<T>> for MySlice<T> {
    fn from(vec: &Vec<T>) -> Self {
        let _ptr = vec.as_ptr() as u32;
        let _len = vec.len() as u32;
        
        Self {
            phantom: std::marker::PhantomData,
            _ptr,
            _len
        }
    }
}

This is both zero-copy on the rust side and single copy between wasm memory and JS on the JS side.
Basically, the type just emulates ref to a slice through standard wasm bindgen API (which means support for returning &[T] should be relatively easy to add if somebody knows internals of bindgen)
A caveat is that you shouldn't save MySlice, it should only be used to return immediate data to JS (because it detaches pointers, the code cannot guarantee lifetimes)

Usage is simple:

#[wasm_bindgen]
pub struct Container {
    data: Vec<f64>,
}

#[wasm_bindgen]
impl Container {
    pub fn read(&self) -> MySlice<f64> {
        (&self.data).into()
    }
}

with the JS side using

const c = Container.new() // somehow create
const values = c.read()
@Zireael07
Copy link

@Zireael07 Zireael07 commented Apr 3, 2021

You can also return Box<[T]> with T being basic numeric type

This is about Vec where T is a struct of some sort, so how is that relevant?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet