Feature request: JS interface + WASM module

[bhouston]

This is a very interesting endeavor. Congratulations @elalish !

This is a wishlist item:

• JS interface so one can write JS/TS and integrate it with Three.js for display and manipulation.
• WASM module so you can use it directly in a browser.

[elalish]

Here here! Contributions welcome!

[FishOrBear]

Compile directly into wasm interface and publish npm package

[elalish]

Agreed; would you like to give it a shot?

[elalish]

@pca006132 Now that you got the emscripten build working for the tests, do you know how to make a build that's callable as a library? I'd love to make an example of calling this WASM blob from a simple three.js project, a la #75. Probably best to skip MeshIO and assimp in that build, to keep size down, but also because plenty of JS libraries like Three already support import/export.

[pca006132]

I am not familiar with emscripten, but it seems that you have to export the symbols as C symbols to avoid name mangling and use their tool to generate the binding: https://emscripten.org/docs/porting/connecting_cpp_and_javascript/Interacting-with-code.html#interacting-with-code-binding-cpp

[pca006132]

I'm having a look at this, but it seems that the API exposed to JS will not be very convenient to use as you will need to call delete everywhere: https://emscripten.org/docs/porting/connecting_cpp_and_javascript/embind.html#memory-management , or you will leak memory pretty quickly.

Writing the entire thing in C++ and compile to WASM is already possible for now.

[pca006132]

OK it seems that three.js also require calling dispose explicitly, so this should not be a problem. Wondering if it will be better to just create a C FFI, so binding to other languages will be easier (if we want to do that in the future).

[elalish]

three.js requires dispose to free up any GPU memory, as WebGL is a C-style API that requires explicit memory management. I think when it comes to C FFI, you know more than I do.

[makc]

perhaps this comment should have been made here. TL;DR right now emmake produces a bunch of .a files that can be put through embind if someone writes that one js file - then the result will be directly useable in either three.js or any other js library of user's choice.

edit: sorry, cpp file not js file, the one with EMSCRIPTEN_BINDINGS

[pca006132]

Indeed the structure looks like the one in pybind11. I guess we should do something similar to https://github.com/elalish/manifold/blob/master/tools/pymanifold.cpp

[elalish]

Agreed, but let's leave out ExportMesh this time, since we plan to factor that out of the python module eventually too.

[elalish]

@makc Would you be interested in making these changes? I see now you're a user of this, which puts you in a good position to test it and ensure ergonomics are good. I totally agree about wanting our WASM to be independent of other libraries like three.js, and without bloat like assimp and gtest. I'd also like to separately build a bit of JS to connect that WASM to three.js, but just as an example.

[makc]

unfortunately I am total noob when it comes to c++ tools like cmake 😅 for example, for my build I have manually removed CMakeLists sections related to assimp and gtest, when ideally they would be made conditional on EMSCRIPTEN - and I just did not know how to do it.

[makc]

what I could try to do is to create bindings file stub and make sure the whole process works, but then someone else would need to properly add this to the project.

[elalish]

That sounds great, we can help! Honestly we're all noobs at CMake too, but we're working through it together.

[pca006132]

Yes we are all noobs at CMake, I wasn't using CMake (modifying CMake files) before this project. I can help to review the changes and clean it up if needed. Sorry I am a bit busy recently so at best I can only do review and some tiny changes.

[makc]

(I will just mention #151 to ping who ever was subscribed to this)

[makc]

@bhouston they have just merged wasm module mvp in #155 - you can now PR stuff you need ✍️

[elalish]

Yeah, I suppose we could close this, but maybe better to leave it till we have the whole API exposed in WASM.

[pca006132]

I plan to improve the js example by:

• Adding a text editor next to the canvas, so users can write code and view the result. There will also be some examples for users to try.
• Add orbital control to the canvas (or is there any other threejs controls?)
• Instead of accessing Module.Manifold, users can call Manifold directly in the text editor, and objects created will be registered in a registry and deleted once the code execution is finished, to reduce memory usage.
• Expose more APIs such as different transformations.
• Improve the ergonomics of the API by allowing optional arguments and accept lists as vectors.

Any other thoughts?

[elalish]

@pca006132 Spectacular! This is exactly where I was hoping to go next with the project. I actually like the simplicity of the existing example, so let's build this JS editor separately from that. I'm planning to make another slightly more complicated example that shows how multi-material objects can be handled with three.js as well. Those will serve better as examples for others that want to integrate this library. I'll make us a landing page that links to all of them.

Regarding orbit controls, for our JS editor I'd actually like to integrate with <model-viewer> (my 80% project at Google). It's three.js-based and does (I like to think) a nice job with 3D user interaction. The simplest way to integrate is to use three's GLTFExporter to make a GLB blob, then pass that as the src to <model-viewer>. I'm happy to work on this part since I know it well.

Do you have any ideas for a good library to handle the live code editing? Ideally we'd want syntax highlighting and other goodies and we shouldn't reinvent the wheel. Things like JSbin and StackBlitz must use some underlying libraries that would be good for this. We might also want to think about TS types for our WASM binding.

[pca006132]

So <model-viewer> works with glb directly? That's great, maybe you can setup the page for it, I was thinking about orbit control because I want to reuse the threejs example but I have no idea about what kind of control it supports.

For code editing, I think the most popular option is code mirror. I'm not sure if we can have decent autocomplete with it though. Another option would be monaco which is the editor powering vscode, even though it says that it doesn't support mobile browsers, it seems pretty usable on my android tablet.

[pca006132]

For examples, I've been making my own keyboard (a clone of the moonlander because I want something that can be open sourced and less expensive):

I'm currently printing it for test, will clean up the code and put it as an example later :)

[elalish]

Looking at codemirror and monaco, I'm leaning toward codemirror. It looks like they do have autocompletion, though I'm not sure how much work is needed beyond their example. I also saw this comparison, and I especially note the difference in bundle size, which is big for page performance.

[elalish]

@pca006132 Do you have a sense of how you'd like to break up this work into manageable chunks? It might be good to file separate issues for these chunks so we can show dependencies and have a place to discuss each. I figure for now you can just copy the non-interactive three.js example to get started and once I've built #196 you can switch to that. Sound good?

[pca006132]

Yes, I can work on writing the API.

[elalish]

Yes, I can work on writing the API.

Great. A page integrating codemirror would also be great as a first step; then we have a platform on which to incrementally add features and test them.

[elalish]

This looks completed as of #224 🎉