feat: add browser support (ECO-967) (ECO-1240)

[bryanchriswhite]

Status

Browser-applicable tests are passing (i.e. unit/bindings/* and unit/lib/index.test.ts). I assume that a web worker based analogue of the worker_thread could be constructed, this is out of scope for my immediate needs.

I also have this module linked locally (yarn link, etc.) in another project which is able to bundle it (once built) like a regular dependency. Once a release is cut which includes the emscripten binding, consumers should only have to yarn add it and ensure their webpack config is compatible. See README -> using emscripten bindings for more.

Missing Members

Some global members are still missing from the emscripten binding (as I haven't figured out how to bind them yet):

• BLS12_381_G1
• BLS12_381_NEG_G1
• BLS12_381_G2
• BLS12_381_NEG_G2

---

Changes

• add pre.js, post.js, and webpack.config.js to eslint ignore patterns
• add bundle.js, blst.wasm, and blst.wasm.js to gitignore
• move prebuild/* to prebuild/swig/*
• add prebuild/emscripten directory
  • JS wrapper code
  • CPP wrapper code
  • output dir for compiled JS and wasm
• add scripts to package.json:
  • inline:wasm - serializes wasm binary to base64 encoded JS, exported as an arrray buffer
  • test:browser - runs browser-compatible tests
• add webpack dependencies (incl. necessary polyfill / fallback dependencies):
  • webpack
  • webpack-cli
  • webpack-dev-server
  • ts-loader
  • path-browserify
  • stream-browserify
  • crypto-browserify
  • buffer
  • assert-browserify
• add WebIDL definition (used to generate initial bindings, not really useful going forward)
• dynamically require appropriate binding module based on environment detection (thanks @dapplion!)
• nest browser-applicable tests in run functions to wait for the emscripten runtime to be ready, if present
• update README to include details on how to build and use the emscripten bindings

Dependencies

• fetchai/blst#1

[dapplion]

Hey @bryanchriswhite ! Super curious about this PR, are you able to run blst JS bindings in the browser? If so, what's your strategy?

[bryanchriswhite]

@dapplion the plan is to use webidl in conjunction with emscripten to generate web assembly bindings. At the moment, I seem to have massaged the generated binding code to the point where it's building the WASM module and supporting JS. Now I'm setting up webpack to integrate this with blst-ts in place of the native node module to get to the point where I can get the blst-ts tests passing in the browser. 😄

(The blst submodule draft changes are in fetchai/blst#1)

[dapplion]

That's amazing! It would be down to help you upstream the browser support if you want. supranational has been very receptive in the past for modifications in the source to support bindings. What's the scope of the changes in blst C side?

[bryanchriswhite]

@dapplion thanks, that's great to hear! 😄

So far the changes in blst are to include the generated WebIDL binding C++ source in the build script along with any emcc-specific arguments like --post-js. I was planning on tidying up after I get things working, at the moment I'm sort-of trampling over things to expedite progress. 😅

I would love to hear your thoughts on how to structure things best so that browser / web assembly support can coexist upstream!

[dapplion]

@bryanchriswhite Once you got something stable and working I would upstream the changes in the blst repo. Keep modifications minimal and add the new bindings into their own folder along with the absolute minimal CI that ensures it works and shows how to integrate downstream. Then we can discuss in that PR with supranational if they prefer different structures.

[bryanchriswhite]

Hey @dapplion! I did my best to setup karma but it looks like the tests aren't run unless you start in one shell and run a few times in another. I haven't used karma in quite a while so it's not immediately obvious to me what I'm doing wrong. I also have mocha running (natively) in the browser which seems to work fine (and presents a friendlier in-browser UI). I added the relevant details to README -> Testing in the browser.

I've added a build script that uses docker, which seemed like the only sane option to me. I'm thinking that it will make sense to include the in-lined wasm binary in the npm release so that consumers don't need to build it.

As you may have noticed, I turned on actions for the fork but have not added anything to the workflow yet. The only changes I've made to the actions were to update paths to things that I moved. The benchmark is failing because of the missing members (see PR description); I removed them from the Blst interface temporarily.

[dapplion]

Amazing progress! How big in the target bin size? Inlining and bundling into the release is the right path IMO.

[bryanchriswhite]

Amazing progress! How big in the target bin size? Inlining and bundling into the release is the right path IMO.

Thanks! 😄

The last build of the in-line JS I did was ~282kB. The original wasm was ~212 kB.

[ejfitzgerald]

Just some suggestions for the C++ and the new stuff

[bryanchriswhite]

Thanks @ejfitzgerald, that eliminates all mallocs! 🙌

[bryanchriswhite]

@dapplion, any idea why the ARM jobs take soooo long, is that normal?

[dapplion]

@dapplion, any idea why the ARM jobs take soooo long, is that normal?

Github does not have ARM runners so we run QEMU with docker buildx. Since the ARM architecture is emulated it's extremely slow to run anything. As it has to install a lot of stuff and do an expensive compilation it takes forever.