Smoother experience for wasm-bindgen-cli

[Pauan]

Motivation

I am currently working on a Webpack loader for wasm-bindgen. This has a lot of advantages over the current Webpack plugin:

• It doesn't use wasm-pack, which makes it a lot easier to install and lighter-weight.

• It handles file watching a lot better (it uses Webpack's watcher rather than a custom watcher).

• It handles errors better.

• It's better integrated into Webpack.

• It's easier for the user to use.

• It's faster.

• The user doesn't need to create .js files and import things, everything Just Works(tm).

• And the biggest benefit of all: it has excellent support for multiple crates, which is something I need.

However, there is one major flaw: the wasm-bindgen-cli version must match with the wasm-bindgen version. Normally this version syncing is handled by wasm-pack.

However, since it doesn't use wasm-pack that means the user has to manually sync, which is really awful (especially if they use different versions of wasm-bindgen in different crates).

Proposed Solution

There should be a simple tool which only does wasm-bindgen-cli version syncing and nothing else. That tool can then be used by both wasm-pack and my Webpack loader.

Ideally wasm-bindgen-cli itself would do this syncing, automatically and transparently.

It would also be great if this syncing tool were distributed on npm, so it could be easily installed by JS users.

I'm willing to work on this tool.

[alexcrichton]

An interesting question! I think though that we want to avoid having multiple tools or multiple plugins to avoid confusion though? Would it be possible to upstream your improvements to the current plugin?

I'm curious what roadblocks you ran into using wasm-pack as well, in theory it shouldn't really add any overhead or be really that different than building through Cargo itself, but there may be bugs for us to work out!

[Pauan]

Would it be possible to upstream your improvements to the current plugin?

That isn't possible: I already improved the plugin as much as possible, but loaders are fundamentally different from plugins, so it requires a new package.

I should also point out this comment, and I should also point out that I contributed my loader to the wasm-tool org, which is the same org that manages the plugin. My intention is definitely not to fragment, instead it's to improve.

I'm curious what roadblocks you ran into using wasm-pack as well, in theory it shouldn't really add any overhead or be really that different than building through Cargo itself

It's not a roadblock (I have successfully used wasm-pack on many of my projects), it's about:

• Lowering dependencies for the user (they don't need to download wasm-pack or keep it up to date). This is actually a pretty big deal: any sort of extra friction turns users away.

• Improving reproducibility: wasm-pack has a single global installation, which makes things complicated if multiple crates rely upon different versions of wasm-pack, or if a crate relies upon an older version of wasm-pack.

• The loader doesn't need wasm-pack, since it's not publishing to the npm registry, and wasm-pack's npm dependencies support doesn't work for this use case, so it's just dead weight.

• wasm-pack (currently) doesn't support multiple crates, and it's not going to support multiple crates anytime soon: there are some complex questions regarding multiple crate support in wasm-pack.

What it comes down to is that wasm-pack just wasn't designed for this use case (seamless integration as a Webpack application). Which is fine, I don't think wasm-pack should handle every possible use case. It works well at what it does, there's no need for it to be an "everything and the kitchen sink" tool.

And since the fundamental issue is keeping wasm-bindgen-cli in sync with wasm-bindgen, it makes sense to me that wasm-bindgen should be the one to do that, rather than using an external tool (wasm-pack).

[alexcrichton]

Ah sorry I'm not really well versed in webpack lingo, so I didn't realize that a load and a plugin were a different thing. Does this mean we should deprecate the plugin and advertise the loader instead? Could the loader just replace the plugin's code?

I think there may also be some clarification needed around wasm-pack's purpose as a command. I think it may have started off on the wrong messaging foot talking about NPM and such so strongly, but it's intention has always been the same which is the tool that we recommend everyone use for all use cases involving wasm-bindgen. I agree it shouldn't become a weird kitchen sink hybrid but in the same way we don't ask you to use a different build of Cargo for Windows and Linux we expect wasm-pack to be used basically whenever you're using wasm-bindgen.

For some of your concerns I don't think that wasm-pack is really adding much of a dependency. The loader/plugin could always download a precompiled version if one isn't found, and similar to how users don't rely typically on older versions of Cargo it's not expected that you rely on older versions of wasm-pack. If there's any sort of reproducibility issue then that's something for wasm-pack to fix, it's sort of like all other reproducibility issues where "you just need the same version of the tools involved", and if it matters quite a lot presumably the loader could take care of that.

wasm-pack's only purpose isn't to serve the NPM registry, but also to ensure that all wasm-bindgen-cli matches wasm-bindgen, everything is called with the right flags, various sections are parsed out of wasm files, etc. Our entire toolchain makes the assumption you're using wasm-pack, so features run a risk of not working if you're using wasm-pack. Previous iterations of the npm deps RFC, for example, wouldn't work without wasm-pack since that's where it was implemented.

I think it's true that wasm-pack may not support everything today, but I'm not really sure how wasm-bindgen supports multiple crates any better than wasm-pack does. You need to run wasm-bindgen once-per-crate and presumably you'd just do the same with wasm-pack?

I understand that the purpose here isn't to split things but I think that if we start extracting tools that's exactly what will happen. We can always update wasm-pack to tweak its behavior and whatnot, but I don't think we'll want to invest in a separate tool to manage wasm-bindgen versions

[ashleygwilliams]

• Lowering dependencies for the user (they don't need to download wasm-pack or keep it up to date). This is actually a pretty big deal: any sort of extra friction turns users away.

We can make wasm-pack npm-installable (i've already done this for similar tools so the code is already existent and functionally cut and pastable.) This way the plugin can manage keeping wasm-pack up to date (which would not be that much overhead).

• Improving reproducibility: wasm-pack has a single global installation, which makes things complicated if multiple crates rely upon different versions of wasm-pack, or if a crate relies upon an older version of wasm-pack.

There shouldn't ever be a reason where a user needs to depend on old versions of wasm-pack, we don't really make breaking changes, and the few times we have we did long deprecation cycles and only touched CLI APIs that aren't used here.

• wasm-pack (currently) doesn't support multiple crates, and it's not going to support multiple crates anytime soon: there are some complex questions regarding multiple crate support in wasm-pack.

This is definitely a feature we want in wasm-pack and I think the friction you are discussing is a great forcing function for getting it prioritized.

What it comes down to is that wasm-pack just wasn't designed for this use case (seamless integration as a Webpack application). Which is fine, I don't think wasm-pack should handle every possible use case. It works well at what it does, there's no need for it to be an "everything and the kitchen sink" tool.

wasm-pack was originally conceived as a npm publishing tool, but it really is our integrated build tool now. This is exactly the usecase we want wasm-pack to be good at so I think that it's a good idea to see if we can improve wasm-pack to work well for it.

And since the fundamental issue is keeping wasm-bindgen-cli in sync with wasm-bindgen, it makes sense to me that wasm-bindgen should be the one to do that, rather than using an external tool (wasm-pack).

The wasm-bindgen CLI is really not designed for users. We have made significantly more breaking changes here, and we do so because the user of wasm-bindgen is wasm-pack which can help mitigate those breaking changes. I think it's important for us to keep this pattern- it gives us a lot of freedom to work on wasm-bindgen, since we have an interface layer in wasm-pack to help smooth it out.

Overall, I think this, will it doesn't intend to fracture the ecosystem, will likely do so. I think all the friction you've had with wasm-pack is stuff we want to solve in wasm-pack, and many of those things (npm installable, multi-crate support) are things we have open issues for on the project and are simply looking for contributors! I think it'd be great o focus this energy on wasm-pack, instead of a new plugin.

[Pauan]

Does this mean we should deprecate the plugin and advertise the loader instead?

That would be my suggestion, yes.

Could the loader just replace the plugin's code?

That's really not a good idea. Webpack has a naming convention where packages have either a -plugin or -loader suffix. So having a loader with a -plugin suffix is really confusing and bad form.

In addition, it would require a major version bump anyways, since plugins are not compatible at all with loaders. They're installed and used in completely separate ways, so changing from a plugin to a loader would break all user's projects.

So the cleanest solution is a new package.

I understand that the purpose here isn't to split things but I think that if we start extracting tools that's exactly what will happen. We can always update wasm-pack to tweak its behavior and whatnot, but I don't think we'll want to invest in a separate tool to manage wasm-bindgen versions

To be clear, I think this should be handled innately by wasm-bindgen-cli, automatically and transparently. In other words, it's not really a "separate tool", it's just wasm-bindgen-cli.

[Pauan]

@ashleygwilliams We can make wasm-pack npm-installable

Would that have support for installing a specific version of wasm-pack, such as "wasm-pack": "=0.8.1"?

This is exactly the usecase we want wasm-pack to be good at so I think that it's a good idea to see if we can improve wasm-pack to work well for it.

That's good to hear. These are the current changes that would be needed to make wasm-pack usable for this use case:

• Installable via npm (the loader should be able to just add wasm-pack into its dependencies and then the binary should be put into node_modules/.bin as usual).

• Some sort of "build, don't package" mode. Right now it does a lot of extra work: checking npm dependencies, outputting .gitignore, outputting package.json, etc.

  That all makes sense if you're publishing the package, but not if you just want to build a crate.

• A --quiet flag which would remove all of the [INFO] messages.

[ashleygwilliams]

thank you so much for listing that out! i'm working on wasm-pack triage today and i can file issues for the feature requests you've listed here that don't yet have issues :) let me know if you want to work on any of them- i think they are all relatively simple to accomplish!

[alexcrichton]

Lately I've sort of been coming around to this personally, and I'm wondering if we could kill two birds with one stone by getting a close-to-cargo-like workflow.

I could imagine a few tools here with a workflow that looks like:

• First, you execute cargo install cargo-wbindgen. Or maybe npm install cargo-wbindgen. Either way you're getting the same code, in the cargo install case you're compiling from source, and in the npm install case you're getting the same code compiled to wasm. Crucially, though, this package is tiny, no more than a few hundred lines and contains zero crate dependencies.

• Next, you execute cargo wbindgen build.

  • First, this checks to see if wasm-pack is installed. If not it installs it with the standard installer script.
  • Next, it executes wasm-pack build, forwarding along arguments as necessary
  • Eventually, wasm-pack executes and runs wasm-bindgen, dynamically managing the version of wasm-bindgen-cli.

• And... that's it! Then you've got your output in pkg/*

While it seems like there's a number of components strewn about here, I think this would be pretty simple from a user perspective where you wouldn't execute wasm-bindgen much but instead you'd just be executing cargo wbindgen. Ideally you'd just do cargo wbindgen test as well. Basically you'd do exactly what you do today for normal native projects, except you'd just throw wbindgen in the middle. We could then also figure out a way to reasonably pass wasm-bindgen options all the way down and through all the tools.

This would also give a good user experience from the perspective that cargo-wbindgen would be very quick to compile and download (or just download from NPM). That tiny shim would then download a precompiled version of the real workhorse (in this scenario wasm-pack) which would do everything else.

How's that sound to others? Perhaps a plausible way to move forward?

[Pauan]

@alexcrichton I'm in favor of anything which improves the experience (which is currently rather bad).

With cargo web this is the process:

1. Run cargo install cargo-web (only needs to be done once)

2. Run cargo web build (or cargo web start for a web server with auto-recompilation).

That's it. Everything Just Works(tm), It doesn't require mucking about with wasm-pack or npm or Webpack. It fits in perfectly with the Rust workflow: you just replace cargo build with cargo web build. The closer we can get the wasm-bindgen ecosystem to that, the better.

I've used both cargo web and wasm-pack extensively, and my experience (and the experience of others) is that cargo web is far more ergonomic and smoother to use.

Cargo web does have downsides: it produces much larger code, it doesn't have much support for customization, and it doesn't work very good for building libraries.

Wasm-pack is basically the opposite: great at building libraries, not so good at building applications, and the Webpack integration allows for a high amount of customization (but is overkill for simple projects).

I think we can have the benefits of cargo web while retaining the benefits of wasm-pack, getting the best of both worlds.

So I generally agree with your plan, except for a few tweaks:

• cargo wbindgen is much too long. I think we should use cargo wasm instead.

• Right now wasm-pack is rather focused on npm libraries, even its default wasm-pack build is designed for npm.

  This is different from cargo web build which is designed exclusively for applications.

  From what I've seen (and my own personal experience) people seem to be more focused on building wasm, not creating npm libraries.

  I've noticed far more Rust programmers wanting to take their existing Rust project and add a DOM UI to it, and not so many JS programmers wanting to add Rust into their JS projects. But that's just my experience.

  I'm not sure how to solve this, but maybe cargo wasm build should be for applications, and we would have a separate command for npm libraries, like cargo wasm npm. Or maybe a --lib flag or something.

• There should be a cargo wasm init command which initializes a template.

• We should have solid support for building without Webpack. In fact I would say that building without Webpack should be the default, with Webpack being reserved for more advanced users.

  I say that even though I've extensively used Webpack for several years. I quite like Webpack, it's a very good tool, but it is a complex beast which is hard to learn, so it really turns people away.

  Despite wasm-bindgen being the "blessed" ecosystem, we've had multiple people switch from wasm-bindgen to cargo web specifically because cargo web works without needing npm/Node/Webpack.

  We really need to focus on making things as simple and easy as possible. The fewer dependencies, the better. The less configuration files, the better. The less the user needs to think about, the better.

  Cargo web really shines because you can use it despite having zero JS experience. That's the sort of experience we need to have: a Rust programmer should be able to jump in and start writing code without needing to deal with any JS stuff.

  So that means we'll need to significantly beef up our support for --target no-modules, and also make no-modules the default.

  Of course we'll still provide a top-quality Webpack plugin for the situations where you do need the extra customization (or when you want to integrate Rust into an already existing Webpack project). It just won't be the default.

• Not really related to the cargo-wasm tool itself, but another area where cargo web really shines is that it has native support for bin crates and test crates.

  That means you can just create a regular Rust bin crate and use cargo web build and it Just Works(tm).

  And it means you can write unit tests in exactly the same way as Rust, except behind a feature flag.

  This is much nicer than the experience with wasm-bindgen, where you have to use a lib crate, set a bunch of options, import things, use wasm_bindgen_test::wasm_bindgen_test_configure!(run_in_browser);, use #[wasm_bindgen_test], etc.

  I really think we all need to take a look at how cargo web does things, and try to emulate it as best as we can in wasm-bindgen, because the experience with cargo web is so much nicer.