std: Add a new wasm32-unknown-unknown target

[alexcrichton]

This commit adds a new target to the compiler: wasm32-unknown-unknown. This target is a reimagining of what it looks like to generate WebAssembly code from Rust. Instead of using Emscripten which can bring with it a weighty runtime this instead is a target which uses only the LLVM backend for WebAssembly and a "custom linker" for now which will hopefully one day be direct calls to lld.

Notable features of this target include:

• There is zero runtime footprint. The target assumes nothing exists other than the wasm32 instruction set.
• There is zero toolchain footprint beyond adding the target. No custom linker is needed, rustc contains everything.
• Very small wasm modules can be generated directly from Rust code using this target.
• Most of the standard library is stubbed out to return an error, but anything related to allocation works (aka HashMap, Vec, etc).
• Naturally, any #[no_std] crate should be 100% compatible with this new target.

This target is currently somewhat janky due to how linking works. The "linking" is currently unconditional whole program LTO (aka LLVM is being used as a linker). Naturally that means compiling programs is pretty slow! Eventually though this target should have a linker.

This target is also intended to be quite experimental. I'm hoping that this can act as a catalyst for further experimentation in Rust with WebAssembly. Breaking changes are very likely to land to this target, so it's not recommended to rely on it in any critical capacity yet. We'll let you know when it's "production ready".

Building yourself

First you'll need to configure the build of LLVM and enable this target

$ ./configure --target=wasm32-unknown-unknown --set llvm.experimental-targets=WebAssembly

Next you'll want to remove any previously compiled LLVM as it needs to be rebuilt with WebAssembly support. You can do that with:

$ rm -rf build

And then you're good to go! A ./x.py build should give you a rustc with the appropriate libstd target.

Test support

Currently testing-wise this target is looking pretty good but isn't complete. I've got almost the entire run-pass test suite working with this target (lots of tests ignored, but many passing as well). The core test suite is still getting LLVM bugs fixed to get that working and will take some time. Relatively simple programs all seem to work though!

In general I've only tested this with a local fork that makes use of LLVM 5 rather than our current LLVM 4 on master. The LLVM 4 WebAssembly backend AFAIK isn't broken per se but is likely missing bug fixes available on LLVM 5. I'm hoping though that we can decouple the LLVM 5 upgrade and adding this wasm target!

But the modules generated are huge!

It's worth nothing that you may not immediately see the "smallest possible wasm module" for the input you feed to rustc. For various reasons it's very difficult to get rid of the final "bloat" in vanilla rustc (again, a real linker should fix all this). For now what you'll have to do is:

cargo install --git https://github.com/alexcrichton/wasm-gc
wasm-gc foo.wasm bar.wasm

And then bar.wasm should be the smallest we can get it!

---

In any case for now I'd love feedback on this, particularly on the various integration points if you've got better ideas of how to approach them!

[alexcrichton]

I'd also like to doubly and triply emphasize how this target is a work in progress. The precise semantics of how it works and how everything gets hooked up will likely change over time, but I feel that it's also beneficial if we can get more testing of this backend and more testing of the integration, and hopefully start seeing some neat projects leveraging these benefits!

[tlively]

@alexcrichton I just want to say how exciting it is to see so much progress toward making wasm work well with Rust :) Thanks for doing all of this!

[shepmaster]

Let's pick a reviewer...

r? @eddyb

[shepmaster]

@alexcrichton you've some tidy failures:

[00:03:55] tidy error: /checkout/src/etc/wasm32-shim.js: incorrect license
[00:03:55] Expected a gate test for the feature 'wasm_import_memory'.
[00:03:55] Hint: create a file named 'feature-gate-wasm_import_memory.rs' in the compile-fail
[00:03:55]       test suite, with its failures due to missing usage of
[00:03:55]       #![feature(wasm_import_memory)].
[00:03:55] Hint: If you already have such a test and don't want to rename it,
[00:03:55]       you can also add a // gate-test-wasm_import_memory line to the test file.
[00:03:55] tidy error: Found 1 features without a gate test.
[00:03:56] some tidy checks failed

[eddyb]

LGTM but this is a pretty large change. cc @rust-lang/compiler @rust-lang/libs

[nagisa]

Stray print?

[alexcrichton]

Nah this is just intended to force an allocation somewhere that LLVM can't optimize away

[NikVolf]

is it really needed here?

[SimonSapin]

anything related to allocation works (aka HashMap, Vec, etc).

Out of curiosity, how does this work? Is there an allocator runtime?

[pepyakin]

@SimonSapin I believe yes.

[jasondavies]

This is great!

In general I've only tested this with a local fork that makes use of LLVM 5 rather than our current LLVM 4 on master.

If I simply clone this branch, run ./configure --target=wasm32-unknown-unknown --set llvm.experimental-targets=WebAssembly and ./x.py build, after much waiting I get an error:

Intrinsic has incorrect return type!
i32 (i32)* @llvm.wasm.grow.memory.i32
LLVM ERROR: Broken function found, compilation aborted!
error: Could not compile `dlmalloc`.

Does this mean I should be testing with LLVM 5, and if so, can you provide some simple instructions on how to do so?

Edit: I managed to build successfully using LLVM 5, using the --llvm-root option when calling ./configure.

[konstin]

There's a small inconsistency here: All the other crates use https://github.com/owner/repo.git instead of https://github.com/owner/repo

[alexcrichton]

@SimonSapin indeed there is! Wasm actually has memory allocation instructions which the allocator runtime leverages to implement a runtime allocator.

[alexcrichton]

@jasondavies oops sorry! Was pinned to the wrong commit of dlmalloc-rs, I believe it should work now.

[ElvishJerricco]

Does https://github.com/WebAssembly/lld/tree/wasm have to be merged upstream before you can just use the WebAssembly port of LLD?

[shepmaster]

New tidy errors:

Testing libstd stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[00:58:15]    Compiling libc v0.2.32
[00:58:15]    Compiling core v0.0.0 (file:///checkout/src/libcore)
[00:58:15]    Compiling std_unicode v0.0.0 (file:///checkout/src/libstd_unicode)
[00:58:15]    Compiling dlmalloc v0.0.0 (file:///checkout/src/rustc/dlmalloc_shim)
[00:58:15] error[E0464]: multiple matching crates for `libc`
[00:58:15]  --> /checkout/src/rustc/dlmalloc_shim/../../dlmalloc/src/linux.rs:1:1
[00:58:15]   |
[00:58:15] 1 | extern crate libc;
[00:58:15]   | ^^^^^^^^^^^^^^^^^^
[00:58:15]   |
[00:58:15]   = note: candidates:
[00:58:15]   = note: path: /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-658d35794c10b003.rlib
[00:58:15]   = note: crate name: libc
[00:58:15]   = note: path: /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-f32a17a3111b01aa.rlib
[00:58:15]   = note: crate name: libc
[00:58:15] 
[00:58:15] error[E0463]: can't find crate for `libc`
[00:58:15]  --> /checkout/src/rustc/dlmalloc_shim/../../dlmalloc/src/linux.rs:1:1
[00:58:15]   |
[00:58:15] 1 | extern crate libc;
[00:58:15]   | ^^^^^^^^^^^^^^^^^^ can't find crate
[00:58:15] 
[00:58:15] error: aborting due to 2 previous errors
[00:58:15] 
[00:58:15] error: Could not compile `dlmalloc`.

[alexcrichton]

@ElvishJerricco "just use the WebAssembly port of LLD" is actually quite involved, so I wanted to get something working (aka this PR) before starting on a new venture to discover even more integration and/or LLD bugs. This is the final state, though, to use LLD to link WebAsembly (AFAIK)

[killercup]

FYI: If someone wants to play with this branch and WASM in Node, I've pushed some of my experiments to this repo: https://github.com/killercup/wasm-experiments

[bors]

☔️ The latest upstream changes (presumably #45870) made this pull request unmergeable. Please resolve the merge conflicts.

[badboy]

😄. Judging by Math examples, this will require quite a bit of runtime on JS side, much like Emscripten's libraries, to make std work.

Luckily enough, a lot of things can still work with the current state

[RReverser]

@badboy Mostly just math works, yeah, but for real app that interacts with filesystem etc. you will need JS runtime.

[pepyakin]

Just passed by and wanted to note that WASM (despite it's name) is not only for the Web. By it's nature WASM, have no ABI and have no notion of filesystem and networking.

So this statement

Most of the standard library is stubbed out to return an error

have a lot of sense for me. For all other things there is Emscripten : )

[RReverser]

By it's nature WASM, have no ABI and have no notion of filesystem and networking.

Heh, so does x86, it doesn't mean they're not needed for apps :)

[pepyakin]

Yeah, Wasm (as x86) is just an ISA. fs and stuff like that is just a different level of abstraction.

For me wasm32-unknown-unknown seems more like a bare metal environment (like Arm MCU). “apps” for mcus doesnt require (and usually doesnt have a notion of) fs and networking. So is wasm

Also, as I mentioned before, you can use Rust with wasm32-unknown-emscripten triple with LLVM wasm backend (you just have to build it yourself). It already provides all these (and many other) abstractions!

[RReverser]

Please, again, I'm not trying to diminish the importance of this addition, no need to try and prove to me how awesome it is 😄 I'm just saying that applicability will be pretty limited until most of standard library gets bindings to JS side.

[badboy]

Let's work together and provide the missing pieces then (either as JavaScript shims or Rust-side implementations of functionality). I don't know how easy it will be to extract this stuff from Emscripten.

Anyway, this is not the place to discuss it.

[RReverser]

Anyway, this is not the place to discuss it.

It seemed like the best place (as it's related to the context of PR), but if you have other suggestions, sure.

[pepyakin]

My point is that many things from the standart library shouldn't get any bindings to host environment. For example, what means to spawn a process in wasm? What character should be used as a path seperator? How std::net should be implemented?
For me, these questions just doesn't make sense.
Also, I want to emphasize, that Web/JS is a important environment but it is not the only possible one!

Please, again, I'm not trying to diminish the importance of this addition, no need to try and prove to me how awesome it is

I didn't even think of that!
(btw sorry for my english)

[RReverser]

@pepyakin At least for Node.js side spawning process, net etc. are pretty easily implemented. Character for path can be taken from host system and so on. I think it makes sense to experiment with reusing Emscripten libraries for this as @badboy suggested.

[RReverser]

Also, I want to emphasize, that Web/JS is a important environment but it is not the only possible one!

Well other environments might provide own implementations of these system bindings if they need to.

[est31]

Anyway, this is not the place to discuss it.

A better place would be #44006 . It is a bit underused sadly :/

[lilianmoraru]

@alexcrichton Now, the linker has some support for wasm: https://reviews.llvm.org/rL318539

[pepyakin]

Thank you @est31 ! I've moved discussion there

[vmx]

@alexcrichton You mention LLVM 5. Where is your branch and how do I use it? I'm playing with the wasm target but get a segfault during compilation. Before I dig deeper I'd like to try a more recent LLVM as it might be fixed already.

[Dessix]

@RReverser the option of not having the extra libraries and runtimes bolted on enables use of this for embedded systems where including additional support code is unnecessary or even undesirable. Ensuring that remains an option would be an important consideration for at least these scenarios.

[RReverser]

@Dessix One absolutely doesn't contradict another, you can always just not include runtime and not use anything that requires it. But anyway, this discussion was moved to another issue, please see above.

[alexcrichton]

I've now also opened a PR to start running tests on CI over at #46291

[jethrogb]

Since this is basically the same as #37133, I wouldn't have expected this to get merged in this state in light of the comments there and in https://internals.rust-lang.org/t/refactoring-std-for-ultimate-portability/4301. Will you reconsider #37133 for merging if I rebase that?

[aturon]

@jethrogb it's definitely worth revisiting, yes. Note that the wasm target is a somewhat special case, since it exists at the liballoc level and will eventually include some of the libstd layer as well (e.g. threads). It's also a specific (highly desired) platform, so the motivation was pretty different.

For #37133, is the motivation primarily to have a kind of "template" for tackling other, similar cases?

re: the PAL, AFAIK not much has happened there, and I think the whole story should be rebooted. In particular, we should take into account (and implement...) the portability lint, which in the long term may allow us to do away with the core/std layering altogether.

In any case, before rebasing, I'd suggest at least an internals thread laying out the motivation and attempting to reach some consensus there. You could ping me, @alexcrichton and @sfackler on the thread to ensure visibility.

[aochagavia]

Looks like you forgot to add a #[link_name = "Math_fdim"] attribute

[alexcrichton]

Unfortunately Math.fdim isn't actually a function, and hopefully none of these will be necessary in the future regardless, it's mostly a bug that they're all here in the first place.