New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

std: Add a new wasm32-unknown-unknown target #45905

Merged
merged 1 commit into from Nov 20, 2017

Conversation

@alexcrichton
Member

alexcrichton commented Nov 10, 2017

This commit adds a new target to the compiler: wasm32-unknown-unknown. This target is a reimagining of what it looks like to generate WebAssembly code from Rust. Instead of using Emscripten which can bring with it a weighty runtime this instead is a target which uses only the LLVM backend for WebAssembly and a "custom linker" for now which will hopefully one day be direct calls to lld.

Notable features of this target include:

  • There is zero runtime footprint. The target assumes nothing exists other than the wasm32 instruction set.
  • There is zero toolchain footprint beyond adding the target. No custom linker is needed, rustc contains everything.
  • Very small wasm modules can be generated directly from Rust code using this target.
  • Most of the standard library is stubbed out to return an error, but anything related to allocation works (aka HashMap, Vec, etc).
  • Naturally, any #[no_std] crate should be 100% compatible with this new target.

This target is currently somewhat janky due to how linking works. The "linking" is currently unconditional whole program LTO (aka LLVM is being used as a linker). Naturally that means compiling programs is pretty slow! Eventually though this target should have a linker.

This target is also intended to be quite experimental. I'm hoping that this can act as a catalyst for further experimentation in Rust with WebAssembly. Breaking changes are very likely to land to this target, so it's not recommended to rely on it in any critical capacity yet. We'll let you know when it's "production ready".

Building yourself

First you'll need to configure the build of LLVM and enable this target

$ ./configure --target=wasm32-unknown-unknown --set llvm.experimental-targets=WebAssembly

Next you'll want to remove any previously compiled LLVM as it needs to be rebuilt with WebAssembly support. You can do that with:

$ rm -rf build

And then you're good to go! A ./x.py build should give you a rustc with the appropriate libstd target.

Test support

Currently testing-wise this target is looking pretty good but isn't complete. I've got almost the entire run-pass test suite working with this target (lots of tests ignored, but many passing as well). The core test suite is still getting LLVM bugs fixed to get that working and will take some time. Relatively simple programs all seem to work though!

In general I've only tested this with a local fork that makes use of LLVM 5 rather than our current LLVM 4 on master. The LLVM 4 WebAssembly backend AFAIK isn't broken per se but is likely missing bug fixes available on LLVM 5. I'm hoping though that we can decouple the LLVM 5 upgrade and adding this wasm target!

But the modules generated are huge!

It's worth nothing that you may not immediately see the "smallest possible wasm module" for the input you feed to rustc. For various reasons it's very difficult to get rid of the final "bloat" in vanilla rustc (again, a real linker should fix all this). For now what you'll have to do is:

cargo install --git https://github.com/alexcrichton/wasm-gc
wasm-gc foo.wasm bar.wasm

And then bar.wasm should be the smallest we can get it!


In any case for now I'd love feedback on this, particularly on the various integration points if you've got better ideas of how to approach them!

@alexcrichton

This comment has been minimized.

Show comment
Hide comment
@alexcrichton

alexcrichton Nov 10, 2017

Member

I'd also like to doubly and triply emphasize how this target is a work in progress. The precise semantics of how it works and how everything gets hooked up will likely change over time, but I feel that it's also beneficial if we can get more testing of this backend and more testing of the integration, and hopefully start seeing some neat projects leveraging these benefits!

Member

alexcrichton commented Nov 10, 2017

I'd also like to doubly and triply emphasize how this target is a work in progress. The precise semantics of how it works and how everything gets hooked up will likely change over time, but I feel that it's also beneficial if we can get more testing of this backend and more testing of the integration, and hopefully start seeing some neat projects leveraging these benefits!

@tlively

This comment has been minimized.

Show comment
Hide comment
@tlively

tlively Nov 10, 2017

Contributor

@alexcrichton I just want to say how exciting it is to see so much progress toward making wasm work well with Rust :) Thanks for doing all of this!

Contributor

tlively commented Nov 10, 2017

@alexcrichton I just want to say how exciting it is to see so much progress toward making wasm work well with Rust :) Thanks for doing all of this!

@shepmaster

This comment has been minimized.

Show comment
Hide comment
@shepmaster

shepmaster Nov 10, 2017

Member

Let's pick a reviewer...

r? @eddyb

Member

shepmaster commented Nov 10, 2017

Let's pick a reviewer...

r? @eddyb

@shepmaster

This comment has been minimized.

Show comment
Hide comment
@shepmaster

shepmaster Nov 10, 2017

Member

@alexcrichton you've some tidy failures:

[00:03:55] tidy error: /checkout/src/etc/wasm32-shim.js: incorrect license
[00:03:55] Expected a gate test for the feature 'wasm_import_memory'.
[00:03:55] Hint: create a file named 'feature-gate-wasm_import_memory.rs' in the compile-fail
[00:03:55]       test suite, with its failures due to missing usage of
[00:03:55]       #![feature(wasm_import_memory)].
[00:03:55] Hint: If you already have such a test and don't want to rename it,
[00:03:55]       you can also add a // gate-test-wasm_import_memory line to the test file.
[00:03:55] tidy error: Found 1 features without a gate test.
[00:03:56] some tidy checks failed
Member

shepmaster commented Nov 10, 2017

@alexcrichton you've some tidy failures:

[00:03:55] tidy error: /checkout/src/etc/wasm32-shim.js: incorrect license
[00:03:55] Expected a gate test for the feature 'wasm_import_memory'.
[00:03:55] Hint: create a file named 'feature-gate-wasm_import_memory.rs' in the compile-fail
[00:03:55]       test suite, with its failures due to missing usage of
[00:03:55]       #![feature(wasm_import_memory)].
[00:03:55] Hint: If you already have such a test and don't want to rename it,
[00:03:55]       you can also add a // gate-test-wasm_import_memory line to the test file.
[00:03:55] tidy error: Found 1 features without a gate test.
[00:03:56] some tidy checks failed
@eddyb

This comment has been minimized.

Show comment
Hide comment
@eddyb

eddyb Nov 10, 2017

Member

LGTM but this is a pretty large change. cc @rust-lang/compiler @rust-lang/libs

Member

eddyb commented Nov 10, 2017

LGTM but this is a pretty large change. cc @rust-lang/compiler @rust-lang/libs

@@ -39,8 +38,7 @@ unsafe impl<'a> Alloc for &'a A {
static GLOBAL: A = A;
fn main() {
env::set_var("FOO", "bar");
drop(env::var("FOO"));
println!("hello!");

This comment has been minimized.

@nagisa

nagisa Nov 10, 2017

Contributor

Stray print?

@nagisa

nagisa Nov 10, 2017

Contributor

Stray print?

This comment has been minimized.

@alexcrichton

alexcrichton Nov 10, 2017

Member

Nah this is just intended to force an allocation somewhere that LLVM can't optimize away

@alexcrichton

alexcrichton Nov 10, 2017

Member

Nah this is just intended to force an allocation somewhere that LLVM can't optimize away

@SimonSapin

This comment has been minimized.

Show comment
Hide comment
@SimonSapin

SimonSapin Nov 10, 2017

Contributor

anything related to allocation works (aka HashMap, Vec, etc).

Out of curiosity, how does this work? Is there an allocator runtime?

Contributor

SimonSapin commented Nov 10, 2017

anything related to allocation works (aka HashMap, Vec, etc).

Out of curiosity, how does this work? Is there an allocator runtime?

@pepyakin

This comment has been minimized.

Show comment
Hide comment
@pepyakin
Contributor

pepyakin commented Nov 10, 2017

@jasondavies

This comment has been minimized.

Show comment
Hide comment
@jasondavies

jasondavies Nov 10, 2017

This is great!

In general I've only tested this with a local fork that makes use of LLVM 5 rather than our current LLVM 4 on master.

If I simply clone this branch, run ./configure --target=wasm32-unknown-unknown --set llvm.experimental-targets=WebAssembly and ./x.py build, after much waiting I get an error:

Intrinsic has incorrect return type!
i32 (i32)* @llvm.wasm.grow.memory.i32
LLVM ERROR: Broken function found, compilation aborted!
error: Could not compile `dlmalloc`.

Does this mean I should be testing with LLVM 5, and if so, can you provide some simple instructions on how to do so?

Edit: I managed to build successfully using LLVM 5, using the --llvm-root option when calling ./configure.

jasondavies commented Nov 10, 2017

This is great!

In general I've only tested this with a local fork that makes use of LLVM 5 rather than our current LLVM 4 on master.

If I simply clone this branch, run ./configure --target=wasm32-unknown-unknown --set llvm.experimental-targets=WebAssembly and ./x.py build, after much waiting I get an error:

Intrinsic has incorrect return type!
i32 (i32)* @llvm.wasm.grow.memory.i32
LLVM ERROR: Broken function found, compilation aborted!
error: Could not compile `dlmalloc`.

Does this mean I should be testing with LLVM 5, and if so, can you provide some simple instructions on how to do so?

Edit: I managed to build successfully using LLVM 5, using the --llvm-root option when calling ./configure.

Show outdated Hide outdated .gitmodules
@alexcrichton

This comment has been minimized.

Show comment
Hide comment
@alexcrichton

alexcrichton Nov 10, 2017

Member

@SimonSapin indeed there is! Wasm actually has memory allocation instructions which the allocator runtime leverages to implement a runtime allocator.

Member

alexcrichton commented Nov 10, 2017

@SimonSapin indeed there is! Wasm actually has memory allocation instructions which the allocator runtime leverages to implement a runtime allocator.

@alexcrichton

This comment has been minimized.

Show comment
Hide comment
@alexcrichton

alexcrichton Nov 10, 2017

Member

@jasondavies oops sorry! Was pinned to the wrong commit of dlmalloc-rs, I believe it should work now.

Member

alexcrichton commented Nov 10, 2017

@jasondavies oops sorry! Was pinned to the wrong commit of dlmalloc-rs, I believe it should work now.

@ElvishJerricco

This comment has been minimized.

Show comment
Hide comment
@ElvishJerricco

ElvishJerricco Nov 11, 2017

Does https://github.com/WebAssembly/lld/tree/wasm have to be merged upstream before you can just use the WebAssembly port of LLD?

ElvishJerricco commented Nov 11, 2017

Does https://github.com/WebAssembly/lld/tree/wasm have to be merged upstream before you can just use the WebAssembly port of LLD?

@shepmaster

This comment has been minimized.

Show comment
Hide comment
@shepmaster

shepmaster Nov 11, 2017

Member

New tidy errors:

Testing libstd stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[00:58:15]    Compiling libc v0.2.32
[00:58:15]    Compiling core v0.0.0 (file:///checkout/src/libcore)
[00:58:15]    Compiling std_unicode v0.0.0 (file:///checkout/src/libstd_unicode)
[00:58:15]    Compiling dlmalloc v0.0.0 (file:///checkout/src/rustc/dlmalloc_shim)
[00:58:15] error[E0464]: multiple matching crates for `libc`
[00:58:15]  --> /checkout/src/rustc/dlmalloc_shim/../../dlmalloc/src/linux.rs:1:1
[00:58:15]   |
[00:58:15] 1 | extern crate libc;
[00:58:15]   | ^^^^^^^^^^^^^^^^^^
[00:58:15]   |
[00:58:15]   = note: candidates:
[00:58:15]   = note: path: /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-658d35794c10b003.rlib
[00:58:15]   = note: crate name: libc
[00:58:15]   = note: path: /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-f32a17a3111b01aa.rlib
[00:58:15]   = note: crate name: libc
[00:58:15] 
[00:58:15] error[E0463]: can't find crate for `libc`
[00:58:15]  --> /checkout/src/rustc/dlmalloc_shim/../../dlmalloc/src/linux.rs:1:1
[00:58:15]   |
[00:58:15] 1 | extern crate libc;
[00:58:15]   | ^^^^^^^^^^^^^^^^^^ can't find crate
[00:58:15] 
[00:58:15] error: aborting due to 2 previous errors
[00:58:15] 
[00:58:15] error: Could not compile `dlmalloc`.
Member

shepmaster commented Nov 11, 2017

New tidy errors:

Testing libstd stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[00:58:15]    Compiling libc v0.2.32
[00:58:15]    Compiling core v0.0.0 (file:///checkout/src/libcore)
[00:58:15]    Compiling std_unicode v0.0.0 (file:///checkout/src/libstd_unicode)
[00:58:15]    Compiling dlmalloc v0.0.0 (file:///checkout/src/rustc/dlmalloc_shim)
[00:58:15] error[E0464]: multiple matching crates for `libc`
[00:58:15]  --> /checkout/src/rustc/dlmalloc_shim/../../dlmalloc/src/linux.rs:1:1
[00:58:15]   |
[00:58:15] 1 | extern crate libc;
[00:58:15]   | ^^^^^^^^^^^^^^^^^^
[00:58:15]   |
[00:58:15]   = note: candidates:
[00:58:15]   = note: path: /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-658d35794c10b003.rlib
[00:58:15]   = note: crate name: libc
[00:58:15]   = note: path: /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-f32a17a3111b01aa.rlib
[00:58:15]   = note: crate name: libc
[00:58:15] 
[00:58:15] error[E0463]: can't find crate for `libc`
[00:58:15]  --> /checkout/src/rustc/dlmalloc_shim/../../dlmalloc/src/linux.rs:1:1
[00:58:15]   |
[00:58:15] 1 | extern crate libc;
[00:58:15]   | ^^^^^^^^^^^^^^^^^^ can't find crate
[00:58:15] 
[00:58:15] error: aborting due to 2 previous errors
[00:58:15] 
[00:58:15] error: Could not compile `dlmalloc`.
@alexcrichton

This comment has been minimized.

Show comment
Hide comment
@alexcrichton

alexcrichton Nov 11, 2017

Member

@ElvishJerricco "just use the WebAssembly port of LLD" is actually quite involved, so I wanted to get something working (aka this PR) before starting on a new venture to discover even more integration and/or LLD bugs. This is the final state, though, to use LLD to link WebAsembly (AFAIK)

Member

alexcrichton commented Nov 11, 2017

@ElvishJerricco "just use the WebAssembly port of LLD" is actually quite involved, so I wanted to get something working (aka this PR) before starting on a new venture to discover even more integration and/or LLD bugs. This is the final state, though, to use LLD to link WebAsembly (AFAIK)

@killercup

This comment has been minimized.

Show comment
Hide comment
@killercup

killercup Nov 11, 2017

Member

FYI: If someone wants to play with this branch and WASM in Node, I've pushed some of my experiments to this repo: https://github.com/killercup/wasm-experiments

Member

killercup commented Nov 11, 2017

FYI: If someone wants to play with this branch and WASM in Node, I've pushed some of my experiments to this repo: https://github.com/killercup/wasm-experiments

@bors

This comment has been minimized.

Show comment
Hide comment
@bors

bors Nov 12, 2017

Contributor

☔️ The latest upstream changes (presumably #45870) made this pull request unmergeable. Please resolve the merge conflicts.

Contributor

bors commented Nov 12, 2017

☔️ The latest upstream changes (presumably #45870) made this pull request unmergeable. Please resolve the merge conflicts.

@bors

This comment has been minimized.

Show comment
Hide comment
@bors

bors Nov 12, 2017

Contributor

☔️ The latest upstream changes (presumably #45902) made this pull request unmergeable. Please resolve the merge conflicts.

Contributor

bors commented Nov 12, 2017

☔️ The latest upstream changes (presumably #45902) made this pull request unmergeable. Please resolve the merge conflicts.

imports.env = {
// These are generated by LLVM itself for various intrinsic calls. Hopefully
// one day this is not necessary and something will automatically do this.
fmod: function(x, y) { return x % y; },

This comment has been minimized.

@RReverser

RReverser Nov 14, 2017

Contributor

Any reason not to implement these on Rust side? (then they would be "automatic" for any consumer)

@RReverser

RReverser Nov 14, 2017

Contributor

Any reason not to implement these on Rust side? (then they would be "automatic" for any consumer)

This comment has been minimized.

@est31

est31 Nov 14, 2017

Contributor

You mean manually emulating the operation as if they were soft floats? I'm pretty sure that this would be much much slower than calling from wasm to js, executing a native float op, and returning the result to the wasm executable. Also not sure how WASM compilation works but it seems like it is possible to inline that function when the JS runtime JIT compiles it...

@est31

est31 Nov 14, 2017

Contributor

You mean manually emulating the operation as if they were soft floats? I'm pretty sure that this would be much much slower than calling from wasm to js, executing a native float op, and returning the result to the wasm executable. Also not sure how WASM compilation works but it seems like it is possible to inline that function when the JS runtime JIT compiles it...

This comment has been minimized.

@sunfishcode

sunfishcode Nov 14, 2017

Contributor

It wouldn't need soft float; it would just be a call to fmod in libm.

@sunfishcode

sunfishcode Nov 14, 2017

Contributor

It wouldn't need soft float; it would just be a call to fmod in libm.

This comment has been minimized.

@RReverser

RReverser Nov 14, 2017

Contributor

to inline that function when the JS runtime JIT compiles it...

It's not because JS and WASM use distinct compilers and runtimes, so every call outside is actually somewhat expensive (just like calls between JS and C++ runtimes).

@RReverser

RReverser Nov 14, 2017

Contributor

to inline that function when the JS runtime JIT compiles it...

It's not because JS and WASM use distinct compilers and runtimes, so every call outside is actually somewhat expensive (just like calls between JS and C++ runtimes).

This comment has been minimized.

@est31

est31 Nov 14, 2017

Contributor

I've discussed this a bit with @nagisa , @rkruppe and @sunfishcode on IRC.

  • There is no (fast) hardware instruction for floating point modulo, so you'll have to come up with a software solution. However, you can use the hardware instructions for other floating point operations to enable this. This gives you a speedup over pure soft-float implementations. I guess due to this point my point that this code is good is moot (as there will be a non-trivial overhead all the time).
  • One can't use libm unless libc is available, as libm is part of libc. The target is named wasm32-unknown-unknown however, it has no libc!
  • @nagisa 's math.rs library is built for 100% soft floats, so it is not fast.... @japaric 's m library doesn't seem to include modulo.
  • As we don't expose libstd, only libcore, we don't need to support the full set of arithmetic operations

So it seems we'd have to implement the functions ourselves.

@est31

est31 Nov 14, 2017

Contributor

I've discussed this a bit with @nagisa , @rkruppe and @sunfishcode on IRC.

  • There is no (fast) hardware instruction for floating point modulo, so you'll have to come up with a software solution. However, you can use the hardware instructions for other floating point operations to enable this. This gives you a speedup over pure soft-float implementations. I guess due to this point my point that this code is good is moot (as there will be a non-trivial overhead all the time).
  • One can't use libm unless libc is available, as libm is part of libc. The target is named wasm32-unknown-unknown however, it has no libc!
  • @nagisa 's math.rs library is built for 100% soft floats, so it is not fast.... @japaric 's m library doesn't seem to include modulo.
  • As we don't expose libstd, only libcore, we don't need to support the full set of arithmetic operations

So it seems we'd have to implement the functions ourselves.

This comment has been minimized.

@RReverser

RReverser Nov 14, 2017

Contributor

math.rs library is built for 100% soft floats, so it is not fast

Might be still worth benchmarking both approaches. It might very well turn out to be still faster than calls to JS side just to modulo two numbers, and even if it's on par, it gives benefit of not forcing each consumer to include these "intrinsics" in their JS consumers.

@RReverser

RReverser Nov 14, 2017

Contributor

math.rs library is built for 100% soft floats, so it is not fast

Might be still worth benchmarking both approaches. It might very well turn out to be still faster than calls to JS side just to modulo two numbers, and even if it's on par, it gives benefit of not forcing each consumer to include these "intrinsics" in their JS consumers.

This comment has been minimized.

@alexcrichton

alexcrichton Nov 14, 2017

Member

Yeah I'm assuming that one day we can just natively depend on the JS implementation here without implementing our own. Right now though it just requires an opt-in from everyone using it :(

@alexcrichton

alexcrichton Nov 14, 2017

Member

Yeah I'm assuming that one day we can just natively depend on the JS implementation here without implementing our own. Right now though it just requires an opt-in from everyone using it :(

This comment has been minimized.

@RReverser

RReverser Nov 14, 2017

Contributor

Which is why I'm suggesting just including soft emulation of these functions on Rust side. Might be not too bad (and operators that are slow to emulate like fmod are not used too frequently anyway).

@RReverser

RReverser Nov 14, 2017

Contributor

Which is why I'm suggesting just including soft emulation of these functions on Rust side. Might be not too bad (and operators that are slow to emulate like fmod are not used too frequently anyway).

@bors

This comment has been minimized.

Show comment
Hide comment
@bors

bors Nov 14, 2017

Contributor

☔️ The latest upstream changes (presumably #45896) made this pull request unmergeable. Please resolve the merge conflicts.

Contributor

bors commented Nov 14, 2017

☔️ The latest upstream changes (presumably #45896) made this pull request unmergeable. Please resolve the merge conflicts.

exp2f: function(x) { return Math.pow(2, x); },
ldexp: function(x, y) { return x * Math.pow(2, y); },
ldexpf: function(x, y) { return x * Math.pow(2, y); },
log10: function(x) { return Math.log10(x); },

This comment has been minimized.

@RReverser

RReverser Nov 14, 2017

Contributor

Btw - no need to wrap these, you can just do log10: Math.log10 since context doesn't matter for static Math functions.

@RReverser

RReverser Nov 14, 2017

Contributor

Btw - no need to wrap these, you can just do log10: Math.log10 since context doesn't matter for static Math functions.

This comment has been minimized.

@alexcrichton

alexcrichton Nov 14, 2017

Member

Nice! I'll try to get that done in a follow-up commit.

@alexcrichton

alexcrichton Nov 14, 2017

Member

Nice! I'll try to get that done in a follow-up commit.

@alexcrichton

This comment has been minimized.

Show comment
Hide comment
@alexcrichton

alexcrichton Nov 14, 2017

Member

@bors: r=aturon

Member

alexcrichton commented Nov 14, 2017

@bors: r=aturon

@bors

This comment has been minimized.

Show comment
Hide comment
@bors

bors Nov 14, 2017

Contributor

📌 Commit f288452 has been approved by aturon

Contributor

bors commented Nov 14, 2017

📌 Commit f288452 has been approved by aturon

@bors

This comment has been minimized.

Show comment
Hide comment
@bors

bors Nov 15, 2017

Contributor

⌛️ Testing commit f288452 with merge 462f540...

Contributor

bors commented Nov 15, 2017

⌛️ Testing commit f288452 with merge 462f540...

bors added a commit that referenced this pull request Nov 15, 2017

Auto merge of #45905 - alexcrichton:add-wasm-target, r=aturon
std: Add a new wasm32-unknown-unknown target

This commit adds a new target to the compiler: wasm32-unknown-unknown. This target is a reimagining of what it looks like to generate WebAssembly code from Rust. Instead of using Emscripten which can bring with it a weighty runtime this instead is a target which uses only the LLVM backend for WebAssembly and a "custom linker" for now which will hopefully one day be direct calls to lld.

Notable features of this target include:

* There is zero runtime footprint. The target assumes nothing exists other than the wasm32 instruction set.
* There is zero toolchain footprint beyond adding the target. No custom linker is needed, rustc contains everything.
* Very small wasm modules can be generated directly from Rust code using this target.
* Most of the standard library is stubbed out to return an error, but anything related to allocation works (aka `HashMap`, `Vec`, etc).
* Naturally, any `#[no_std]` crate should be 100% compatible with this new target.

This target is currently somewhat janky due to how linking works. The "linking" is currently unconditional whole program LTO (aka LLVM is being used as a linker). Naturally that means compiling programs is pretty slow! Eventually though this target should have a linker.

This target is also intended to be quite experimental. I'm hoping that this can act as a catalyst for further experimentation in Rust with WebAssembly. Breaking changes are very likely to land to this target, so it's not recommended to rely on it in any critical capacity yet. We'll let you know when it's "production ready".

### Building yourself

First you'll need to configure the build of LLVM and enable this target

```
$ ./configure --target=wasm32-unknown-unknown --set llvm.experimental-targets=WebAssembly
```

Next you'll want to remove any previously compiled LLVM as it needs to be rebuilt with WebAssembly support. You can do that with:

```
$ rm -rf build
```

And then you're good to go! A `./x.py build` should give you a rustc with the appropriate libstd target.

### Test support

Currently testing-wise this target is looking pretty good but isn't complete. I've got almost the entire `run-pass` test suite working with this target (lots of tests ignored, but many passing as well). The `core` test suite is [still getting LLVM bugs fixed](https://reviews.llvm.org/D39866) to get that working and will take some time. Relatively simple programs all seem to work though!

In general I've only tested this with a local fork that makes use of LLVM 5 rather than our current LLVM 4 on master. The LLVM 4 WebAssembly backend AFAIK isn't broken per se but is likely missing bug fixes available on LLVM 5. I'm hoping though that we can decouple the LLVM 5 upgrade and adding this wasm target!

### But the modules generated are huge!

It's worth nothing that you may not immediately see the "smallest possible wasm module" for the input you feed to rustc. For various reasons it's very difficult to get rid of the final "bloat" in vanilla rustc (again, a real linker should fix all this). For now what you'll have to do is:

    cargo install --git https://github.com/alexcrichton/wasm-gc
    wasm-gc foo.wasm bar.wasm

And then `bar.wasm` should be the smallest we can get it!

---

In any case for now I'd love feedback on this, particularly on the various integration points if you've got better ideas of how to approach them!
@bors

This comment has been minimized.

Show comment
Hide comment
@bors

bors Nov 15, 2017

Contributor

💔 Test failed - status-travis

Contributor

bors commented Nov 15, 2017

💔 Test failed - status-travis

@shepmaster

This comment has been minimized.

Show comment
Hide comment
@shepmaster

shepmaster Nov 15, 2017

Member
[00:02:49] thread 'main' panicked at 'failed to copy `/checkout/src/binaryen/test/emscripten/system/include/libc/bits` to `/checkout/obj/build/tmp/dist/rustc-1.23.0-dev-src/src/binaryen/test/emscripten/system/include/libc/bits`: the source path is not an existing regular file', /checkout/src/bootstrap/util.rs:44:8
Member

shepmaster commented Nov 15, 2017

[00:02:49] thread 'main' panicked at 'failed to copy `/checkout/src/binaryen/test/emscripten/system/include/libc/bits` to `/checkout/obj/build/tmp/dist/rustc-1.23.0-dev-src/src/binaryen/test/emscripten/system/include/libc/bits`: the source path is not an existing regular file', /checkout/src/bootstrap/util.rs:44:8
@alexcrichton

This comment has been minimized.

Show comment
Hide comment
@alexcrichton

alexcrichton Nov 15, 2017

Member

@bors: r=aturon

Member

alexcrichton commented Nov 15, 2017

@bors: r=aturon

@bors

This comment has been minimized.

Show comment
Hide comment
@bors

bors Nov 15, 2017

Contributor

📌 Commit 6f5c9b4 has been approved by aturon

Contributor

bors commented Nov 15, 2017

📌 Commit 6f5c9b4 has been approved by aturon

@bors

This comment has been minimized.

Show comment
Hide comment
@bors

bors Nov 15, 2017

Contributor

⌛️ Testing commit 6f5c9b4 with merge fe8684b...

Contributor

bors commented Nov 15, 2017

⌛️ Testing commit 6f5c9b4 with merge fe8684b...

bors added a commit that referenced this pull request Nov 15, 2017

Auto merge of #45905 - alexcrichton:add-wasm-target, r=aturon
std: Add a new wasm32-unknown-unknown target

This commit adds a new target to the compiler: wasm32-unknown-unknown. This target is a reimagining of what it looks like to generate WebAssembly code from Rust. Instead of using Emscripten which can bring with it a weighty runtime this instead is a target which uses only the LLVM backend for WebAssembly and a "custom linker" for now which will hopefully one day be direct calls to lld.

Notable features of this target include:

* There is zero runtime footprint. The target assumes nothing exists other than the wasm32 instruction set.
* There is zero toolchain footprint beyond adding the target. No custom linker is needed, rustc contains everything.
* Very small wasm modules can be generated directly from Rust code using this target.
* Most of the standard library is stubbed out to return an error, but anything related to allocation works (aka `HashMap`, `Vec`, etc).
* Naturally, any `#[no_std]` crate should be 100% compatible with this new target.

This target is currently somewhat janky due to how linking works. The "linking" is currently unconditional whole program LTO (aka LLVM is being used as a linker). Naturally that means compiling programs is pretty slow! Eventually though this target should have a linker.

This target is also intended to be quite experimental. I'm hoping that this can act as a catalyst for further experimentation in Rust with WebAssembly. Breaking changes are very likely to land to this target, so it's not recommended to rely on it in any critical capacity yet. We'll let you know when it's "production ready".

### Building yourself

First you'll need to configure the build of LLVM and enable this target

```
$ ./configure --target=wasm32-unknown-unknown --set llvm.experimental-targets=WebAssembly
```

Next you'll want to remove any previously compiled LLVM as it needs to be rebuilt with WebAssembly support. You can do that with:

```
$ rm -rf build
```

And then you're good to go! A `./x.py build` should give you a rustc with the appropriate libstd target.

### Test support

Currently testing-wise this target is looking pretty good but isn't complete. I've got almost the entire `run-pass` test suite working with this target (lots of tests ignored, but many passing as well). The `core` test suite is [still getting LLVM bugs fixed](https://reviews.llvm.org/D39866) to get that working and will take some time. Relatively simple programs all seem to work though!

In general I've only tested this with a local fork that makes use of LLVM 5 rather than our current LLVM 4 on master. The LLVM 4 WebAssembly backend AFAIK isn't broken per se but is likely missing bug fixes available on LLVM 5. I'm hoping though that we can decouple the LLVM 5 upgrade and adding this wasm target!

### But the modules generated are huge!

It's worth nothing that you may not immediately see the "smallest possible wasm module" for the input you feed to rustc. For various reasons it's very difficult to get rid of the final "bloat" in vanilla rustc (again, a real linker should fix all this). For now what you'll have to do is:

    cargo install --git https://github.com/alexcrichton/wasm-gc
    wasm-gc foo.wasm bar.wasm

And then `bar.wasm` should be the smallest we can get it!

---

In any case for now I'd love feedback on this, particularly on the various integration points if you've got better ideas of how to approach them!
@badboy

This comment has been minimized.

Show comment
Hide comment
@badboy

badboy Nov 20, 2017

Member

Wuhu! \o/

Member

badboy commented Nov 20, 2017

Wuhu! \o/

@steveklabnik

This comment has been minimized.

Show comment
Hide comment
@steveklabnik
Member

steveklabnik commented Nov 20, 2017

YAY!

@mgattozzi

This comment has been minimized.

Show comment
Hide comment
@mgattozzi

mgattozzi Nov 20, 2017

Contributor

This is a Very Good Thing™. Can't wait to see it expanded over time!

Contributor

mgattozzi commented Nov 20, 2017

This is a Very Good Thing™. Can't wait to see it expanded over time!

@alexcrichton

This comment has been minimized.

Show comment
Hide comment
@alexcrichton

alexcrichton Nov 20, 2017

Member

I've opened up #46115 as the next step which enables distribution of the wasm32 target

Member

alexcrichton commented Nov 20, 2017

I've opened up #46115 as the next step which enables distribution of the wasm32 target

let arg = process.argv[idx + 2];
let view = new Uint8Array(memory.buffer);
for (var i = 0; i < arg.length; i++) {
view[ptr + i] = arg.charCodeAt(i);

This comment has been minimized.

@RReverser

RReverser Nov 20, 2017

Contributor

Same here about strings, but I guess too late now that it's merged? @alexcrichton

@RReverser

RReverser Nov 20, 2017

Contributor

Same here about strings, but I guess too late now that it's merged? @alexcrichton

This comment has been minimized.

@alexcrichton

alexcrichton Nov 20, 2017

Member

@RReverser yes I will try to fix these when testing later. This doesn't matter for correctness and it's just internal debugging, it's not massively critical this is working.

@alexcrichton

alexcrichton Nov 20, 2017

Member

@RReverser yes I will try to fix these when testing later. This doesn't matter for correctness and it's just internal debugging, it's not massively critical this is working.

This comment has been minimized.

@RReverser

RReverser Nov 20, 2017

Contributor

Actually, even better could be to just have UTF-16 OsString for WASM target - I think it would make sense than Vec<u8> given that in JS string operations are all based on UTF-16 units.

@RReverser

RReverser Nov 20, 2017

Contributor

Actually, even better could be to just have UTF-16 OsString for WASM target - I think it would make sense than Vec<u8> given that in JS string operations are all based on UTF-16 units.

This comment has been minimized.

@alexcrichton

alexcrichton Nov 20, 2017

Member

Again though the intent here is to not go crazy, if we're dealing with utf-16 then let's just delete this code. I just used it for debugging and it shouldn't affect the standard library in any way. This isn't a node target, it's a wasm target and there's no strings in wasm yet in the abi

@alexcrichton

alexcrichton Nov 20, 2017

Member

Again though the intent here is to not go crazy, if we're dealing with utf-16 then let's just delete this code. I just used it for debugging and it shouldn't affect the standard library in any way. This isn't a node target, it's a wasm target and there's no strings in wasm yet in the abi

This comment has been minimized.

@RReverser

RReverser Nov 20, 2017

Contributor

I'm not sure wasm will ever have strings in the ABI when there is TextEncoder / TextDecoder, and for interaction with any JS it seems only fair to have OsString to be UTF-16 like it is on some other platforms.

@RReverser

RReverser Nov 20, 2017

Contributor

I'm not sure wasm will ever have strings in the ABI when there is TextEncoder / TextDecoder, and for interaction with any JS it seems only fair to have OsString to be UTF-16 like it is on some other platforms.

@est31

This comment has been minimized.

Show comment
Hide comment
@est31

est31 Nov 20, 2017

Contributor

@RReverser now that it is merged you could do a PR yourself if @alexcrichton doesn't want to :)

Contributor

est31 commented Nov 20, 2017

@RReverser now that it is merged you could do a PR yourself if @alexcrichton doesn't want to :)

@RReverser

This comment has been minimized.

Show comment
Hide comment
@RReverser

RReverser Nov 20, 2017

Contributor

Not to sound ungrateful - this PR is a great change, but I feel there is some connection between

There is zero runtime footprint.

and

Most of the standard library is stubbed out to return an error

😄. Judging by Math examples, this will require quite a bit of runtime on JS side, much like Emscripten's libraries, to make std work.

Contributor

RReverser commented Nov 20, 2017

Not to sound ungrateful - this PR is a great change, but I feel there is some connection between

There is zero runtime footprint.

and

Most of the standard library is stubbed out to return an error

😄. Judging by Math examples, this will require quite a bit of runtime on JS side, much like Emscripten's libraries, to make std work.

@badboy

This comment has been minimized.

Show comment
Hide comment
@badboy

badboy Nov 20, 2017

Member

😄. Judging by Math examples, this will require quite a bit of runtime on JS side, much like Emscripten's libraries, to make std work.

Luckily enough, a lot of things can still work with the current state

Member

badboy commented Nov 20, 2017

😄. Judging by Math examples, this will require quite a bit of runtime on JS side, much like Emscripten's libraries, to make std work.

Luckily enough, a lot of things can still work with the current state

@RReverser

This comment has been minimized.

Show comment
Hide comment
@RReverser

RReverser Nov 20, 2017

Contributor

@badboy Mostly just math works, yeah, but for real app that interacts with filesystem etc. you will need JS runtime.

Contributor

RReverser commented Nov 20, 2017

@badboy Mostly just math works, yeah, but for real app that interacts with filesystem etc. you will need JS runtime.

@pepyakin

This comment has been minimized.

Show comment
Hide comment
@pepyakin

pepyakin Nov 20, 2017

Contributor

Just passed by and wanted to note that WASM (despite it's name) is not only for the Web. By it's nature WASM, have no ABI and have no notion of filesystem and networking.

So this statement

Most of the standard library is stubbed out to return an error

have a lot of sense for me. For all other things there is Emscripten : )

Contributor

pepyakin commented Nov 20, 2017

Just passed by and wanted to note that WASM (despite it's name) is not only for the Web. By it's nature WASM, have no ABI and have no notion of filesystem and networking.

So this statement

Most of the standard library is stubbed out to return an error

have a lot of sense for me. For all other things there is Emscripten : )

@RReverser

This comment has been minimized.

Show comment
Hide comment
@RReverser

RReverser Nov 20, 2017

Contributor

By it's nature WASM, have no ABI and have no notion of filesystem and networking.

Heh, so does x86, it doesn't mean they're not needed for apps :)

Contributor

RReverser commented Nov 20, 2017

By it's nature WASM, have no ABI and have no notion of filesystem and networking.

Heh, so does x86, it doesn't mean they're not needed for apps :)

@pepyakin

This comment has been minimized.

Show comment
Hide comment
@pepyakin

pepyakin Nov 20, 2017

Contributor

Yeah, Wasm (as x86) is just an ISA. fs and stuff like that is just a different level of abstraction.

For me wasm32-unknown-unknown seems more like a bare metal environment (like Arm MCU). “apps” for mcus doesnt require (and usually doesnt have a notion of) fs and networking. So is wasm

Also, as I mentioned before, you can use Rust with wasm32-unknown-emscripten triple with LLVM wasm backend (you just have to build it yourself). It already provides all these (and many other) abstractions!

Contributor

pepyakin commented Nov 20, 2017

Yeah, Wasm (as x86) is just an ISA. fs and stuff like that is just a different level of abstraction.

For me wasm32-unknown-unknown seems more like a bare metal environment (like Arm MCU). “apps” for mcus doesnt require (and usually doesnt have a notion of) fs and networking. So is wasm

Also, as I mentioned before, you can use Rust with wasm32-unknown-emscripten triple with LLVM wasm backend (you just have to build it yourself). It already provides all these (and many other) abstractions!

@RReverser

This comment has been minimized.

Show comment
Hide comment
@RReverser

RReverser Nov 20, 2017

Contributor

Please, again, I'm not trying to diminish the importance of this addition, no need to try and prove to me how awesome it is 😄 I'm just saying that applicability will be pretty limited until most of standard library gets bindings to JS side.

Contributor

RReverser commented Nov 20, 2017

Please, again, I'm not trying to diminish the importance of this addition, no need to try and prove to me how awesome it is 😄 I'm just saying that applicability will be pretty limited until most of standard library gets bindings to JS side.

@badboy

This comment has been minimized.

Show comment
Hide comment
@badboy

badboy Nov 20, 2017

Member

Let's work together and provide the missing pieces then (either as JavaScript shims or Rust-side implementations of functionality). I don't know how easy it will be to extract this stuff from Emscripten.

Anyway, this is not the place to discuss it.

Member

badboy commented Nov 20, 2017

Let's work together and provide the missing pieces then (either as JavaScript shims or Rust-side implementations of functionality). I don't know how easy it will be to extract this stuff from Emscripten.

Anyway, this is not the place to discuss it.

@RReverser

This comment has been minimized.

Show comment
Hide comment
@RReverser

RReverser Nov 20, 2017

Contributor

Anyway, this is not the place to discuss it.

It seemed like the best place (as it's related to the context of PR), but if you have other suggestions, sure.

Contributor

RReverser commented Nov 20, 2017

Anyway, this is not the place to discuss it.

It seemed like the best place (as it's related to the context of PR), but if you have other suggestions, sure.

@pepyakin

This comment has been minimized.

Show comment
Hide comment
@pepyakin

pepyakin Nov 20, 2017

Contributor

My point is that many things from the standart library shouldn't get any bindings to host environment. For example, what means to spawn a process in wasm? What character should be used as a path seperator? How std::net should be implemented?
For me, these questions just doesn't make sense.
Also, I want to emphasize, that Web/JS is a important environment but it is not the only possible one!

Please, again, I'm not trying to diminish the importance of this addition, no need to try and prove to me how awesome it is

I didn't even think of that!
(btw sorry for my english)

Contributor

pepyakin commented Nov 20, 2017

My point is that many things from the standart library shouldn't get any bindings to host environment. For example, what means to spawn a process in wasm? What character should be used as a path seperator? How std::net should be implemented?
For me, these questions just doesn't make sense.
Also, I want to emphasize, that Web/JS is a important environment but it is not the only possible one!

Please, again, I'm not trying to diminish the importance of this addition, no need to try and prove to me how awesome it is

I didn't even think of that!
(btw sorry for my english)

@RReverser

This comment has been minimized.

Show comment
Hide comment
@RReverser

RReverser Nov 20, 2017

Contributor

@pepyakin At least for Node.js side spawning process, net etc. are pretty easily implemented. Character for path can be taken from host system and so on. I think it makes sense to experiment with reusing Emscripten libraries for this as @badboy suggested.

Contributor

RReverser commented Nov 20, 2017

@pepyakin At least for Node.js side spawning process, net etc. are pretty easily implemented. Character for path can be taken from host system and so on. I think it makes sense to experiment with reusing Emscripten libraries for this as @badboy suggested.

@RReverser

This comment has been minimized.

Show comment
Hide comment
@RReverser

RReverser Nov 20, 2017

Contributor

Also, I want to emphasize, that Web/JS is a important environment but it is not the only possible one!

Well other environments might provide own implementations of these system bindings if they need to.

Contributor

RReverser commented Nov 20, 2017

Also, I want to emphasize, that Web/JS is a important environment but it is not the only possible one!

Well other environments might provide own implementations of these system bindings if they need to.

@est31

This comment has been minimized.

Show comment
Hide comment
@est31

est31 Nov 21, 2017

Contributor

Anyway, this is not the place to discuss it.

A better place would be #44006 . It is a bit underused sadly :/

Contributor

est31 commented Nov 21, 2017

Anyway, this is not the place to discuss it.

A better place would be #44006 . It is a bit underused sadly :/

@lilianmoraru

This comment has been minimized.

Show comment
Hide comment
@lilianmoraru

lilianmoraru Nov 21, 2017

@alexcrichton Now, the linker has some support for wasm: https://reviews.llvm.org/rL318539

lilianmoraru commented Nov 21, 2017

@alexcrichton Now, the linker has some support for wasm: https://reviews.llvm.org/rL318539

@pepyakin

This comment has been minimized.

Show comment
Hide comment
@pepyakin

pepyakin Nov 21, 2017

Contributor

Thank you @est31 ! I've moved discussion there

Contributor

pepyakin commented Nov 21, 2017

Thank you @est31 ! I've moved discussion there

OpenGG added a commit to OpenGG/docker-rust-wasm that referenced this pull request Nov 21, 2017

@vmx

This comment has been minimized.

Show comment
Hide comment
@vmx

vmx Nov 25, 2017

Contributor

@alexcrichton You mention LLVM 5. Where is your branch and how do I use it? I'm playing with the wasm target but get a segfault during compilation. Before I dig deeper I'd like to try a more recent LLVM as it might be fixed already.

Contributor

vmx commented Nov 25, 2017

@alexcrichton You mention LLVM 5. Where is your branch and how do I use it? I'm playing with the wasm target but get a segfault during compilation. Before I dig deeper I'd like to try a more recent LLVM as it might be fixed already.

@Dessix

This comment has been minimized.

Show comment
Hide comment
@Dessix

Dessix Nov 26, 2017

@RReverser the option of not having the extra libraries and runtimes bolted on enables use of this for embedded systems where including additional support code is unnecessary or even undesirable. Ensuring that remains an option would be an important consideration for at least these scenarios.

Dessix commented Nov 26, 2017

@RReverser the option of not having the extra libraries and runtimes bolted on enables use of this for embedded systems where including additional support code is unnecessary or even undesirable. Ensuring that remains an option would be an important consideration for at least these scenarios.

@RReverser

This comment has been minimized.

Show comment
Hide comment
@RReverser

RReverser Nov 26, 2017

Contributor

@Dessix One absolutely doesn't contradict another, you can always just not include runtime and not use anything that requires it. But anyway, this discussion was moved to another issue, please see above.

Contributor

RReverser commented Nov 26, 2017

@Dessix One absolutely doesn't contradict another, you can always just not include runtime and not use anything that requires it. But anyway, this discussion was moved to another issue, please see above.

@alexcrichton

This comment has been minimized.

Show comment
Hide comment
@alexcrichton

alexcrichton Nov 27, 2017

Member

I've now also opened a PR to start running tests on CI over at #46291

Member

alexcrichton commented Nov 27, 2017

I've now also opened a PR to start running tests on CI over at #46291

@jethrogb

This comment has been minimized.

Show comment
Hide comment
@jethrogb

jethrogb Nov 28, 2017

Contributor

Since this is basically the same as #37133, I wouldn't have expected this to get merged in this state in light of the comments there and in https://internals.rust-lang.org/t/refactoring-std-for-ultimate-portability/4301. Will you reconsider #37133 for merging if I rebase that?

Contributor

jethrogb commented Nov 28, 2017

Since this is basically the same as #37133, I wouldn't have expected this to get merged in this state in light of the comments there and in https://internals.rust-lang.org/t/refactoring-std-for-ultimate-portability/4301. Will you reconsider #37133 for merging if I rebase that?

@aturon

This comment has been minimized.

Show comment
Hide comment
@aturon

aturon Nov 29, 2017

Member

@jethrogb it's definitely worth revisiting, yes. Note that the wasm target is a somewhat special case, since it exists at the liballoc level and will eventually include some of the libstd layer as well (e.g. threads). It's also a specific (highly desired) platform, so the motivation was pretty different.

For #37133, is the motivation primarily to have a kind of "template" for tackling other, similar cases?

re: the PAL, AFAIK not much has happened there, and I think the whole story should be rebooted. In particular, we should take into account (and implement...) the portability lint, which in the long term may allow us to do away with the core/std layering altogether.

In any case, before rebasing, I'd suggest at least an internals thread laying out the motivation and attempting to reach some consensus there. You could ping me, @alexcrichton and @sfackler on the thread to ensure visibility.

Member

aturon commented Nov 29, 2017

@jethrogb it's definitely worth revisiting, yes. Note that the wasm target is a somewhat special case, since it exists at the liballoc level and will eventually include some of the libstd layer as well (e.g. threads). It's also a specific (highly desired) platform, so the motivation was pretty different.

For #37133, is the motivation primarily to have a kind of "template" for tackling other, similar cases?

re: the PAL, AFAIK not much has happened there, and I think the whole story should be rebooted. In particular, we should take into account (and implement...) the portability lint, which in the long term may allow us to do away with the core/std layering altogether.

In any case, before rebasing, I'd suggest at least an internals thread laying out the motivation and attempting to reach some consensus there. You could ping me, @alexcrichton and @sfackler on the thread to ensure visibility.

pub fn cosh(n: f64) -> f64;
#[link_name = "Math_expm1"]
pub fn expm1(n: f64) -> f64;
pub fn fdim(a: f64, b: f64) -> f64;

This comment has been minimized.

@aochagavia

aochagavia Dec 3, 2017

Contributor

Looks like you forgot to add a #[link_name = "Math_fdim"] attribute

@aochagavia

aochagavia Dec 3, 2017

Contributor

Looks like you forgot to add a #[link_name = "Math_fdim"] attribute

This comment has been minimized.

@alexcrichton

alexcrichton Dec 3, 2017

Member

Unfortunately Math.fdim isn't actually a function, and hopefully none of these will be necessary in the future regardless, it's mostly a bug that they're all here in the first place.

@alexcrichton

alexcrichton Dec 3, 2017

Member

Unfortunately Math.fdim isn't actually a function, and hopefully none of these will be necessary in the future regardless, it's mostly a bug that they're all here in the first place.

arielb1 added a commit to arielb1/rust that referenced this pull request Dec 21, 2017

@its-pointless its-pointless referenced this pull request Mar 9, 2018

Closed

Request: Rust extras #28

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment