Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

std: Add a new wasm32-unknown-unknown target #45905

Merged
merged 1 commit into from Nov 20, 2017

Conversation

@alexcrichton
Copy link
Member

alexcrichton commented Nov 10, 2017

This commit adds a new target to the compiler: wasm32-unknown-unknown. This target is a reimagining of what it looks like to generate WebAssembly code from Rust. Instead of using Emscripten which can bring with it a weighty runtime this instead is a target which uses only the LLVM backend for WebAssembly and a "custom linker" for now which will hopefully one day be direct calls to lld.

Notable features of this target include:

  • There is zero runtime footprint. The target assumes nothing exists other than the wasm32 instruction set.
  • There is zero toolchain footprint beyond adding the target. No custom linker is needed, rustc contains everything.
  • Very small wasm modules can be generated directly from Rust code using this target.
  • Most of the standard library is stubbed out to return an error, but anything related to allocation works (aka HashMap, Vec, etc).
  • Naturally, any #[no_std] crate should be 100% compatible with this new target.

This target is currently somewhat janky due to how linking works. The "linking" is currently unconditional whole program LTO (aka LLVM is being used as a linker). Naturally that means compiling programs is pretty slow! Eventually though this target should have a linker.

This target is also intended to be quite experimental. I'm hoping that this can act as a catalyst for further experimentation in Rust with WebAssembly. Breaking changes are very likely to land to this target, so it's not recommended to rely on it in any critical capacity yet. We'll let you know when it's "production ready".

Building yourself

First you'll need to configure the build of LLVM and enable this target

$ ./configure --target=wasm32-unknown-unknown --set llvm.experimental-targets=WebAssembly

Next you'll want to remove any previously compiled LLVM as it needs to be rebuilt with WebAssembly support. You can do that with:

$ rm -rf build

And then you're good to go! A ./x.py build should give you a rustc with the appropriate libstd target.

Test support

Currently testing-wise this target is looking pretty good but isn't complete. I've got almost the entire run-pass test suite working with this target (lots of tests ignored, but many passing as well). The core test suite is still getting LLVM bugs fixed to get that working and will take some time. Relatively simple programs all seem to work though!

In general I've only tested this with a local fork that makes use of LLVM 5 rather than our current LLVM 4 on master. The LLVM 4 WebAssembly backend AFAIK isn't broken per se but is likely missing bug fixes available on LLVM 5. I'm hoping though that we can decouple the LLVM 5 upgrade and adding this wasm target!

But the modules generated are huge!

It's worth nothing that you may not immediately see the "smallest possible wasm module" for the input you feed to rustc. For various reasons it's very difficult to get rid of the final "bloat" in vanilla rustc (again, a real linker should fix all this). For now what you'll have to do is:

cargo install --git https://github.com/alexcrichton/wasm-gc
wasm-gc foo.wasm bar.wasm

And then bar.wasm should be the smallest we can get it!


In any case for now I'd love feedback on this, particularly on the various integration points if you've got better ideas of how to approach them!

@alexcrichton
Copy link
Member Author

alexcrichton commented Nov 10, 2017

I'd also like to doubly and triply emphasize how this target is a work in progress. The precise semantics of how it works and how everything gets hooked up will likely change over time, but I feel that it's also beneficial if we can get more testing of this backend and more testing of the integration, and hopefully start seeing some neat projects leveraging these benefits!

@alexcrichton alexcrichton force-pushed the alexcrichton:add-wasm-target branch from df2c76e to 7552aa5 Nov 10, 2017
@tlively
Copy link
Contributor

tlively commented Nov 10, 2017

@alexcrichton I just want to say how exciting it is to see so much progress toward making wasm work well with Rust :) Thanks for doing all of this!

@shepmaster
Copy link
Member

shepmaster commented Nov 10, 2017

Let's pick a reviewer...

r? @eddyb

@shepmaster
Copy link
Member

shepmaster commented Nov 10, 2017

@alexcrichton you've some tidy failures:

[00:03:55] tidy error: /checkout/src/etc/wasm32-shim.js: incorrect license
[00:03:55] Expected a gate test for the feature 'wasm_import_memory'.
[00:03:55] Hint: create a file named 'feature-gate-wasm_import_memory.rs' in the compile-fail
[00:03:55]       test suite, with its failures due to missing usage of
[00:03:55]       #![feature(wasm_import_memory)].
[00:03:55] Hint: If you already have such a test and don't want to rename it,
[00:03:55]       you can also add a // gate-test-wasm_import_memory line to the test file.
[00:03:55] tidy error: Found 1 features without a gate test.
[00:03:56] some tidy checks failed
@eddyb
Copy link
Member

eddyb commented Nov 10, 2017

LGTM but this is a pretty large change. cc @rust-lang/compiler @rust-lang/libs

@@ -39,8 +38,7 @@ unsafe impl<'a> Alloc for &'a A {
static GLOBAL: A = A;

fn main() {
env::set_var("FOO", "bar");
drop(env::var("FOO"));
println!("hello!");

This comment has been minimized.

@nagisa

nagisa Nov 10, 2017 Contributor

Stray print?

This comment has been minimized.

@alexcrichton

alexcrichton Nov 10, 2017 Author Member

Nah this is just intended to force an allocation somewhere that LLVM can't optimize away

//!
//! Right now we don't support this, so this is just stubs

#![allow(private_no_mangle_fns)]

This comment has been minimized.

@NikVolf

NikVolf Nov 10, 2017 Member

is it really needed here?

@SimonSapin
Copy link
Contributor

SimonSapin commented Nov 10, 2017

anything related to allocation works (aka HashMap, Vec, etc).

Out of curiosity, how does this work? Is there an allocator runtime?

@pepyakin
Copy link
Contributor

pepyakin commented Nov 10, 2017

@jasondavies
Copy link

jasondavies commented Nov 10, 2017

This is great!

In general I've only tested this with a local fork that makes use of LLVM 5 rather than our current LLVM 4 on master.

If I simply clone this branch, run ./configure --target=wasm32-unknown-unknown --set llvm.experimental-targets=WebAssembly and ./x.py build, after much waiting I get an error:

Intrinsic has incorrect return type!
i32 (i32)* @llvm.wasm.grow.memory.i32
LLVM ERROR: Broken function found, compilation aborted!
error: Could not compile `dlmalloc`.

Does this mean I should be testing with LLVM 5, and if so, can you provide some simple instructions on how to do so?

Edit: I managed to build successfully using LLVM 5, using the --llvm-root option when calling ./configure.

.gitmodules Outdated
url = https://github.com/alexcrichton/dlmalloc-rs
[submodule "src/binaryen"]
path = src/binaryen
url = https://github.com/WebAssembly/binaryen

This comment has been minimized.

@konstin

konstin Nov 10, 2017

There's a small inconsistency here: All the other crates use https://github.com/owner/repo.git instead of https://github.com/owner/repo

@alexcrichton alexcrichton force-pushed the alexcrichton:add-wasm-target branch from 7552aa5 to dde8f95 Nov 10, 2017
@alexcrichton
Copy link
Member Author

alexcrichton commented Nov 10, 2017

@SimonSapin indeed there is! Wasm actually has memory allocation instructions which the allocator runtime leverages to implement a runtime allocator.

@alexcrichton alexcrichton force-pushed the alexcrichton:add-wasm-target branch from dde8f95 to f483780 Nov 10, 2017
@alexcrichton
Copy link
Member Author

alexcrichton commented Nov 10, 2017

@jasondavies oops sorry! Was pinned to the wrong commit of dlmalloc-rs, I believe it should work now.

@alexcrichton alexcrichton force-pushed the alexcrichton:add-wasm-target branch from f41e64f to f483780 Nov 10, 2017
@ElvishJerricco
Copy link

ElvishJerricco commented Nov 11, 2017

Does https://github.com/WebAssembly/lld/tree/wasm have to be merged upstream before you can just use the WebAssembly port of LLD?

@shepmaster
Copy link
Member

shepmaster commented Nov 11, 2017

New tidy errors:

Testing libstd stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[00:58:15]    Compiling libc v0.2.32
[00:58:15]    Compiling core v0.0.0 (file:///checkout/src/libcore)
[00:58:15]    Compiling std_unicode v0.0.0 (file:///checkout/src/libstd_unicode)
[00:58:15]    Compiling dlmalloc v0.0.0 (file:///checkout/src/rustc/dlmalloc_shim)
[00:58:15] error[E0464]: multiple matching crates for `libc`
[00:58:15]  --> /checkout/src/rustc/dlmalloc_shim/../../dlmalloc/src/linux.rs:1:1
[00:58:15]   |
[00:58:15] 1 | extern crate libc;
[00:58:15]   | ^^^^^^^^^^^^^^^^^^
[00:58:15]   |
[00:58:15]   = note: candidates:
[00:58:15]   = note: path: /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-658d35794c10b003.rlib
[00:58:15]   = note: crate name: libc
[00:58:15]   = note: path: /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-f32a17a3111b01aa.rlib
[00:58:15]   = note: crate name: libc
[00:58:15] 
[00:58:15] error[E0463]: can't find crate for `libc`
[00:58:15]  --> /checkout/src/rustc/dlmalloc_shim/../../dlmalloc/src/linux.rs:1:1
[00:58:15]   |
[00:58:15] 1 | extern crate libc;
[00:58:15]   | ^^^^^^^^^^^^^^^^^^ can't find crate
[00:58:15] 
[00:58:15] error: aborting due to 2 previous errors
[00:58:15] 
[00:58:15] error: Could not compile `dlmalloc`.
@alexcrichton alexcrichton force-pushed the alexcrichton:add-wasm-target branch from f483780 to 9bdb989 Nov 11, 2017
@alexcrichton
Copy link
Member Author

alexcrichton commented Nov 11, 2017

@ElvishJerricco "just use the WebAssembly port of LLD" is actually quite involved, so I wanted to get something working (aka this PR) before starting on a new venture to discover even more integration and/or LLD bugs. This is the final state, though, to use LLD to link WebAsembly (AFAIK)

@killercup
Copy link
Member

killercup commented Nov 11, 2017

FYI: If someone wants to play with this branch and WASM in Node, I've pushed some of my experiments to this repo: https://github.com/killercup/wasm-experiments

@bors
Copy link
Contributor

bors commented Nov 12, 2017

The latest upstream changes (presumably #45870) made this pull request unmergeable. Please resolve the merge conflicts.

@alexcrichton alexcrichton force-pushed the alexcrichton:add-wasm-target branch from 9bdb989 to 3a89d09 Nov 12, 2017
@pepyakin
Copy link
Contributor

pepyakin commented Nov 20, 2017

Just passed by and wanted to note that WASM (despite it's name) is not only for the Web. By it's nature WASM, have no ABI and have no notion of filesystem and networking.

So this statement

Most of the standard library is stubbed out to return an error

have a lot of sense for me. For all other things there is Emscripten : )

@RReverser
Copy link
Contributor

RReverser commented Nov 20, 2017

By it's nature WASM, have no ABI and have no notion of filesystem and networking.

Heh, so does x86, it doesn't mean they're not needed for apps :)

@pepyakin
Copy link
Contributor

pepyakin commented Nov 20, 2017

Yeah, Wasm (as x86) is just an ISA. fs and stuff like that is just a different level of abstraction.

For me wasm32-unknown-unknown seems more like a bare metal environment (like Arm MCU). “apps” for mcus doesnt require (and usually doesnt have a notion of) fs and networking. So is wasm

Also, as I mentioned before, you can use Rust with wasm32-unknown-emscripten triple with LLVM wasm backend (you just have to build it yourself). It already provides all these (and many other) abstractions!

@RReverser
Copy link
Contributor

RReverser commented Nov 20, 2017

Please, again, I'm not trying to diminish the importance of this addition, no need to try and prove to me how awesome it is 😄 I'm just saying that applicability will be pretty limited until most of standard library gets bindings to JS side.

@badboy
Copy link
Member

badboy commented Nov 20, 2017

Let's work together and provide the missing pieces then (either as JavaScript shims or Rust-side implementations of functionality). I don't know how easy it will be to extract this stuff from Emscripten.

Anyway, this is not the place to discuss it.

@RReverser
Copy link
Contributor

RReverser commented Nov 20, 2017

Anyway, this is not the place to discuss it.

It seemed like the best place (as it's related to the context of PR), but if you have other suggestions, sure.

@pepyakin
Copy link
Contributor

pepyakin commented Nov 20, 2017

My point is that many things from the standart library shouldn't get any bindings to host environment. For example, what means to spawn a process in wasm? What character should be used as a path seperator? How std::net should be implemented?
For me, these questions just doesn't make sense.
Also, I want to emphasize, that Web/JS is a important environment but it is not the only possible one!

Please, again, I'm not trying to diminish the importance of this addition, no need to try and prove to me how awesome it is

I didn't even think of that!
(btw sorry for my english)

@RReverser
Copy link
Contributor

RReverser commented Nov 20, 2017

@pepyakin At least for Node.js side spawning process, net etc. are pretty easily implemented. Character for path can be taken from host system and so on. I think it makes sense to experiment with reusing Emscripten libraries for this as @badboy suggested.

@RReverser
Copy link
Contributor

RReverser commented Nov 20, 2017

Also, I want to emphasize, that Web/JS is a important environment but it is not the only possible one!

Well other environments might provide own implementations of these system bindings if they need to.

@est31
Copy link
Contributor

est31 commented Nov 21, 2017

Anyway, this is not the place to discuss it.

A better place would be #44006 . It is a bit underused sadly :/

@lilianmoraru
Copy link

lilianmoraru commented Nov 21, 2017

@alexcrichton Now, the linker has some support for wasm: https://reviews.llvm.org/rL318539

@pepyakin
Copy link
Contributor

pepyakin commented Nov 21, 2017

Thank you @est31 ! I've moved discussion there

OpenGG added a commit to OpenGG/docker-rust-wasm that referenced this pull request Nov 21, 2017
@vmx
Copy link
Contributor

vmx commented Nov 25, 2017

@alexcrichton You mention LLVM 5. Where is your branch and how do I use it? I'm playing with the wasm target but get a segfault during compilation. Before I dig deeper I'd like to try a more recent LLVM as it might be fixed already.

@Dessix
Copy link

Dessix commented Nov 26, 2017

@RReverser the option of not having the extra libraries and runtimes bolted on enables use of this for embedded systems where including additional support code is unnecessary or even undesirable. Ensuring that remains an option would be an important consideration for at least these scenarios.

@RReverser
Copy link
Contributor

RReverser commented Nov 26, 2017

@Dessix One absolutely doesn't contradict another, you can always just not include runtime and not use anything that requires it. But anyway, this discussion was moved to another issue, please see above.

@alexcrichton
Copy link
Member Author

alexcrichton commented Nov 27, 2017

I've now also opened a PR to start running tests on CI over at #46291

@jethrogb
Copy link
Contributor

jethrogb commented Nov 28, 2017

Since this is basically the same as #37133, I wouldn't have expected this to get merged in this state in light of the comments there and in https://internals.rust-lang.org/t/refactoring-std-for-ultimate-portability/4301. Will you reconsider #37133 for merging if I rebase that?

@aturon
Copy link
Member

aturon commented Nov 29, 2017

@jethrogb it's definitely worth revisiting, yes. Note that the wasm target is a somewhat special case, since it exists at the liballoc level and will eventually include some of the libstd layer as well (e.g. threads). It's also a specific (highly desired) platform, so the motivation was pretty different.

For #37133, is the motivation primarily to have a kind of "template" for tackling other, similar cases?

re: the PAL, AFAIK not much has happened there, and I think the whole story should be rebooted. In particular, we should take into account (and implement...) the portability lint, which in the long term may allow us to do away with the core/std layering altogether.

In any case, before rebasing, I'd suggest at least an internals thread laying out the motivation and attempting to reach some consensus there. You could ping me, @alexcrichton and @sfackler on the thread to ensure visibility.

pub fn cosh(n: f64) -> f64;
#[link_name = "Math_expm1"]
pub fn expm1(n: f64) -> f64;
pub fn fdim(a: f64, b: f64) -> f64;

This comment has been minimized.

@aochagavia

aochagavia Dec 3, 2017 Contributor

Looks like you forgot to add a #[link_name = "Math_fdim"] attribute

This comment has been minimized.

@alexcrichton

alexcrichton Dec 3, 2017 Author Member

Unfortunately Math.fdim isn't actually a function, and hopefully none of these will be necessary in the future regardless, it's mostly a bug that they're all here in the first place.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

You can’t perform that action at this time.