Make wasm32 buildbot test LLVM backend

[tlively]

This adds the experimental targets option to configure so it can be used
by the builders and changes the wasm32 Dockerfile accordingly. Instead
of using LLVM from the emsdk, the builder's emscripten tools now uses
the Rust in-tree LLVM, since this is the one built with wasm support.

[rust-highfive]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @aturon (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

[tlively]

r? @alexcrichton

cc @brson @eholk

The only thing this PR is missing to make the wasm32 builder fully functional is that clang (https://github.com/kripken/emscripten-fastcomp-clang/tree/incoming) needs to be in the src/llvm/tools directory so that emscripten can build the wasm libc bitcode files. It would be great if we could instead have a binary download to populate the emscripten cache or some other workaround, since having clang in-tree is undesirable.

Edit: I am exploring having the builder download a wasm-ready emscripten from the WebAssembly waterfall (https://wasm-stat.us/console) as a workaround.

[tlively]

This would have to be changed if the builder was ever moved out of the docker/disabled directory, but for now lets the builder be run locally without modification.

[alexcrichton]

Is this something that rustc should do automatically itself? (when compiling for the experimental wasm target)

[tlively]

Without this environment variable set, Emscripten tries to link the wasm bitcode file with the cached libc bitcode files for asmjs, which fails because asmjs and wasm have different memory layouts. Since this isn't just a debugging variable, rustc should definitely be setting it itself. When/if wasm32-experimental-emscripten is renamed to replace wasm32-unknown-emscripten, the part checking for "experimental" will need to be removed. It can't be removed now because then the current wasm32-unknown-emscripten would break.

[alexcrichton]

Oh yeah sure having this is fine, it just seems like it should be in rustc's backend (e.g. the invocation of the linker) rather than only in compiletest.

[tlively]

Oh good point! I'll fix that for the next revision.

[alexcrichton]

Mind expanding the comment here a bit? I'm actually not even sure what it means for emscripten to use someone else's llvm, isn't it precompiled?

[tlively]

This is problematic because as far as I know, rustc and LLVM aren't rebuilt for each bot. To actually get LLVM built with WebAssembly enabled will require setting this flag when it is built the first time. Is there a specific builder that is responsible for that initial build?

[Mark-Simulacrum]

We rebuild LLVM and rustc in each bot. The LLVM build will possibly be problematic, since we normally cache it, so with this I'm not sure whether the cache will work properly for both the wasm32 and the non-wasm32 bots.

[alexcrichton]

Oh this is fine, everything will work out with this configuration!

[alexcrichton]

@tlively at a high level can you explain to me why clang is needed? My current understanding of wasm is that the experimental backend in LLVM emites pure-wasm files (not bitcode) and then we use... something (emcc?) to link these together in the way that ld does for object files. Or does the experimental wasm backend still operate over bitcode?

[tlively]

@alexcrichton Unfortunately it's not quite as powerful as that. In the new experimental target, rustc still just emits bitcode files and tosses them over the fence to emcc. The difference is that it also tells emcc to use the experimental LLVM wasm backend, instead of going through fastcomp and asm.js like it otherwise would. In order for this to work 1) emscripten needs to be configured to use llvm tools built with the wasm backend enabled, and 2) emscripten needs to be able to build libc using the wasm backend or have it already cached so libc can be linked with the rustc-generated bitcode files. There are three ways to meet these requirements.

1. Build Rust's LLVM (and llvm tools) with the WebAssembly backend enabled, then configure emscripten to use those tools. Include clang in the tree so emscripten can use it as well. This is what I've been doing locally.

2. Build Rust's LLVM (and llvm tools) with the WebAssembly backend enabled, then configure emscripten to use those tools. Populate emscripten's cache by downloading prebuilt wasm libc bitcode files. I don't know where we could download prebuilt libc from, and this may still fail if emscripten looks for the existence of clang++ anyway.

3. Download wasm-ready emscripten and the llvm tools from the last known good build from the WebAssembly waterfall (https://wasm-stat.us/console). Probably no need for Rust's LLVM to be built with WebAssembly enabled, since its llvm tools will not be used. This is what I am trying to get working locally, but it's a little hairy.

[alexcrichton]

Ok thanks for the info! So it sounds like even though we're enabling the WebAssembly target in LLVM's backend, the rustc compiler itself isn't actually using it? As in, rustc is still emitting bitcode. I thought it was intended that we generate wasm "object files" with the wasm backend in LLVM, but is this not the case?

I guess though that emscripten then uses the wasm backend in LLVM to emit wasm files? (loads up bitcode and emits wasm files).

Where does this libc come from? Is it a bunch of C files that are compiled to wasm as well?

I guess what I'm sort of getting at is that this sounds like we're pretty close to not depending on wasm at all. If we leveraged LLVM's wasm back to generate wasm code, found some sort of linker for wasm code (maybe this is just LLVM?), and then got our "libc" written in Rust perhaps (or prebuilt), then we wouldn't need emcc at all?

I may be totally off base here though and not understanding. If you'd like I wouldn't mind chatting about this in a more high bandwidth setting (video/irc/etc), feel free to ping me on IRC if you'd like that!

[eholk]

@alexcrichton:

Ok thanks for the info! So it sounds like even though we're enabling the WebAssembly target in LLVM's backend, the rustc compiler itself isn't actually using it? As in, rustc is still emitting bitcode. I thought it was intended that we generate wasm "object files" with the wasm backend in LLVM, but is this not the case?

As far as I know, at the moment we don't have a great solution for linking wasm object files together. @sbc100 is making quick progress on this though, so hopefully we'll have something soon if we don't have it already.

[alexcrichton]

Good to hear @eholk!

For posterity @eholk, @tlively, and I chatted a bit on video about this, and the things I learned were:

• The wasm32-unknown-emscripten target as implemented today generates asm.js, and then generates wasm from that asm.js
• The wasm32-experimental-emscripten uses LLVM's native ability to emit wasm files.
• Shelling out to emcc loads a bunch of bitcode into a giant LLVM module and then emits wasm from there.
• The libc support library is lazily compiled by emcc, and requires clang/clang++
• The rustc compiler, nor clang, require the experimental WebAssembly backend in LLVM. Only emcc which runs the LLVM tools in "wasm mode" requires the wasm backend.
• The waterfall previously mentioned is a conglomeration of a bunch of wasm-realted tools. It's also generating a ton of artifacts from each successful build.

So it sounded to me like the wasm32-unknown-emscripten and wasm32-experimental-emscripten targets are quite similar today, but differ in rustc slightly based on how we call emcc and very differently in terms of the emcc compile-time requirements. The near term vision is then to get a travis builder for ourselves with these pieces:

• We'll download emsdk as we're doing today, which includes clang but doesn't include a WebAssembly-backend-enabled LLVM
• We'll build an LLVM with a WebAssembly backend for emcc to use
• We'll arrange for the LLVM root we build to have clang (symlinked from the emsdk)
• We'll instruct emcc to use the LLVM root so it uses WebAssembly-enabled tools.

In the longer term we'll:

• Create a script to download tools from the waterfall mentioned above. These tools are the right clang/llvm/emcc/etc versions and have WebAssembly enabled as appropriate.
• In our travis builder we'll download one of these waterfall artifacts as the "emsdk"
• We'll still build our own LLVM, but we won't need to enable WebAssembly (for now)
• We'll invoke emcc as usual

That "script" will ideally be usable by the community as well, and should hopefully enable easy usage of the native wasm backend of LLVM!

[tlively]

@alexcrichton This should be good to go! Instead of needing to symlink the wasm-ready LLVM tools, I only needed to change the emscripten configuration file, which should be relatively easy for end users to do. One advantage of this strategy over the previous one is that the builder automatically picks up the latest improvements to the wasm LLVM backend. There are tests that fail in WebAssembly instruction selection, but I have verified that these are real bugs, not bugs in the setup of the builder. I will let the right people know about these bugs and hopefully they will get fixed quickly.

[alexcrichton]

Looks good! It looks like one of the tests is failing though. We currently verify that we can roundtrip a Target through json so our custom target specs can take advantage of all the keys we have internally. You've added link_env here though which just means that we'll also need the associated serialize/deserialize functionality.

You can find deserialization here and serialization here. Perhaps this could be serialize/deserialized as a flat list of strings (with = signs) and then we parse that back out into key/value pairs?

[eholk]

FYI @vadimcn

[alexcrichton]

@bors: r+

Looks great!

[bors]

📌 Commit 4ad6a95 has been approved by alexcrichton

[bors]

⌛️ Testing commit 4ad6a95 with merge 3a90ff1...

[bors]

💔 Test failed - status-travis

[Mark-Simulacrum]

Looks like the asmjs failure is non-spurious:

[00:00:04] /scripts/emscripten.sh: line 37: jq: command not found

[alexcrichton]

@bors: r+

[bors]

📌 Commit c130b83 has been approved by alexcrichton

[bors]

⌛️ Testing commit c130b83 with merge bc9822a...

[bors]

☀️ Test successful - status-appveyor, status-travis
Approved by: alexcrichton
Pushing bc9822a to master...