Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a new wasm32-unknown-wasi target #59464

Merged
merged 1 commit into from Mar 30, 2019

Conversation

Projects
None yet
9 participants
@alexcrichton
Copy link
Member

alexcrichton commented Mar 27, 2019

This commit adds a new wasm32-based target distributed through rustup,
supported in the standard library, and implemented in the compiler. The
wasm32-unknown-wasi target is intended to be a WebAssembly target
which matches the WASI proposal recently announced. In summary
the WASI target is an effort to define a standard set of syscalls for
WebAssembly modules, allowing WebAssembly modules to not only be
portable across architectures but also be portable across environments
implementing this standard set of system calls.

The wasi target in libstd is still somewhat bare bones. This PR does not
fill out the filesystem, networking, threads, etc. Instead it only
provides the most basic of integration with the wasi syscalls, enabling
features like:

  • Instant::now and SystemTime::now work
  • env::args is hooked up
  • env::vars will look up environment variables
  • println! will print to standard out
  • process::{exit, abort} should be hooked up appropriately

None of these APIs can work natively on the wasm32-unknown-unknown
target, but with the assumption of the WASI set of syscalls we're able
to provide implementations of these syscalls that engines can implement.
Currently the primary engine implementing wasi is wasmtime, but more
will surely emerge!

In terms of future development of libstd, I think this is something
we'll probably want to discuss. The purpose of the WASI target is to
provide a standardized set of syscalls, but it's also to provide a
standard C sysroot for compiling C/C++ programs. This means it's
intended that functions like read and write are implemented for this
target with a relatively standard definition and implementation. It's
unclear, therefore, how we want to expose file descriptors and how we'll
want to implement system primitives. For example should std::fs::File
have a libc-based file descriptor underneath it? The raw wasi file
descriptor? We'll see! Currently these details are all intentionally
hidden and things we can change over time.

A WasiFd sample struct was added to the standard library as part of
this commit, but it's not currently used. It shows how all the wasi
syscalls could be ergonomically bound in Rust, and they offer a possible
implementation of primitives like std::fs::File if we bind wasi file
descriptors exactly.

Apart from the standard library, there's also the matter of how this
target is integrated with respect to its C standard library. The
reference sysroot, for example, provides managment of standard unix file
descriptors and also standard APIs like open (as opposed to the
relative openat inspiration for the wasi ssycalls). Currently the
standard library relies on the C sysroot symbols for operations such as
environment management, process exit, and read/write of stdio fds.
We want these operations in Rust to be interoperable with C if they're
used in the same process. Put another way, if Rust and C are linked into
the same WebAssembly binary they should work together, but that requires
that the same C standard library is used.

We also, however, want the wasm32-unknown-wasi target to be
usable-by-default with the Rust compiler without requiring a separate
toolchain to get downloaded and configured. With that in mind, there's
two modes of operation for the wasm32-unknown-wasi target:

  1. By default the C standard library is statically provided inside of
    liblibc.rlib distributed as part of the sysroot. This means that
    you can rustc foo.wasm --target wasm32-unknown-unknown and you're
    good to go, a fully workable wasi binary pops out. This is
    incompatible with linking in C code, however, which may be compiled
    against a different sysroot than the Rust code was previously
    compiled against. In this mode the default of rust-lld is used to
    link binaries.

  2. For linking with C code, the -C target-feature=-crt-static flag
    needs to be passed. This takes inspiration from the musl target for
    this flag, but the idea is that you're no longer using the provided
    static C runtime, but rather one will be provided externally. This
    flag is intended to also get coupled with an external clang
    compiler configured with its own sysroot. Therefore you'll typically
    use this flag with -C linker=/path/to/clang-script-wrapper. Using
    this mode the Rust code will continue to reference standard C
    symbols, but the definition will be pulled in by the linker configured.

Alright so that's all the current state of this PR. I suspect we'll
definitely want to discuss this before landing of course! This PR is
coupled with libc changes as well which I'll be posting shortly.

@rust-highfive

This comment has been minimized.

Copy link
Collaborator

rust-highfive commented Mar 27, 2019

r? @Mark-Simulacrum

(rust_highfive has picked a reviewer for you, use r? to override)

@alexcrichton

This comment has been minimized.

Copy link
Member Author

alexcrichton commented Mar 27, 2019

This is coupled with rust-lang/libc#1307 for the libc crate changes, and we'll need to publish that before merging this.

r? @fitzgen

@alexcrichton alexcrichton force-pushed the alexcrichton:wasi-pr branch from 1f92bf8 to 8fbc5b8 Mar 27, 2019

@fitzgen
Copy link
Member

fitzgen left a comment

🎉

Show resolved Hide resolved src/bootstrap/compile.rs Outdated
Show resolved Hide resolved src/bootstrap/compile.rs Outdated
Show resolved Hide resolved src/librustc_target/spec/wasm32_unknown_wasi.rs Outdated
//! The wasi target is **very** new in its specification. It's likely going to
//! be a long effort to get it standardized and stable. We'll be following it as
//! best we can with this target. Don't start relying on too much here unless
//! you know what you're getting in to!

This comment has been minimized.

Copy link
@fitzgen

fitzgen Mar 27, 2019

Member

Thanks for this very nice doc comment 👍

}

pub fn temp_dir() -> PathBuf {
panic!("no filesystem on wasm")

This comment has been minimized.

Copy link
@fitzgen

fitzgen Mar 27, 2019

Member

Might make sense to have an inline(never) function that all the unsupported panics defer to. I think this should make code sizes smaller.

This comment has been minimized.

Copy link
@alexcrichton

alexcrichton Mar 27, 2019

Author Member

Perhaps yeah! I figure we can fix that up as it goes along because unconditionally panicking binaries aren't too useful anyway :)

Show resolved Hide resolved src/libstd/sys/wasi/os.rs Outdated
@gnzlbg

This comment has been minimized.

Copy link
Contributor

gnzlbg commented Mar 27, 2019

For example should std::fs::File
have a libc-based file descriptor underneath it? The raw wasi file
descriptor? We'll see! Currently these details are all intentionally
hidden and things we can change over time.

A WasiFd sample struct was added to the standard library as part of
this commit, but it's not currently used. It shows how all the wasi
syscalls could be ergonomically bound in Rust, and they offer a possible
implementation of primitives like std::fs::File if we bind wasi file
descriptors exactly.

I'm not sure I understand. Is there something that prevents std::fs::File from being implemented on top of the libc wasi file descriptor? Are you arguing that maybe we should expose a WasiFd instead of std::fs::File for this target ?

@rust-highfive

This comment has been minimized.

Copy link
Collaborator

rust-highfive commented Mar 27, 2019

Your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@alexcrichton alexcrichton force-pushed the alexcrichton:wasi-pr branch from 8fbc5b8 to 0f0dd71 Mar 27, 2019

@alexcrichton

This comment has been minimized.

Copy link
Member Author

alexcrichton commented Mar 27, 2019

Is there something that prevents std::fs::File from being implemented on top of the libc wasi file descriptor? Are you arguing that maybe we should expose a WasiFd instead of std::fs::File for this target ?

This sort of depends on how the file descriptor situation is settled in wasi's sysroot. It seems like there's the concept of a "wasi file descriptor" as well as a "libc file descriptor", indexing different spaces. For those purposes we'd need to convert between them.

This is sort of similar as well to "what if we used FILE as the implementation of std::fs::File on Unix?". I think the answer for wasi is "all primitives should internally contain WasiFd", but we just need to figure out a way how to do that which still interoperates with C code well.

@rust-highfive

This comment has been minimized.

Copy link
Collaborator

rust-highfive commented Mar 27, 2019

Your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

bors added a commit to rust-lang/libc that referenced this pull request Mar 27, 2019

Auto merge of #1307 - alexcrichton:wasi-pr, r=gnzlbg
Add intial support for wasm32-unknown-wasi

This target is [being proposed][LINK] int he rust-lang/rust repository
and this is intended to get coupled with that proposal. The definitions
here all match the upstream reference-sysroot definitions and the
functions all match the reference sysroot as well. The linkage here is
described more in detail on the Rust PR itself, but in general it's
similar to musl.

Automatic verification has been implemented in the same manner as other
targets, and it's been used locally to develop this PR and catch errors
in the bindings already written (also to help match the evolving sysroot
of wasi). The verification isn't hooked up to CI yet though because
there is no wasi target distributed via rustup just yet, but once that's
done I'll file a follow-up PR to execute verification on CI.

[LINK]: rust-lang/rust#59464

set -ex

curl http://releases.llvm.org/8.0.0/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-14.04.tar.xz | \

This comment has been minimized.

Copy link
@chicoxyzzy

chicoxyzzy Mar 28, 2019

Might make sense to use latest LTS (18.04) instead

This comment has been minimized.

Copy link
@alexcrichton

alexcrichton Mar 28, 2019

Author Member

We will eventually! For now though the rest of the image here is 17.10, but when that upgrades we can upgrade this

bors added a commit to rust-lang/libc that referenced this pull request Mar 28, 2019

Auto merge of #1307 - alexcrichton:wasi-pr, r=gnzlbg
Add intial support for wasm32-unknown-wasi

This target is [being proposed][LINK] int he rust-lang/rust repository
and this is intended to get coupled with that proposal. The definitions
here all match the upstream reference-sysroot definitions and the
functions all match the reference sysroot as well. The linkage here is
described more in detail on the Rust PR itself, but in general it's
similar to musl.

Automatic verification has been implemented in the same manner as other
targets, and it's been used locally to develop this PR and catch errors
in the bindings already written (also to help match the evolving sysroot
of wasi). The verification isn't hooked up to CI yet though because
there is no wasi target distributed via rustup just yet, but once that's
done I'll file a follow-up PR to execute verification on CI.

[LINK]: rust-lang/rust#59464

bors added a commit to rust-lang/libc that referenced this pull request Mar 28, 2019

Auto merge of #1307 - alexcrichton:wasi-pr, r=gnzlbg
Add intial support for wasm32-unknown-wasi

This target is [being proposed][LINK] int he rust-lang/rust repository
and this is intended to get coupled with that proposal. The definitions
here all match the upstream reference-sysroot definitions and the
functions all match the reference sysroot as well. The linkage here is
described more in detail on the Rust PR itself, but in general it's
similar to musl.

Automatic verification has been implemented in the same manner as other
targets, and it's been used locally to develop this PR and catch errors
in the bindings already written (also to help match the evolving sysroot
of wasi). The verification isn't hooked up to CI yet though because
there is no wasi target distributed via rustup just yet, but once that's
done I'll file a follow-up PR to execute verification on CI.

[LINK]: rust-lang/rust#59464
@sunfishcode

This comment has been minimized.

Copy link
Contributor

sunfishcode commented Mar 28, 2019

This sort of depends on how the file descriptor situation is settled in wasi's sysroot. It seems like there's the concept of a "wasi file descriptor" as well as a "libc file descriptor", indexing different spaces. For those purposes we'd need to convert between them.

WASI uses __wasi_fd_t which is unsigned, while libc/POSIX use int which is signed so that it can use negative numbers to indicate errors, but it's the same index space.

@alexcrichton

This comment has been minimized.

Copy link
Member Author

alexcrichton commented Mar 28, 2019

@sunfishcode oh ok thanks! So the idea is that a libc file descriptor is interchangeable with a wasi file descriptor? If that's the case sounds like we'll be using a design similar to unix for libstd where every type just internally contains a 32-bit file descriptor (probably the WasiFd type in this PR)

@sunfishcode

This comment has been minimized.

Copy link
Contributor

sunfishcode commented Mar 28, 2019

Yes, they're interchangeable. (If WASI ever produces an fd >= 231, libc might get confused, but realistically you'll hit other limits well before that happens.)

@alexcrichton alexcrichton force-pushed the alexcrichton:wasi-pr branch from 0f0dd71 to f21d15b Mar 29, 2019

@alexcrichton

This comment was marked as outdated.

Copy link
Member Author

alexcrichton commented Mar 29, 2019

@bors: r=fitzgen

@bors

This comment was marked as outdated.

Copy link
Contributor

bors commented Mar 29, 2019

📌 Commit f21d15b has been approved by fitzgen

@alexcrichton alexcrichton force-pushed the alexcrichton:wasi-pr branch from f21d15b to d73d497 Mar 29, 2019

@alexcrichton

This comment was marked as outdated.

Copy link
Member Author

alexcrichton commented Mar 29, 2019

@bors: r=fitzgen

@bors

This comment was marked as outdated.

Copy link
Contributor

bors commented Mar 29, 2019

📌 Commit d73d497 has been approved by fitzgen

@rust-highfive

This comment was marked as resolved.

Copy link
Collaborator

rust-highfive commented Mar 29, 2019

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:0653b3f8:start=1553827120828308087,finish=1553827123013924573,duration=2185616486
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
Setting environment variables from .travis.yml
---

[00:04:18] travis_fold:start:tidy
travis_time:start:tidy
tidy check
[00:04:19] tidy error: /checkout/src/ci/docker/dist-various-2/build-wasi-toolchain.sh:5: line longer than 100 chars
[00:04:19] tidy error: /checkout/src/ci/docker/dist-various-2/build-wasi-toolchain.sh:6: line longer than 100 chars
[00:04:20] some tidy checks failed
[00:04:20] 
[00:04:20] 
[00:04:20] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/tidy" "/checkout/src" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "--no-vendor" "--quiet"
[00:04:20] 
[00:04:20] 
[00:04:20] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
[00:04:20] Build completed unsuccessfully in 0:00:46
[00:04:20] Build completed unsuccessfully in 0:00:46
[00:04:20] make: *** [tidy] Error 1
[00:04:20] Makefile:67: recipe for target 'tidy' failed
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:15670647
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Fri Mar 29 02:43:16 UTC 2019
---
travis_time:end:201b9e02:start=1553827397744973737,finish=1553827397752838945,duration=7865208
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:0157a544
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:0d56d55c
travis_time:start:0d56d55c
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:218f36b4
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@skyne98 skyne98 referenced this pull request Mar 29, 2019

Open

Implement WASI strandard #11

@Centril

This comment has been minimized.

Copy link
Contributor

Centril commented Mar 29, 2019

For rollup planning, @bors p=2

@Centril

This comment has been minimized.

Copy link
Contributor

Centril commented Mar 29, 2019

@bors p=2 rollup-

@alexcrichton alexcrichton force-pushed the alexcrichton:wasi-pr branch from c880ef6 to e59ddbf Mar 29, 2019

@alexcrichton

This comment has been minimized.

Copy link
Member Author

alexcrichton commented Mar 29, 2019

@bors: r=fitzgen

@bors

This comment has been minimized.

Copy link
Contributor

bors commented Mar 29, 2019

📌 Commit e59ddbf has been approved by fitzgen

@bors

This comment has been minimized.

Copy link
Contributor

bors commented Mar 29, 2019

⌛️ Testing commit e59ddbf with merge a31f4e2...

bors added a commit that referenced this pull request Mar 29, 2019

Auto merge of #59464 - alexcrichton:wasi-pr, r=fitzgen
Add a new wasm32-unknown-wasi target

This commit adds a new wasm32-based target distributed through rustup,
supported in the standard library, and implemented in the compiler. The
`wasm32-unknown-wasi` target is intended to be a WebAssembly target
which matches the [WASI proposal recently announced][LINK]. In summary
the WASI target is an effort to define a standard set of syscalls for
WebAssembly modules, allowing WebAssembly modules to not only be
portable across architectures but also be portable across environments
implementing this standard set of system calls.

The wasi target in libstd is still somewhat bare bones. This PR does not
fill out the filesystem, networking, threads, etc. Instead it only
provides the most basic of integration with the wasi syscalls, enabling
features like:

* `Instant::now` and `SystemTime::now` work
* `env::args` is hooked up
* `env::vars` will look up environment variables
* `println!` will print to standard out
* `process::{exit, abort}` should be hooked up appropriately

None of these APIs can work natively on the `wasm32-unknown-unknown`
target, but with the assumption of the WASI set of syscalls we're able
to provide implementations of these syscalls that engines can implement.
Currently the primary engine implementing wasi is [wasmtime], but more
will surely emerge!

In terms of future development of libstd, I think this is something
we'll probably want to discuss. The purpose of the WASI target is to
provide a standardized set of syscalls, but it's *also* to provide a
standard C sysroot for compiling C/C++ programs. This means it's
intended that functions like `read` and `write` are implemented for this
target with a relatively standard definition and implementation. It's
unclear, therefore, how we want to expose file descriptors and how we'll
want to implement system primitives. For example should `std::fs::File`
have a libc-based file descriptor underneath it? The raw wasi file
descriptor? We'll see! Currently these details are all intentionally
hidden and things we can change over time.

A `WasiFd` sample struct was added to the standard library as part of
this commit, but it's not currently used. It shows how all the wasi
syscalls could be ergonomically bound in Rust, and they offer a possible
implementation of primitives like `std::fs::File` if we bind wasi file
descriptors exactly.

Apart from the standard library, there's also the matter of how this
target is integrated with respect to its C standard library. The
reference sysroot, for example, provides managment of standard unix file
descriptors and also standard APIs like `open` (as opposed to the
relative `openat` inspiration for the wasi ssycalls). Currently the
standard library relies on the C sysroot symbols for operations such as
environment management, process exit, and `read`/`write` of stdio fds.
We want these operations in Rust to be interoperable with C if they're
used in the same process. Put another way, if Rust and C are linked into
the same WebAssembly binary they should work together, but that requires
that the same C standard library is used.

We also, however, want the `wasm32-unknown-wasi` target to be
usable-by-default with the Rust compiler without requiring a separate
toolchain to get downloaded and configured. With that in mind, there's
two modes of operation for the `wasm32-unknown-wasi` target:

1. By default the C standard library is statically provided inside of
   `liblibc.rlib` distributed as part of the sysroot. This means that
   you can `rustc foo.wasm --target wasm32-unknown-unknown` and you're
   good to go, a fully workable wasi binary pops out. This is
   incompatible with linking in C code, however, which may be compiled
   against a different sysroot than the Rust code was previously
   compiled against. In this mode the default of `rust-lld` is used to
   link binaries.

2. For linking with C code, the `-C target-feature=-crt-static` flag
   needs to be passed. This takes inspiration from the musl target for
   this flag, but the idea is that you're no longer using the provided
   static C runtime, but rather one will be provided externally. This
   flag is intended to also get coupled with an external `clang`
   compiler configured with its own sysroot. Therefore you'll typically
   use this flag with `-C linker=/path/to/clang-script-wrapper`. Using
   this mode the Rust code will continue to reference standard C
   symbols, but the definition will be pulled in by the linker configured.

Alright so that's all the current state of this PR. I suspect we'll
definitely want to discuss this before landing of course! This PR is
coupled with libc changes as well which I'll be posting shortly.

[LINK]: https://hacks.mozilla.org/2019/03/standardizing-wasi-a-webassembly-system-interface/
[wasmtime]: https://github.com/cranestation/wasmtime-wasi
@Centril

This comment has been minimized.

Copy link
Contributor

Centril commented Mar 29, 2019

@bors retry

@bors

This comment has been minimized.

Copy link
Contributor

bors commented Mar 29, 2019

🔒 Merge conflict

This pull request and the master branch diverged in a way that cannot be automatically merged. Please rebase on top of the latest master branch, and let the reviewer approve again.

How do I rebase?

Assuming self is your fork and upstream is this repository, you can resolve the conflict following these steps:

  1. git checkout wasi-pr (switch to your branch)
  2. git fetch upstream master (retrieve the latest master)
  3. git rebase upstream/master -p (rebase on top of it)
  4. Follow the on-screen instruction to resolve conflicts (check git status if you got lost).
  5. git push self wasi-pr --force-with-lease (update this PR)

You may also read Git Rebasing to Resolve Conflicts by Drew Blessing for a short tutorial.

Please avoid the "Resolve conflicts" button on GitHub. It uses git merge instead of git rebase which makes the PR commit history more difficult to read.

Sometimes step 4 will complete without asking for resolution. This is usually due to difference between how Cargo.lock conflict is handled during merge and rebase. This is normal, and you should still perform step 5 to update this PR.

Error message
warning: Cannot merge binary files: Cargo.lock (HEAD vs. heads/homu-tmp)
Auto-merging src/bootstrap/compile.rs
Auto-merging src/bootstrap/bin/rustc.rs
Auto-merging Cargo.lock
CONFLICT (content): Merge conflict in Cargo.lock
Automatic merge failed; fix conflicts and then commit the result.

Add a new wasm32-unknown-wasi target
This commit adds a new wasm32-based target distributed through rustup,
supported in the standard library, and implemented in the compiler. The
`wasm32-unknown-wasi` target is intended to be a WebAssembly target
which matches the [WASI proposal recently announced.][LINK]. In summary
the WASI target is an effort to define a standard set of syscalls for
WebAssembly modules, allowing WebAssembly modules to not only be
portable across architectures but also be portable across environments
implementing this standard set of system calls.

The wasi target in libstd is still somewhat bare bones. This PR does not
fill out the filesystem, networking, threads, etc. Instead it only
provides the most basic of integration with the wasi syscalls, enabling
features like:

* `Instant::now` and `SystemTime::now` work
* `env::args` is hooked up
* `env::vars` will look up environment variables
* `println!` will print to standard out
* `process::{exit, abort}` should be hooked up appropriately

None of these APIs can work natively on the `wasm32-unknown-unknown`
target, but with the assumption of the WASI set of syscalls we're able
to provide implementations of these syscalls that engines can implement.
Currently the primary engine implementing wasi is [wasmtime], but more
will surely emerge!

In terms of future development of libstd, I think this is something
we'll probably want to discuss. The purpose of the WASI target is to
provide a standardized set of syscalls, but it's *also* to provide a
standard C sysroot for compiling C/C++ programs. This means it's
intended that functions like `read` and `write` are implemented for this
target with a relatively standard definition and implementation. It's
unclear, therefore, how we want to expose file descriptors and how we'll
want to implement system primitives. For example should `std::fs::File`
have a libc-based file descriptor underneath it? The raw wasi file
descriptor? We'll see! Currently these details are all intentionally
hidden and things we can change over time.

A `WasiFd` sample struct was added to the standard library as part of
this commit, but it's not currently used. It shows how all the wasi
syscalls could be ergonomically bound in Rust, and they offer a possible
implementation of primitives like `std::fs::File` if we bind wasi file
descriptors exactly.

Apart from the standard library, there's also the matter of how this
target is integrated with respect to its C standard library. The
reference sysroot, for example, provides managment of standard unix file
descriptors and also standard APIs like `open` (as opposed to the
relative `openat` inspiration for the wasi ssycalls). Currently the
standard library relies on the C sysroot symbols for operations such as
environment management, process exit, and `read`/`write` of stdio fds.
We want these operations in Rust to be interoperable with C if they're
used in the same process. Put another way, if Rust and C are linked into
the same WebAssembly binary they should work together, but that requires
that the same C standard library is used.

We also, however, want the `wasm32-unknown-wasi` target to be
usable-by-default with the Rust compiler without requiring a separate
toolchain to get downloaded and configured. With that in mind, there's
two modes of operation for the `wasm32-unknown-wasi` target:

1. By default the C standard library is statically provided inside of
   `liblibc.rlib` distributed as part of the sysroot. This means that
   you can `rustc foo.wasm --target wasm32-unknown-unknown` and you're
   good to go, a fully workable wasi binary pops out. This is
   incompatible with linking in C code, however, which may be compiled
   against a different sysroot than the Rust code was previously
   compiled against. In this mode the default of `rust-lld` is used to
   link binaries.

2. For linking with C code, the `-C target-feature=-crt-static` flag
   needs to be passed. This takes inspiration from the musl target for
   this flag, but the idea is that you're no longer using the provided
   static C runtime, but rather one will be provided externally. This
   flag is intended to also get coupled with an external `clang`
   compiler configured with its own sysroot. Therefore you'll typically
   use this flag with `-C linker=/path/to/clang-script-wrapper`. Using
   this mode the Rust code will continue to reference standard C
   symbols, but the definition will be pulled in by the linker configured.

Alright so that's all the current state of this PR. I suspect we'll
definitely want to discuss this before landing of course! This PR is
coupled with libc changes as well which I'll be posting shortly.

[LINK]:
[wasmtime]:

@alexcrichton alexcrichton force-pushed the alexcrichton:wasi-pr branch from e59ddbf to ace7124 Mar 29, 2019

@alexcrichton

This comment has been minimized.

Copy link
Member Author

alexcrichton commented Mar 29, 2019

@bors: r=fitzgen

@bors

This comment has been minimized.

Copy link
Contributor

bors commented Mar 29, 2019

📌 Commit ace7124 has been approved by fitzgen

@bors

This comment has been minimized.

Copy link
Contributor

bors commented Mar 30, 2019

⌛️ Testing commit ace7124 with merge 709b72e...

bors added a commit that referenced this pull request Mar 30, 2019

Auto merge of #59464 - alexcrichton:wasi-pr, r=fitzgen
Add a new wasm32-unknown-wasi target

This commit adds a new wasm32-based target distributed through rustup,
supported in the standard library, and implemented in the compiler. The
`wasm32-unknown-wasi` target is intended to be a WebAssembly target
which matches the [WASI proposal recently announced][LINK]. In summary
the WASI target is an effort to define a standard set of syscalls for
WebAssembly modules, allowing WebAssembly modules to not only be
portable across architectures but also be portable across environments
implementing this standard set of system calls.

The wasi target in libstd is still somewhat bare bones. This PR does not
fill out the filesystem, networking, threads, etc. Instead it only
provides the most basic of integration with the wasi syscalls, enabling
features like:

* `Instant::now` and `SystemTime::now` work
* `env::args` is hooked up
* `env::vars` will look up environment variables
* `println!` will print to standard out
* `process::{exit, abort}` should be hooked up appropriately

None of these APIs can work natively on the `wasm32-unknown-unknown`
target, but with the assumption of the WASI set of syscalls we're able
to provide implementations of these syscalls that engines can implement.
Currently the primary engine implementing wasi is [wasmtime], but more
will surely emerge!

In terms of future development of libstd, I think this is something
we'll probably want to discuss. The purpose of the WASI target is to
provide a standardized set of syscalls, but it's *also* to provide a
standard C sysroot for compiling C/C++ programs. This means it's
intended that functions like `read` and `write` are implemented for this
target with a relatively standard definition and implementation. It's
unclear, therefore, how we want to expose file descriptors and how we'll
want to implement system primitives. For example should `std::fs::File`
have a libc-based file descriptor underneath it? The raw wasi file
descriptor? We'll see! Currently these details are all intentionally
hidden and things we can change over time.

A `WasiFd` sample struct was added to the standard library as part of
this commit, but it's not currently used. It shows how all the wasi
syscalls could be ergonomically bound in Rust, and they offer a possible
implementation of primitives like `std::fs::File` if we bind wasi file
descriptors exactly.

Apart from the standard library, there's also the matter of how this
target is integrated with respect to its C standard library. The
reference sysroot, for example, provides managment of standard unix file
descriptors and also standard APIs like `open` (as opposed to the
relative `openat` inspiration for the wasi ssycalls). Currently the
standard library relies on the C sysroot symbols for operations such as
environment management, process exit, and `read`/`write` of stdio fds.
We want these operations in Rust to be interoperable with C if they're
used in the same process. Put another way, if Rust and C are linked into
the same WebAssembly binary they should work together, but that requires
that the same C standard library is used.

We also, however, want the `wasm32-unknown-wasi` target to be
usable-by-default with the Rust compiler without requiring a separate
toolchain to get downloaded and configured. With that in mind, there's
two modes of operation for the `wasm32-unknown-wasi` target:

1. By default the C standard library is statically provided inside of
   `liblibc.rlib` distributed as part of the sysroot. This means that
   you can `rustc foo.wasm --target wasm32-unknown-unknown` and you're
   good to go, a fully workable wasi binary pops out. This is
   incompatible with linking in C code, however, which may be compiled
   against a different sysroot than the Rust code was previously
   compiled against. In this mode the default of `rust-lld` is used to
   link binaries.

2. For linking with C code, the `-C target-feature=-crt-static` flag
   needs to be passed. This takes inspiration from the musl target for
   this flag, but the idea is that you're no longer using the provided
   static C runtime, but rather one will be provided externally. This
   flag is intended to also get coupled with an external `clang`
   compiler configured with its own sysroot. Therefore you'll typically
   use this flag with `-C linker=/path/to/clang-script-wrapper`. Using
   this mode the Rust code will continue to reference standard C
   symbols, but the definition will be pulled in by the linker configured.

Alright so that's all the current state of this PR. I suspect we'll
definitely want to discuss this before landing of course! This PR is
coupled with libc changes as well which I'll be posting shortly.

[LINK]: https://hacks.mozilla.org/2019/03/standardizing-wasi-a-webassembly-system-interface/
[wasmtime]: https://github.com/cranestation/wasmtime-wasi
@bors

This comment has been minimized.

Copy link
Contributor

bors commented Mar 30, 2019

☀️ Test successful - checks-travis, status-appveyor
Approved by: fitzgen
Pushing 709b72e to master...

@bors bors added the merged-by-bors label Mar 30, 2019

@bors bors merged commit ace7124 into rust-lang:master Mar 30, 2019

1 check passed

homu Test successful
Details

@chpio chpio referenced this pull request Apr 2, 2019

Open

Compile to WASM #33

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.