From 43880bd242d8c9a020e6578216109af3ce1af732 Mon Sep 17 00:00:00 2001 From: Frank Denis Date: Fri, 7 Jun 2019 13:07:08 +0200 Subject: [PATCH] Remove README.md, direct users to the Wiki --- README.md | 402 +------------------------------------------ lucet-idl/README.md | 129 -------------- lucet-wasi/README.md | 48 ------ 3 files changed, 5 insertions(+), 574 deletions(-) delete mode 100644 lucet-idl/README.md delete mode 100644 lucet-wasi/README.md diff --git a/README.md b/README.md index a7c559494..db9fa0fda 100644 --- a/README.md +++ b/README.md @@ -3,8 +3,8 @@ [Build Status]: https://travis-ci.org/fastly/lucet.svg?branch=master [travis]: https://travis-ci.org/fastly/lucet -**Lucet is a native WebAssembly compiler and runtime. It is designed to safely -execute untrusted WebAssembly programs inside your application.** +**Lucet is a native WebAssembly compiler and runtime. It is designed +to safely execute untrusted WebAssembly programs inside your application.** Check out our [announcement post on the Fastly blog](https://www.fastly.com/blog/announcing-lucet-fastly-native-webassembly-compiler-runtime). @@ -12,400 +12,8 @@ blog](https://www.fastly.com/blog/announcing-lucet-fastly-native-webassembly-com Lucet uses, and is developed in collaboration with, Mozilla's [Cranelift](http://github.com/cranestation/cranelift) code generator. -Lucet powers Fastly's [Terrarium](https://wasm.fastlylabs.com) platform. +It powers Fastly's [Terrarium](https://wasm.fastlylabs.com) platform. ---- +Lucet's documentation is available in the Wiki: -## Status - -Lucet supports running WebAssembly programs written in C (via `clang`), Rust, -and AssemblyScript. It does not yet support the entire WebAssembly spec, but -full support is [coming in the near future](#lucet-spectest). - -Lucet's runtime currently only supports x86-64 based Linux systems, with -experimental support for macOS. - -## Contents - -### `lucetc` - -`lucetc` is the Lucet Compiler. - -The Rust crate `lucetc` provides an executable `lucetc`. It compiles -WebAssembly modules (`.wasm` or `.wat` files) into native code (`.o` or `.so` -files). - -### `lucet-runtime` - -`lucet-runtime` is the runtime for WebAssembly modules compiled through -`lucetc`. It is a Rust crate that provides the functionality to load modules -from shared object files, instantiate them, and call exported WebAssembly -functions. `lucet-runtime` manages the resources used by each WebAssembly -instance (linear memory & globals), and the exception mechanisms that detect -and recover from illegal operations. - -The bulk of the library is defined in the child crate -`lucet-runtime-internals`. The public API is exposed in `lucet-runtime`. Test -suites are defined in the child crate `lucet-runtime-tests`. Many of these -tests invoke `lucetc` and the `wasi-sdk` tools. - -`lucet-runtime` is usable as a Rust crate or as a C library. The C language -interface is found at `lucet-runtime/include/lucet.h`. - -### `lucet-wasi` - -`lucet-wasi` is a crate providing runtime support for the [WebAssembly System -Interface (WASI)](https://wasi.dev). It can be used as a library to support -WASI in another application, or as an executable, `lucet-wasi`, to execute WASI -programs compiled through `lucetc`. - -See ["Your first Lucet application"](#your-first-lucet-application) for an -example that builds a C program and executes it with `lucet-wasi`. - -For details on WASI's implementation, see -[`lucet-wasi/README.md`](lucet-wasi/README.md). - -### `lucet-wasi-sdk` - -[`wasi-sdk`](https://github.com/cranestation/wasi-sdk) is a Cranelift project -that packages a build of the Clang toolchain, the WASI reference sysroot, and a -libc based on WASI syscalls. `lucet-wasi-sdk` is a Rust crate that provides -wrappers around these tools for building C programs into Lucet modules. We use -this crate to build test cases in `lucet-runtime-tests` and `lucet-wasi`. - -### `lucet-module-data` - -`lucet-module-data` is a crate with data structure definitions and serialization -functions that we emit into shared objects with `lucetc`, and read with -`lucet-runtime`. - -### `lucet-analyze` - -`lucet-analyze` is a Rust executable for inspecting the contents of a shared -object generated by `lucetc`. - -### `lucet-idl` - -`lucet-idl` is a Rust executable that implements code generation via an -Interface Description Language (IDL). The generated code provides zero-copy -accessor and constructor functions for datatypes that have the same -representation in both the WebAssembly guest program and the host program. - -Functionality is incomplete at the time of writing, and not yet integrated with -other parts of the project. Rust code generator, definition of import and -export function interfaces, and opaque type definitions are planned for the -near future. - -### `lucet-spectest` - -`lucet-spectest` is a Rust crate that uses `lucetc` and `lucet-runtime`, as well -as the (external) `wabt` crate, to run the official WebAssembly spec test suite, -which is provided as a submodule in this directory. Lucet is not yet fully spec -compliant, and the implementation of `lucet-spectest` has not been maintained -very well during recent codebase evolutions. We expect to fix this up and reach -spec compliance in the near future. - -### `lucet-builtins` - -`lucet-builtins` is a C library that provides optimized native versions of libc -primitives. `lucetc` can substitute the implementations defined in this library -for the WebAssembly implementations. - -`lucet-builtins/wasmonkey` is the Rust crate that `lucetc` uses to transform -function definitions in a WebAssembly module into uses of an import function. - -### Vendor libraries - -Lucet is tightly coupled to several upstream dependencies, and Lucet -development often requires making changes to these dependencies which are -submitted upstream once fully baked. To reduce friction in this development -cycle, we use git submodules to vendor these modules into the Lucet source -tree. - -#### Cranelift - -We keep the primary Cranelift project repository as a submodule at -`/cranelift`. - -Cranelift provides the native code generator used by `lucetc`, and a ton of -supporting infrastructure. - -Cranelift was previously known as Cretonne. Project developers hang out in the -`#cranelift` channel on [`irc.mozilla.org:6697`](https://wiki.mozilla.org/IRC). - -#### Faerie - -`faerie` is a Rust crate for producing ELF files. Faerie is used by Cranelift -(through the module system's `cranelift-faerie` backend) and also directly by -`lucetc`, for places where the `cranelift-module` API can't do everything we -need. - -### Tests - -Most of the crates in this repository have some form of unit tests. In addition, -`lucet-runtime/lucet-runtime-tests` defines a number of integration tests for -the runtime, and `lucet-wasi` has a number of integration tests using WASI C -programs. - -### Benchmarks - -We created the `sightglass` benchmarking tool to measure the runtime of C code -compiled through a standard native toolchain against the Lucet toolchain. It -is provided as a submodule at `/sightglass`. - -Sightglass ships with a set of microbenchmarks called `shootout`. The scripts -to build the shootout tests with native and various versions of the Lucet -toolchain are in `/benchmarks/shootout`. - -Furthermore, there is a suite of benchmarks of various Lucet runtime functions, -such as instance creation and teardown, in `/benchmarks/lucet-benchmarks`. - -## Development Environment - -### Operating System - -Lucet is developed and tested on Linux and macOS. We expect it to work on any -POSIX system which supports shared libraries. - -### Dependencies - -Lucet requires: - -* Stable Rust, and `rustfmt`. We typically track the latest stable release. -* [`wasi-sdk`](https://github.com/CraneStation/wasi-sdk), providing a Clang - toolchain with wasm-ld, the WASI reference sysroot, and a libc based on WASI - syscalls. -* GNU Make, CMake, & various standard Unix utilities for the build system. - -### Getting started - -The easiest way to get started with the Lucet toolchain is by using the provided -Docker-based development environment. - -This repository includes a `Dockerfile` to build a complete environment for -compiling and running WebAssembly code with Lucet, but you shouldn't have to use -Docker commands directly. A set of shell scripts with the `devenv_` prefix are -used to manage the container. - -#### Setting up the environment - -1) The Lucet repository uses git submodules. Make sure they are checked out - by running `git submodule init && git submodule update`. - -2) Install and run the `docker` service. We do not support `podman` at this - time. On MacOS, [Docker for - Mac](https://docs.docker.com/docker-for-mac/install/) is an option. - -3) Once Docker is running, in a terminal, and at the root of the cloned - repository, run: `source devenv_setenv.sh`. (This command requires the - current shell to be `zsh`, `ksh` or `bash`). After a couple minutes, the - Docker image is built and a new container is run. - -4) Check that new commands are now available: - -```sh -lucetc --help -``` - -You're now all set! - -#### Your first Lucet application - -The `devenv_setenv.sh` shell script ensures the Lucet executables are available -in your shell. Under the hood, these commands are executed in the Docker -container. The container has limited visibility into the host's filesystem - it -can only see files under the `lucet` repository. - -Create a new work directory in the `lucet` directory: - -```sh -mkdir -p src/hello - -cd src/hello -``` - -Save the following C source code as `hello.c`: - -```c -#include - -int main(void) -{ - puts("Hello world"); - return 0; -} -``` - -Time to compile to WebAssembly! The development environment includes a version -of the Clang toolchain that is built to generate WebAssembly by default. The -related commands are accessible from your current shell, and are prefixed by -`wasm32-wasi-`. - -For example, to create a WebAssembly module `hello.wasm` from `hello.c`: - -```sh -wasm32-wasi-clang -Ofast -o hello.wasm hello.c -``` - -The next step is to use Lucet to build native `x86_64` code from that -WebAssembly file: - -```sh -lucetc-wasi -o hello.so hello.wasm -``` - -`lucetc` is the WebAssembly to native code compiler. The `lucetc-wasi` command -runs the same compiler, but automatically configures it to target WASI. - -`hello.so` is created and ready to be run: - -```sh -lucet-wasi hello.so -``` - -#### Additional shell commands - -* `./devenv_build_container.sh` rebuilds the container image. This is never - required unless you edit the `Dockerfile`. -* `./devenv_run.sh [] [...]` runs a command in the container. If - a command is not provided, an interactive shell is spawned. In this - container, Lucet tools are installed in `/opt/lucet` by default. The command - `source /opt/lucet/bin/devenv_setenv.sh` can be used to initialize the - environment. -* `./devenv_start.sh` and `./devenv_stop.sh` start and stop the container. - -### Compiling the toolchain without Docker - -Support for WebAssembly was introduced in LLVM 8, released in March 2019. - -As a result, Lucet can be compiled with an existing LLVM installation, provided -that it is up to date. - -We successfully compiled it on macOS, Arch Linux and Ubuntu 19.04 using standard -system packages. - -#### Compilation on Ubuntu 19.04 - -On recent Ubuntu versions, the `cmake`, `clang` and `lld` packages must be -installed: - -```sh -apt install curl ca-certificates clang lld - -update-alternatives --install /usr/bin/wasm-ld wasm-ld /usr/bin/wasm-ld-8 100 -``` - -In order to compile applications to WebAssembly, builtins need to be installed -as well: - -```sh -curl -sL https://github.com/CraneStation/wasi-sdk/releases/download/wasi-sdk-5/libclang_rt.builtins-wasm32-wasi-5.0.tar.gz | \ -sudo tar x -zf - -C /usr/lib/llvm-8/lib/clang/8.0.0 -``` - -Install the latest stable version of the Rust compiler: - -```sh -curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -source $HOME/.cargo/env -``` - -Fetch, compile and install the WASI sysroot: - -```sh -git clone --recursive https://github.com/CraneStation/wasi-sysroot - -cd wasi-sysroot - -sudo make WASM_CC=clang-8 WASM_NM=llvm-nm-8 WASM_AR=llvm-ar-8 \ - INSTALL_DIR=/opt/wasi-sysroot install - -cd - && sudo rm -fr wasi-sysroot -``` - -Enter the Lucet git repository clone, and fetch/update the submodules: - -```sh -cd lucet - -git submodule update --init -``` - -Set relevant environment variables: - -```sh -export WASI_SYSROOT=/opt/wasi-sysroot -export CLANG_ROOT=/usr/lib/llvm-8/lib/clang/8.0.0 -export CLANG=clang-8 -``` - -Finally, compile the toolchain: - -```sh -cargo build --release -``` - -### Compilation on macOS - -Install `llvm`, `rust` and `cmake` using [Homebrew](https://brew.sh): - -```sh -brew install llvm rust cmake -``` - -In order to compile applications to WebAssembly, builtins need to be installed -as well: - -```sh -curl -sL https://github.com/CraneStation/wasi-sdk/releases/download/wasi-sdk-5/libclang_rt.builtins-wasm32-wasi-5.0.tar.gz | \ -sudo tar x -zf - -C /usr/local/opt/llvm/lib/clang/8* -``` - -Fetch, compile and install the WASI sysroot: - -```sh -git clone --recursive https://github.com/CraneStation/wasi-sysroot - -cd wasi-sysroot - -sudo env PATH=/usr/local/opt/llvm/bin:$PATH \ - make INSTALL_DIR=/opt/wasi-sysroot install - -cd - && sudo rm -fr wasi-sysroot -``` - -Enter the Lucet git repository clone, and fetch/update the submodules: - -```sh -cd lucet - -git submodule update --init -``` - -Set relevant environment variables: - -```sh -export WASI_SYSROOT=/opt/wasi-sysroot -export CLANG_ROOT="$(echo /usr/local/opt/llvm/lib/clang/8*)" -export CLANG=/usr/local/opt/llvm/bin/clang -``` - -Finally, compile the toolchain: - -```sh -cargo build --release -``` - -## Security - -The lucet project aims to provide support for secure execution of untrusted code. Security is achieved through a combination of lucet-supplied security controls and user-supplied security controls. See [SECURITY.md](SECURITY.md) for more information on the lucet security model. - -### Reporting Security Issues - -The Lucet project team welcomes security reports and is committed to providing -prompt attention to security issues. Security issues should be reported -privately via [Fastly’s security issue reporting -process](https://www.fastly.com/security/report-security-issue). Remediation of -security vulnerabilities is prioritized. The project teams endeavors to -coordinate remediation with third-party stakeholders, and is committed to -transparency in the disclosure process. +# [Lucet documentation](https://github.com/fastly/lucet/wiki) diff --git a/lucet-idl/README.md b/lucet-idl/README.md deleted file mode 100644 index 69817edb3..000000000 --- a/lucet-idl/README.md +++ /dev/null @@ -1,129 +0,0 @@ -# Lucet-IDL - -This is an IDL. A tool that reads descriptions of data types, and spits out these types' definitions, plus a bunch of functions to represent them as a serialized, platform-independent way. - -## Usage - -```text -USAGE: - lucet-idl [FLAGS] [OPTIONS] --input - -FLAGS: - -h, --help Prints help information - -V, --version Prints version information - -z, --zero-native-pointers Do not serialize native pointers - -OPTIONS: - -b, --backend Backend, one of: c [default: c] - -i, --input Path to the input file - -t, --target Target, one of: x86, x86_64, x86_64_32, generic [default: Generic] -``` - -## The description language in one example - -```text -// Primitive types: -// `i8`, `i16`, `i32`, `i64`, `u8`, `u16`, `u32`, `u64`, `f32`, `f64`. - -// Enumerations - -enum color { - red, blue, green -} - -// Aliases - -type colour = color -type col = colour - -// Structures - -struct st { - a: i8, - b: **i32, - c: color, - self: *st -} -``` - -## Sample output - -[Output (C backend) for the example above](https://gist.github.com/jedisct1/db5f81aa5e21b280d6f0f0350215889e). - -## Generated type definitions (C) - -### Structures - -No matter what the target and backend are, the generator will attempt to align data the same way as the reference target (C on x86_64). - -Using the example above, the `st` structure above gets defined as: - -```c -struct st { - int8_t a; - uint8_t ___pad8_1[7]; - int32_t **b; - ___POINTER_PAD(8) // pad pointer `b` at offset 8 to match alignment of the reference target (8 bytes) - color /* (enum ___color) */ c; - struct st *self; - ___POINTER_PAD(24) // pad pointer `self` at offset 24 to match alignment of the reference target (8 bytes) -}; - -#define BYTES_ST 32 -``` - -Explicit padding has been added after the first element `a` so that `b` is 64-bit aligned. - -However, native pointers can be 32-bit even though we require them to be 64-bit aligned. - -The `___POINTER_PAD` macro adds extra padding after a pointer on platforms where pointers are not 64-bit long. - -The `self` pointer will always be 64-bit aligned, as the previous member `c` is 64-bit aligned, and 64-bit long. Therefore, no extra padding is added before, but optional padding is added after the pointer itself. - -For every structure, a macro representing the number of bytes required to store it is defined as `BYTES_`, such as `BYTES_ST` in the example above. - -### Enumerations - -Enumerations are stored as a signed 64-bit integer: - -```c -typedef int64_t color; // enum, should be in the [0...2] range -enum ___color { - COLOR_RED, // 0 - COLOR_BLUE, // 1 - COLOR_GREEN, // 2 -}; - -#define BYTES_COLOR 8 -``` - - -## Accessors (C) - -The generated types are designed to be directly used by applications. - -However, they can also be represented as platform-independent serialized data. - -In particular, the `st` structure above generates the following accessors: - -```c -static inline void store_st(unsigned char buf[static BYTES_ST], const struct st *v); - -static inline void load_st(struct st *v_p, const unsigned char buf[static BYTES_ST]); -``` - -On platforms that can share the same endianness and alignment rules as the target platform, -these operations translate into a single `memcpy()` call. - -On other platforms, individual values are re-aligned and byte-swapped accordingly. - -Accessors for individual values are also generated. - -Subsets of types can thus be directly loaded and modified from a serialized representation. - -## Pointers - -Pointers are not automatically dereferenced. Their value can be replaced with zeros in -serialized representations. - -The `--zero-native-pointers` command-line option enables it. diff --git a/lucet-wasi/README.md b/lucet-wasi/README.md deleted file mode 100644 index a6f31894c..000000000 --- a/lucet-wasi/README.md +++ /dev/null @@ -1,48 +0,0 @@ -# lucet-wasi - -Experimental WASI embedding for the Lucet runtime. - -## Examples - -Example WASI programs are in the [`examples`](examples) directory. - -## Supported syscalls - -We support the entire [WASI API](https://github.com/CraneStation/wasmtime/blob/master/docs/WASI-api.md), -with the exception of socket-related syscalls. These will be added when -network access is standardized. - -## Thread safety - -Lucet guests are currently single-threaded only. The WASI embedding assumes -this, and so the syscall implementations are not thread-safe. This is not a -fundamental limitation, should Lucet support multi-threaded guests in the -future. - -## TODOs - -### Introduce optional abstraction between system clocks and WASI clocks - -The current implementations of `__wasi_clock_*` delegate to the host system's -`clock_getres` and `clock_gettime`. For untrusted code, it would be useful to -limit the precision of these clocks to reduce the potential impact of side -channels. Furthermore, the `CLOCK_*_CPUTIME_ID` clocks currently give timings -for the host process, but a measure of the guest instance runtime would be -more accurate. - -### Rewrite the code that implements capability checking - -Much of this code is a direct port of the [`wasmtime` C implementation](https://github.com/CraneStation/wasmtime/tree/master/wasmtime-wasi/sandboxed-system-primitives), -and as such contains a fair amount of unsafety and low-level operations on -bytestrings and bitfields. Since this code is critical to the integrity of the -sandboxing model, we intend to rewrite this code in higher-level Rust that is -easier to test and verify. - -## Third-Party Code - -`src/wasm32.rs` is copied from -[wasmtime](https://github.com/CraneStation/wasmtime/blob/master/wasmtime-wasi/src/wasm32.rs), -along with the associated `LICENSE.wasmtime` file. - -Parts of our syscall implementations are derived from the C implementations in -`cloudabi-utils`. See `LICENSE.cloudabi-utils` for license information.