Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft of WASI support #610

Closed
wants to merge 14 commits into from
Closed

Draft of WASI support #610

wants to merge 14 commits into from

Commits on Aug 5, 2022

  1. Draft of WASI support

    This is a first pass at implementing support for [WASI](https://wasi.dev/).  It
    adds a new `TeaVMTargetType` of `WASI`, which is mostly the same as
    `WEBASSEMBLY` except it generates a module suitable for execution by a
    WASI-compatible runtime such as
    [Wasmtime](https://github.com/bytecodealliance/wasmtime).
    
    Currently supported features:
    
    - A simple allocator for supporting the component model [canonical
      ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/CanonicalABI.md)
      calls
    - CLI argument marshaling
    - Standard in/out/err
    - Filesystem operations and I/O
    - System clock
    
    See tests/wasi/src/main/java/wasi/Test.java and tests/wasi/test.sh for examples.
    
    Not yet implemented:
    
    - Networking (i.e. sockets)
    - Environment variables
    - System random numbers
    - Process exit, yield, etc.
    - [DWARF](https://dwarfstd.org/) debugging support
    - Better logging of uncaught exceptions
    
    Note that the current implementation is probably overly paranoid in that all
    pointer arguments passed to WASI functions are allocated by the aforementioned
    allocator rather than on the Java heap.  I've done this because I'm not clear on
    when TeaVM guarantees that object address will not change.  Thus I've assumed
    that they could change at any time whatsoever, which is probably not true.
    
    Also note that the filesystem implementation currently throws generic
    `IOException`s and `ErrnoException`s rather than those specified by the standard
    (e.g. `FileNotFoundException`).  Additional work will be needed to throw the
    correct exception based on the `errno` value.
    
    The main goal of this PR is to start a discussion about WASI support in TeaVM
    and answer questions such as:
    
    - Is WASI support a reasonable goal for the TeaVM project?
    - If so, is the approach of this PR roughly correct, or are significant changes needed?
    - What other features are missing to make this useful?
    
    Signed-off-by: Joel Dice <joel.dice@fermyon.com>
    dicej committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    65d5328 View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2022

  1. allocate buffers for WASI calls from Java heap

    My previous commit was based on the assumption that objects allocated on the
    Java heap could be moved by TeaVM at any time.  It turns out TeaVM will not move
    objects which are referenced by stack variables, so we can use those instead of
    `malloc`ed buffers.
    
    Signed-off-by: Joel Dice <joel.dice@fermyon.com>
    dicej committed Aug 8, 2022
    Configuration menu
    Copy the full SHA
    2f2159f View commit details
    Browse the repository at this point in the history
  2. remove WASI target type and generate WASI modules by default

    Per review feedback, this removes `TeaVMTargetType.WASI` and makes
    `TeaVMTargetType.WEBASSEMBLY` generate a WASI module unconditionally.  Such
    modules won't run in a browser without a polyfill, which I plan to add in a
    later commit.
    
    Along the way, I noticed the `Console` intrinsics weren't working properly and
    fixed them.
    
    Finally, this adds a new `WasiRunStrategy` which runs the full TeaVM test suite
    using either Wasmtime or Wasmer.  You can try it by running the following from
    the test directory:
    
    ```
    mvn -e install \
      -Dteavm.junit.js=false \
      -Dteavm.junit.js.runner=none \
      -Dteavm.junit.wasm=true \
      -Dteavm.junit.wasm.runner=wasi-wasmer
    ```
    
    Specify `-Dteavm.junit.wasm.runner=wasi-wasmtime` to use Wasmtime.
    
    Currently many of the tests are failing, but I haven't had a chance to
    investigate yet or compare the results to the same tests running in a browser
    using an unmodified TeaVM.
    
    Signed-off-by: Joel Dice <joel.dice@fermyon.com>
    dicej committed Aug 8, 2022
    Configuration menu
    Copy the full SHA
    69394e1 View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2022

  1. add tests and fix bugs related to filesystem failure cases

    Signed-off-by: Joel Dice <joel.dice@fermyon.com>
    dicej committed Aug 9, 2022
    Configuration menu
    Copy the full SHA
    1a0f2f4 View commit details
    Browse the repository at this point in the history
  2. add support for WASI environmont variables

    Signed-off-by: Joel Dice <joel.dice@fermyon.com>
    dicej committed Aug 9, 2022
    Configuration menu
    Copy the full SHA
    35b56d4 View commit details
    Browse the repository at this point in the history
  3. reimplement imported functions for Wasm target

    Previously, TeaVM used imported functions for timezone resolution, char case
    conversion, and floating point tests.  Since WASI doesn't support any of those
    things, we implement them in Java as best we can.  Floating point tests are
    easy; timezone resolution will have to wait for
    WebAssembly/WASI#467 (we hard-code UTC for now); case
    conversion can be done entirely in Java, although I've only handled ASCII
    characters in this commit.
    
    Signed-off-by: Joel Dice <joel.dice@fermyon.com>
    dicej committed Aug 9, 2022
    Configuration menu
    Copy the full SHA
    17bc74b View commit details
    Browse the repository at this point in the history

Commits on Aug 10, 2022

  1. implement WASI-based random number generation

    Signed-off-by: Joel Dice <joel.dice@fermyon.com>
    dicej committed Aug 10, 2022
    Configuration menu
    Copy the full SHA
    9cdc13b View commit details
    Browse the repository at this point in the history
  2. add timeout to WasiRunStrategy

    One of the tests had been running for 90 minutes when I gave up on it.  Now we
    time out any tests that takes longer than 5 minutes.
    
    Signed-off-by: Joel Dice <joel.dice@fermyon.com>
    dicej committed Aug 10, 2022
    Configuration menu
    Copy the full SHA
    49b9330 View commit details
    Browse the repository at this point in the history
  3. handle "/" preopened directory if present

    Signed-off-by: Joel Dice <joel.dice@fermyon.com>
    dicej committed Aug 10, 2022
    Configuration menu
    Copy the full SHA
    c6fef38 View commit details
    Browse the repository at this point in the history
  4. revert earlier stub implementations

    A previous commit provided stub implementations of `getNativeOffset`,
    `toUpperCase`, and `toLowerCase`.  This reverts that change and instead uses the
    JS versions in the browser.  It won't work in a WASI runtime without providing
    the extra imports to the module, but there's currently no way to get the local
    timezone in WASI, and I'm not yet ready to take on a proper implementation of
    `toUpperCase` and `toLowerCase`.
    
    Signed-off-by: Joel Dice <joel.dice@fermyon.com>
    dicej committed Aug 10, 2022
    Configuration menu
    Copy the full SHA
    6aff71c View commit details
    Browse the repository at this point in the history
  5. refactor WASI tests so they always use _start

    Signed-off-by: Joel Dice <joel.dice@fermyon.com>
    dicej committed Aug 10, 2022
    Configuration menu
    Copy the full SHA
    71c29b6 View commit details
    Browse the repository at this point in the history
  6. update wasm-runtime.js to use a WASI shim

    This uses https://github.com/bjorn3/browser_wasi_shim to emulate WASI features.
    Note that I had to change the API for TeaVM `main` since `browser_wasi_shim`
    requires that CLI arguments are known before the module is loaded.
    
    Signed-off-by: Joel Dice <joel.dice@fermyon.com>
    dicej committed Aug 10, 2022
    Configuration menu
    Copy the full SHA
    b3d2aeb View commit details
    Browse the repository at this point in the history

Commits on Aug 11, 2022

  1. upgrade to latest browser_wasi_shim

    Signed-off-by: Joel Dice <joel.dice@fermyon.com>
    dicej committed Aug 11, 2022
    Configuration menu
    Copy the full SHA
    d33f8d4 View commit details
    Browse the repository at this point in the history
  2. update browser_wasi_shim dep to use NPM-hosted package

    Signed-off-by: Joel Dice <joel.dice@fermyon.com>
    dicej committed Aug 11, 2022
    Configuration menu
    Copy the full SHA
    ecc35ba View commit details
    Browse the repository at this point in the history