Sync WIT with upstream specification, use, packages, and all
#867
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
alexcrichton
force-pushed
the
use-syntax-round-2
branch
from
a836039
to
af067c9
Compare
alexcrichton
added a commit
to alexcrichton/wasm-tools
that referenced
this pull request
Jan 4, 2023
This commit implements part of WebAssembly/component-model#129, specifically the portion where an `export` item now introduces a fresh item into the relevant index space. This is going to be used by bytecodealliance#867 to implement encoding/decoding of wit documents, specifically for `use` statements. This commit mostly implements the infrastructure necessary for this change rather than any other major changes internally. Functionally `export` statements now basically behave the same as `alias` statements within the validator. Eventually more validation will be required to ensure that functions use transitively-named types in all locations. Additionally the name that a type index is exported at will be relevant and will need exposing, but that'll come at a later date as well. It's hoped that exposing this information, as required for WIT encoding/decoding, will be a relatively small layer on the validator from this commit.
alexcrichton
added a commit
that referenced
this pull request
Jan 5, 2023
* Implement exports-introduce-indice semantics This commit implements part of WebAssembly/component-model#129, specifically the portion where an `export` item now introduces a fresh item into the relevant index space. This is going to be used by #867 to implement encoding/decoding of wit documents, specifically for `use` statements. This commit mostly implements the infrastructure necessary for this change rather than any other major changes internally. Functionally `export` statements now basically behave the same as `alias` statements within the validator. Eventually more validation will be required to ensure that functions use transitively-named types in all locations. Additionally the name that a type index is exported at will be relevant and will need exposing, but that'll come at a later date as well. It's hoped that exposing this information, as required for WIT encoding/decoding, will be a relatively small layer on the validator from this commit. * Update more tests and tools for types-define-aliases * Update the version number header of components This commit performs the first update of the binary header version number of the component binary encoding. The 4-byte header value, for core WebAssembly, is 0x00000001. This 4-byte header value is proposed to be reinterpreted as two 16-bit integers. The upper 16-bit integer of the 32-bit header version is reinterpreted as: * 0x0000 - this is a core module * 0x0001 - this is a component The lower 16-bits are then the version within the format. For core wasm the only supported version is 0x0001 and for components the previous supported version was 0x000a (with the intention to move to 0x0001 on stabilization). This 0xa header value has never changed throughout the lifetime of components but I would personally like to start using it for breaking changes to the binary format during development. In the spirit of that I've bumped it here to 0xb due to the change of how `export` introduces an alias. This should break almost all preexisting component binaries, indicating that a recompilation with newer tooling is required. It's hoped that the error message produced here will be more actionable than obscure indexing error messages. * Update with review comments * Update expected error message * Fix another test expectation
alexcrichton
force-pushed
the
use-syntax-round-2
branch
2 times, most recently
from
1b74337
to
c4e23ac
Compare
This commit is the first in what is likely to be a number of commits to implement the full proposal for `use` in `wit-parser` and `wit-component`. The immediate goal of this commit was to get a basic structure in place for resolution and an AST for `wit-parser` to have. To that end a new `UnresolvedPackage` container is added with a `Resolve` container which can have unresolved packages added to it. The main focus here is to get the `wit-parser` crate into a position where most of its tricky bits are implemented and out of the way. This includes resolution of multi-file packages, resolution of `use` within a package, etc. This additionally includes new features such as automatically topologically sorting everything inside containers and full elaboration of world imports/exports with. It's intended that the test suite of `wit-parser` is expanded to cover all of this new functionality, errors and success cases. To that end the tests here are: * Existing tests all become "single file WIT packages". They are packages with one document and no sibling documents. * A new test format, a directory, was added to the `tests/ui` folder which could house multi-file WIT packages which have optional package dependencies in a sub-`deps` folder. * The JSON output of a successful parse was removed due to it getting unwieldy to continue updating and in general not being too interesting. The `wit-component` crate is broken as a result of this commit, and the next step is to update that to using `Resolve` primarily.
Currently the crate compiles but most of it is commented out. This commit specifically works on updating the printing portion as well as the decoding portion from binary. Basic-to-semi-advanced binaries can be decoded and printed now. No tests are currently written. Printing is now centered around a document within a `Resolve`. The CLI print currently requires that a document be specified if there's more than one document. The plan is to also add something like `--out-dir` to print out all documents at once.
This commit implements the new `wasm-tools component wit` subcommand which is a swiss-army-knife of working with: * WIT textual documents as a single file-package * WIT textual packages as a directory * WIT packages encoded as a WebAssembly binary Inter-conversion between all of these should now be available via all the various flags and such on the CLI. This is all intended to help with debugging, inspecting, and otherwise working with the low-level parts of the component model. In other words this is trying to expose as much functionality as possible implemented as libraries through a CLI for when it's needed as a CLI is more quickly accessible than writing a custom wrapper.
alexcrichton
force-pushed
the
use-syntax-round-2
branch
from
c4e23ac
to
9157bc7
Compare
This commit implements all the necessary infrastructure to roundtrip a WIT package through the WebAssembly component binary format. The `wasm-tools component wit` subcommand is the tooling for this mode and the `tests/interfaces` directory for `wit-component` has all been updated. Tests are now either a single-file wit package or a directory which is a wit-package. Tests have a `$name.wat` assertion for the expected output and a `$name.wit.print` assertion for each document for the machine-printed WIT as-decoded-from-wasm. The `tests/interfaces` test also tests round-trip-ness to ensure that everything stays stable. I've taken a number of liberties throughout the implementation of encoding/decoding, primarily around the handling of URLs. Additionally much of the encoding code related to creating an actual component from a core wasm module is commented out here since I'll handle that in subsequent commits. Tests were all updated here to the new syntax and additionally all of the examples from the upstream component-model PR have been added here as tests.
alexcrichton
force-pushed
the
use-syntax-round-2
branch
from
9157bc7
to
f1e8523
Compare
This commit gets the CLI command for `wasm-tools component new` working again. In doing this a number of changes were made to the CLI: * The `--types-only` flag was dropped as this is entirely subsumed, or will be, by `wasm-tools component wit` * The `--wit` and `--encoding` flags were dropped. These were moved to a separate `wasm-tools component embed` subcommand. These aren't too useful in general and are primarily intended for developers working on raw internals. They also got more complicated with the current structure of WIT as a WIT package is now being specified with the `*.wit` file and then within that package a world needs to be selected. This logic all now lives in a separate subcommand The new `embed` subcommand is fully implemented here and its internals will be used for the eventual `components.rs` test to come in a bit.
alexcrichton
force-pushed
the
use-syntax-round-2
branch
2 times, most recently
from
3fe1513
to
bf5cc6b
Compare
|
Ok all the tests pass, phew! Pieces remaining are:
at the very least though progress is being made! |
alexcrichton
added a commit
to alexcrichton/component-model
that referenced
this pull request
Jan 10, 2023
Reflects the progress made on bytecodealliance/wasm-tools#867
This gets the rest of `wit-component` compiling again, updating it to the new AST representation inside of the `wit-parser` crate.
alexcrichton
force-pushed
the
use-syntax-round-2
branch
from
783d44c
to
485c6e9
Compare
This commit updates the WIT decoding process to not actually rely on the structure of `pkg:/...` URLs at all, instead purely maintaining a hash map of them as opaque identifiers. The purpose of this is to hopefully future-proof against changes in the URL field which only requires that they be unique. This still relies on the URL structure, however, for foreign dependencies. I'm not sure how to get around that at this time because there's no other way to organize just imported interfaces back into packages and documents.
|
Thanks for the thorough review! I'm putting the finishing touches on the fuzzer but have otherwise pushed up addressed feedback plus bugs the fuzzer has found so far. |
This'll help any embedders iterating over the resulting documents to always iterate in a topological order, as is typically expected.
Previously the exported interface was accidentally not filled in. This commit fills it in in the same manner as imports
This commit updates how foreign deps are resolved during `Resolve::push` to remove the restriction that the `unresolved.foreign_deps` field lists documents in the same order as they are within the main arena. This will be difficult to maintain during decoding and otherwise isn't all that necessary. Additionally this commit adds further asserts that an unresolved package's items are indeed paritioned into remote and local parts since much of the indexing code here relies on that.
Fix a case in decoding where a type reexport accidentally redefined the type.
This commit is an overhaul and reimplementation for the `roundtrip-wit` fuzzer in the `wit-component` package. This updates the AST generation to instead generate a textual format with a string which is then parsed. Various properties about round-tripping are then tested as well. about this document. Additionally this lifts the fuzzer into the top-level directory to run on oss-fuzz since this has run long enough locally without bugs and I think it would be good to suss out bugs sooner rather than later via oss-fuzz.
|
Ok I've run the fuzzer long enough locally that the bugs are becoming more obscure that I think it's time to get this in. @peterhuene mind double-checking the commits I added since your last review? Thanks again for taking the time to go through all of this! |
Don't let the fuzzer go wild and blow the hardcoded limits in the wasmparser crate just yet.
Previously the toposort of doc deps would be different because printing would list imports before exports but visitation would visit them in the textual order. Now the print order is the same as the visit order which should preserve the order of deps within the AST.
Don't let the tuple/option/etc types subvert the size bound accidentally.
peterhuene
approved these changes
Jan 17, 2023
This was referenced Jan 17, 2023
alexcrichton
added a commit
to bytecodealliance/wasmtime
that referenced
this pull request
Jan 18, 2023
* Update WIT tooling used by Wasmtime This commit updates the WIT tooling, namely the wasm-tools family of crates, with recent updates. Notably: * bytecodealliance/wasm-tools#867 * bytecodealliance/wasm-tools#871 This updates index spaces in components and additionally bumps the minimum required version of the component binary format to be consumed by Wasmtime (because of the index space changes). Additionally WIT tooling now fully supports `use`. Note that WIT tooling doesn't, at this time, fully support packages and depending on remotely defined WIT packages. Currently WIT still needs to be vendored in the project. It's hoped that future work with `cargo component` and possible integration here could make the story about depending on remotely-defined WIT more ergonomic and streamlined. * Fix `bindgen!` codegen tests * Add a test for `use` paths an implement support * Update to crates.io versions of wasm-tools * Uncomment codegen tests
juntyr
added a commit
to juntyr/serde-wit
that referenced
this pull request
Jan 21, 2023
alexcrichton
added a commit
to alexcrichton/preview1.wasm
that referenced
this pull request
Jan 23, 2023
This commit updates the `wasmtime` dependency and the `wit-bindgen-guest-rust` dependencies to namely update the `wit-parser` and component underlying implementations. This pulls in bytecodealliance/wasm-tools#867 which is the implementation of `use` for WIT. This does not currently break apart the one large `wasi.wit` file, instead just gets tests working.
alexcrichton
added a commit
to alexcrichton/preview1.wasm
that referenced
this pull request
Jan 24, 2023
This commit updates the `wasmtime` dependency and the `wit-bindgen-guest-rust` dependencies to namely update the `wit-parser` and component underlying implementations. This pulls in bytecodealliance/wasm-tools#867 which is the implementation of `use` for WIT. This does not currently break apart the one large `wasi.wit` file, instead just gets tests working.
sunfishcode
pushed a commit
to bytecodealliance/preview2-prototyping
that referenced
this pull request
Jan 25, 2023
* Update wasmtime and wit-bindgen This commit updates the `wasmtime` dependency and the `wit-bindgen-guest-rust` dependencies to namely update the `wit-parser` and component underlying implementations. This pulls in bytecodealliance/wasm-tools#867 which is the implementation of `use` for WIT. This does not currently break apart the one large `wasi.wit` file, instead just gets tests working. * Split apart `wasi.wit` into multiple `*.wit` files This commit refactors the monolithic `wasi.wit` file into multiple files. I've taken some liberties in naming here so suggestions are definitely welcome! I've resolved some TODO annotations as well about `use`-ing types between interfaces. There are two issues remaining, however: * The `wasi-command` world still hardcodes `u32` because `world` items don't support `use` just yet. That isn't a hard limitation of WIT, however, it's just a temporary limitation of the implementation. * The `wasi-clocks` interface defines the `wasi-future` type. This is due to a cyclic dependency where `wasi-future` is defined within `wasi-poll` which depends on `wasi-clocks` for types. I've left this to a future refactoring to probably have a `types` interface of some form somewhere. * Update patch for wasm-tools * Switch to wasmtime upstream * Update CI for build * Remove patch overrides * Update names of uploaded files
pchickey
pushed a commit
to pchickey/wasmtime
that referenced
this pull request
May 12, 2023
* Update wasmtime and wit-bindgen This commit updates the `wasmtime` dependency and the `wit-bindgen-guest-rust` dependencies to namely update the `wit-parser` and component underlying implementations. This pulls in bytecodealliance/wasm-tools#867 which is the implementation of `use` for WIT. This does not currently break apart the one large `wasi.wit` file, instead just gets tests working. * Split apart `wasi.wit` into multiple `*.wit` files This commit refactors the monolithic `wasi.wit` file into multiple files. I've taken some liberties in naming here so suggestions are definitely welcome! I've resolved some TODO annotations as well about `use`-ing types between interfaces. There are two issues remaining, however: * The `wasi-command` world still hardcodes `u32` because `world` items don't support `use` just yet. That isn't a hard limitation of WIT, however, it's just a temporary limitation of the implementation. * The `wasi-clocks` interface defines the `wasi-future` type. This is due to a cyclic dependency where `wasi-future` is defined within `wasi-poll` which depends on `wasi-clocks` for types. I've left this to a future refactoring to probably have a `types` interface of some form somewhere. * Update patch for wasm-tools * Switch to wasmtime upstream * Update CI for build * Remove patch overrides * Update names of uploaded files
github-merge-queue bot
pushed a commit
to bytecodealliance/wasmtime
that referenced
this pull request
May 15, 2023
…ime (#6374) * An initial preview1 polyfill stub. * Add test on CI and infrastructure for wasm globals (#1) * Updates * Add a verification test to CI * Produce a global-referencing object on stable * Restrict CI again * Add a comment about a disabled lint. * Add an `extern "C"` to the Rust code comment. * Use `*mut u8` instead of `usize` to hold pointers. This will reduce casting, and better preserve pointer provenance. * Mark the WASI functions `unsafe`. * Fix the inline asm comment to match the code. * Implement a polyfill for `path_readlink`. Implement a polyfill for `path_readlink` in terms of the preview2 `readlink_at` function. This involves adding a second wasm global, to hold the buffer size. * Use raw_strings, disable integer overflow checks, and suppress jump tables. This also updates the verifier to accept the new wasi imports. * Use a compiler_fence. * Add a slow-path to handle smaller buffers passed to `path_readlink`. * These `forget`s aren't needed because they just have references. * Properly forget returned buffers. * Remove the code that disabled the global allocator. * Mark `path_readlink_slow` as inline(never). * Minor cleanups. * Remove `unregister_buffer`. When `cabi_realloc` uses the buffer pointer, it sets the stored buffer pointer to null, so we don't need a separate `unregister_buffer` call. * Implement file descriptors. * Implement more stuff. * Implement more functions. * Fixes to avoid static inits. * Add a file descriptor bounds check. * Simplify a few casts. * Multiply memory.grow's result by the page size, and handle failure. * Remove an unneeded `register_buffer`. * Convert to use `u32` indices instead of handles. This is a temporary experiment. * fd_info fixup * switch to latest wit_bindgen_guest_rust macro * notes for what to do next * adapter: take args, have two cabi_{import,export}_realloc funcs, and call adaptee start * skip codegen for command, and fill in the entrypoint manually * cabi_export_realloc is another bad allocator for our bad allocator museum * Implement polyfills for several wasi-filesystem functions Implement fd_advise, fd_filestat_set_times, path_filestat_set_times, and path_remove_directory. * Implement fd_filestat_set_size, fd_pread, and fd_pwrite. * Implement path_create_directory, path_filestat_get, path_link. * Implement path_rename, path_symlink, and path_unlink_file. * Swap `info` and `fd-info` See the last commit in WebAssembly/wasi-filesystem#66 for details. * Implement `fd_fdstat_get` and `fd_fdstat_set_flags`. This incoporates several wasi-filesystem repo changes too. * Implement `clock_{time,res}_get` (#12) Currently traps for the `*_CPUTIME_*` clocks and doesn't check for overflow on nanoseconds. * Update Wasmtime/tooling dependencies (#14) This forces all the `*.wit.md` file to go into one large `*.wit` file for now which will get split up later once `use` is re-introduced. * Simplify global state management in adapter (#13) * Simplify global state management in adapter I was looking recently to implement args-related syscalls but that would require yet-more globals and yet-more state to be managed. Instead of adding a slew of new globals for this which are all manually kept in sync I opted to instead redesign how global state is managed in the adapter. The previous multiple `global`s are all removed in favor of just one, as sort of a "tls slot" of sorts. This one remaining slot points to a one-time-allocated `State` object which internally stores information like buffer metadata, fd information, etc. Along the way I've also simplified syscalls with new methods and `?`-using closures. * Turn off incremental for dev builds Helps with CGU splitting and ensuring that appropriate code is produced even without `--release`. * Review comments * Add accessors with specific errors * Update handling of `*_global_ptr` * Update internal mutability around path buffer Use an `UnsafeCell` layering to indicate that mutation may happen through `&T`. * Implement `args_{get,sizes_get}` functions (#16) This commit updates the implementation of `cabi_export_realloc` to allocate from a bump-allocated-region in `State` rather than allocating a separate page for each argument as previously done. Additionally the argument data is now stored within `State` as well enabling a full implementation of the `args_get` and `args_sizes_get` syscalls. * sketch of `poll_oneoff` polyfill (#11) This adds a `poll_oneoff` implementation to src/lib.rs. It's completely untested. I was planning on adding a host implementation and using that to test end-to-end, but that raised some tough questions about how much of the existing `wasi-common` scheduler(s) should be reused. I've decided to focus on other, more widely-used parts of WASI first, but I wanted to share the work I've already done here. Note that I've moved the clock- and socket-specific functions out of `wasi-poll` and into `wasi-clocks` and `wasi-tcp`, respectively. The latter is a new interface which will eventually contain functions and types resembling @npmccallum's https://github.com/npmccallum/wasi-snapshot-preview2#wasi-tcp proposal. Per discussion with @sunfishcode: - `wasi-tcp` includes an `error` enum type, intended to represent only socket-related errors. - It also includes a `socket` pseudo-handle type, distinct from `wasi-filesystem`'s `descriptor` type. These fine-grained types help move us away from the "everything is a file descriptor" and "all errors are errnos" approaches of Preview 1. If we decide `poll-oneoff` should be usable with files as well as sockets, we can add `subscribe-read` and `subscribe-write` functions to `wasi-filesystem` which accept file `descriptor`s. Likewise for any other pseudo-handle type from which we'd like to create futures. Signed-off-by: Joel Dice <joel.dice@fermyon.com> Signed-off-by: Joel Dice <joel.dice@fermyon.com> * Update a few aspects of the adapter build (#17) * Update a few aspects of the adapter build * Use the `wasm32-unknown-unknown` target for the adapter and specify flags in `.cargo/config.toml` to avoid having to pass the same flags everywhere. This allows using `wasm32-wasi` for tests to ensure the flags only apply to the adapter. * Use `opt-level=s` since speed is not of the utmost concern for this wasm but since it's likely to be included in many places size is likely more important. * Use `strip = 'debuginfo'` for the release build to remove the standard library's debugging information which isn't necessary. * Remove `debug = 0` from the `dev` profile to have debugging information for development. * Add a small `README.md` describing what's here for now. * Move `command` support behind a `command` feature This commit adds a `command` feature to main crate to avoid importing the `_start` function when the `command` feature is disabled, making this adapter useful for non-command WASI programs as well. For now this still emits the `command` export in the final component but with `use` in `*.wit` files it should be easier to avoid that export. * Implement `path_open`. (#18) * Implement `path_open`. * Implement `fd_close`, and finish `path_open`. * Stub out `close` in the host impl. * Add a `badf` error code. This pulls in WebAssembly/wasi-filesystem#62. * Fix prints in non-command builds (#19) * Fix prints in non-command builds When the `command` entrypoint isn't invoked, such as for non-command builds, then prints were not working as they would return `ERRNO_BADF` which is swallowed by at least Rust right now. This instead sets the initialization of `State` to reflect how fd 0 is an empty stdin stream, fd 1 goes to a `log` call, and fd 2 also goes to `log`. During `command` initialization fds 0 and 1 are overwritten with the provided input and for non-command builds this is the state permanently for the program. It's possible we can add further configuration hooks for this in the future, but this should get at least the initial state of non-command builds more workable. * Use BADF for reading log fds * Implement the `fd_readdir` function (#23) * Implement the `fd_readdir` function This roughly matches the implementation in `wasi-common` today where it uses the directory entry stream as a form of iterator and the `cookie` represents the `n`th iteration. Not exactly efficient if the `buf` provided to the hostcall is too small to hold many of the entries but there's not a whole lot else that can be done at this time. This also updates the WIT for `read-dir-entry` to return `option<dir-entry>` instead of `dir-entry` to indicate EOF. * Fix host compile * Add a specific method to close a dir-entry-stream * Add a cache to avoid quadratic dirent behavior When a readdir call is truncated due to the output buffer not being large enough save the final state of the readdir iterator into `State` to possibly get reused on the next call to `readdir`. Some limits are put in place such as: * The next call to readdir must be for the same fd and the required cookie. * The dirent that didn't fit must have its full name fit within the path cache. * If `fd_close` is called after a readdir it clears the cache so a future `fd_readdir` doesn't actually resume now-stale state. There's a fair bit of trickiness here so I've attempted to structure things as "simply" as possible to hopefully reduce the chance there's an issue, but this is all untested so there's still likely an off-by-one or similar bug. * Implement `fd_renumber`. (#21) * Implement `fd_renumber`. * Implement `Drop` for `Descriptor` to free backend resources. Instead of trying to remember to call `close` on file descriptors when they're replaced or removed, just have the `Drop` impl for `Descriptor`. * Use the local `unwrap_result` instead of `.unwrap()` This avoids static memory inits. * Don't print extra metadata in stdout/stderr, and implement proc_exit (#27) Add an API to implement proc_exit with, and implement it. * add tests for `wasi-random`, `wasi-clocks`, and stdin (#30) This required fleshing out the `wasi-clocks` host implementation a bit and adding a `read_vectored_at` implementation for `ReadPipe`. Signed-off-by: Joel Dice <joel.dice@fermyon.com> Signed-off-by: Joel Dice <joel.dice@fermyon.com> * Add a way to get a random `u64`. (#36) This anticipates WebAssembly/wasi-random#18. * Fix `fd_renumber` to handle output fds that aren't previously allocated. (#38) This fixes the crates/test-programs/wasi-tests/src/bin/stdio.rs test in the Wasmtime testsuite. * add support for environment variables and preopens Per #31, we pass env vars and preopens via `command`, just like CLI arguments. Later, we plan to add another interface (e.g. `cli-reactor`) for reactors which need env vars and/or preopens, which will have a single function `initialize` that takes the same two parameters (but not stdio or CLI arg parameters). This also removes unused parts of `wasi-common` (e.g. args, and env vars, and preopen paths) which have been superceded by the `command` interface. Our eventual goal is to provide a more explicit interface for environment variables using WIT star imports such that the guest will import a zero-argument function (or value, when that is supported) for each variable it needs, allowing the host to statically verify that it can provide all those variables. However, star imports are not yet supported in `wit-bindgen`, and once they are, we'll likely still need a dynamic interface to support existing software. Note that I've added a `file_read` test which does not actually do anything yet, since not all the required host functions have been implemented. I plan to address that soon. Signed-off-by: Joel Dice <joel.dice@fermyon.com> * implement more host filesystem functions The `file_read` test now passes. Signed-off-by: Joel Dice <joel.dice@fermyon.com> * Add pseudo-streams. (#29) * Add pseudo-streams. This add a pseudo-stream type to the wasi-poll interface, and adds ways to obtain streams from command invocation and from files. In the future, it can support sockets too. With this, `command` takes streams for stdin/stdout, rather than filesystem descriptors. Streams support reading and writing, as well as skipping, repeated-element writing, and splicing from one stream to another. And there are `subscribe-*` functions to produce pseudo-futures from pseudo-streams, allowing them to be polled. This makes the polyfill somewhat more complex, but this is largely due to the polyfill being tied to the preview1 API. This replaces the `seek` and `tell` functions, and implemented `fd_seek` and `fd_tell` in terms of the polyfill's own position. Also, add a dedicated stderr API for writing to stderr in a way that tolerates strings that aren't necessarily expected to be newlines. And add a way to test whether stderr is a terminal. * Implement the host side of `poll_oneoff`. This implements pseudo-futures and subscription functions, and adds polling for streams. * Implement clock subscriptions. wasi.wit: - Remove the "timers" API from wasi-clocks, as it's now redundant with pseudo-future clock subscriptions. - Remove `subscribe-wall-clock`. Wall-clock timeouts were implemented by converting them to monotonic-clock timeouts anyway, so just make that explicit in the WASI API, and teach the polyfill how to convert wall-clock timeouts into monotonic-clock timeouts. - Move `subscribe-monotonic-clock` out of wasi-clocks and into wasi-poll, as it's closely tied to the pseudo-futures mechanism and the `poll-oneoff` implementation. - While here, fix `stream-read` and related functions to return an end-of-stream/file indicator. Code changes: - `default_wall_clock()` and `default_monotonic_clock()` now always create a new table entry, rather than holding a table index in the `WasiCtx` which could potentially dangle. - Add support for monotonic-clock poll subscriptions. - Say "wall clock" instead of "system clock" when we have a choice. * Remove the `OFlags::APPEND` flag, which is no longer used. * Implement `command` exit statuses. (#40) Add a `result` return type to `command` so that it can indicate success or failure. The idea here is that this isn't a full `i32` return value because the meaning of return values isn't portable across platforms. Also, Typed Main is a better long-term answer for users that want rich error return values from commands. * implement `wasi-filesystem::readdir` and related functions (#45) * implement `wasi-filesystem::readdir` and related functions This adds a `directory_list` test and provides the required host implementation. I've also added a file length check to the `file_read` test, just to cover a bit more of the API. Signed-off-by: Joel Dice <joel.dice@fermyon.com> * fix memory corruption in `fd_readdir` polyfill We were copying `name.len() * 256` bytes instead of just `name.len()` bytes, which was overwriting other parts of `State` and causing untold havoc. Signed-off-by: Joel Dice <joel.dice@fermyon.com> * check type of entry in `Table::delete` Signed-off-by: Joel Dice <joel.dice@fermyon.com> Signed-off-by: Joel Dice <joel.dice@fermyon.com> * Minor cleanups (#49) * Use `use` imports consistently. * Rename `flags_from_descriptor_flags` to `descriptor_flags_from_flags`. * Rename `black_box` to `obscure`. * Make some comments be doc comments. * Use a hyphen for compound adjectives. * Delete an unused variable. * Update the name of `change-file-permissions-at`. * Use generated typedefs instead of hard-coding u64. * Use core::hint::black_box now that it's stabilized. * Try to detect memory corruption with "magic" canaries (#52) After two separate rounds of memory corruption now happening it's probably best to place some protections in place to try to detect when this happens earlier rather than much later after the fact. * Implement file append functionality. (#50) * Implement file append functionality. - In the preview1-to-preview2 polyfill, using `append_via_stream`. - In the host implementation, using a new system-interface `FileIoExt::append` function. * Add a basic testcase. * Fix some bugs turned up in the Wasmtime testsuite. * Allocate bool results at the end of the events buffer. * Don't fail poll for per-fd failures. When a fd is not pollable, report it as an immediately-ready future, rather than as a poll failure. * Implement `sync` and `datasync` on files and directories. On Windows, there doesn't appear to be a way to sync a directory, to ensure that the directory entry for a file is sync'd. So for now, just silently succeed. I've opened WebAssembly/wasi-filesystem#79 to track this at the spec level. * Add `eprintln`, `unreachable`, and other macros. (#62) Add a `byte-array` proc-macro crate for converting strings into byte arrays that don't use static initializers, and use it to implement `eprintln`, an `unreachable` that prints the line number, and other macros. * Use the `wasi-stderr` API for stderr (#59) Use the dedicated `wasi-stderr` API for stderr, instead of `wasi-logging`. Fixes #57. * Change the filesystem timestamp to seconds+nanoseconds. (#55) This corresponds to WebAssembly/wasi-filesystem#76. * Update wasmtime and wit-bindgen (#68) * Update wasmtime and wit-bindgen This commit updates the `wasmtime` dependency and the `wit-bindgen-guest-rust` dependencies to namely update the `wit-parser` and component underlying implementations. This pulls in bytecodealliance/wasm-tools#867 which is the implementation of `use` for WIT. This does not currently break apart the one large `wasi.wit` file, instead just gets tests working. * Split apart `wasi.wit` into multiple `*.wit` files This commit refactors the monolithic `wasi.wit` file into multiple files. I've taken some liberties in naming here so suggestions are definitely welcome! I've resolved some TODO annotations as well about `use`-ing types between interfaces. There are two issues remaining, however: * The `wasi-command` world still hardcodes `u32` because `world` items don't support `use` just yet. That isn't a hard limitation of WIT, however, it's just a temporary limitation of the implementation. * The `wasi-clocks` interface defines the `wasi-future` type. This is due to a cyclic dependency where `wasi-future` is defined within `wasi-poll` which depends on `wasi-clocks` for types. I've left this to a future refactoring to probably have a `types` interface of some form somewhere. * Update patch for wasm-tools * Switch to wasmtime upstream * Update CI for build * Remove patch overrides * Update names of uploaded files * Use a more targeted means of specifying link flags (#71) * Use a more targeted means of specifying link flags I had forgotten earlier that this could be done with build scripts so do that in the adapter's build script rather than as auxiliary rust flags. * Remove `.cargo/config.toml` file This now only serves the purpose to enable bulk-memory which is relatively minor. This removes the file for now and wasm features can always be reenabled at a later date if file size is truly an issue. * Fix the non-command build (#72) This commit fixes the non-command build of the WASI adapter by adding a new `wasi.wit` which only has the imports and nothing else. * Rename `wasi-future` to `waitable`. (#70) * Rename `wasi-future` to `waitable`. You can wait on `waitable`s multiple times, which differs from proper futures, which you can only wait on once. * Rename `waitable` to `pollable`. Because the function they're passed to is `poll-oneoff`. * use main module's `cabi_realloc` instead of `memory.grow` This depends on bytecodealliance/wasm-tools#900, which will polyfill an implementation if the main module does not export it. Signed-off-by: Joel Dice <joel.dice@fermyon.com> * Add basic TCP socket APIs. (#33) * Add basic TCP socket APIs. This is based on @npmccullum's [wasi-snapshot-preview2 draft] which is in turn based on the [wasi-sockets proposal], though for simplify for now it omits UDP sockets and some other features. It's also based on the [pseudo-streams PR]; so that should proceed first before this. [draft wasi-snapshot-preview2]: https://github.com/npmccallum/wasi-snapshot-preview2 [wasi-sockets proposal]: https://github.com/WebAssembly/wasi-sockets [pseudo-streams PR]: bytecodealliance/preview2-prototyping#29 * Add the preview1 `sock_accept` function declaration to the polyfill. (#77) * Split wasi-stream into separate input and output stream types. (#73) * Split wasi-stream into separate input and output stream types. This syncs the prototype's streams with the upstream wasi-io repo. Streams are unidirectional, so this allows us to statically describe whether something is an input stream or an output stream in an interface. This differs a little from the component model async streams, which don't have separate input and output streams, but it does partially reflect how the component model async design differentiates between input streams in type signatures, which are passed in as arguments, and output streams, which are returned as return values. * Fix compilation on Windows. * polyfill: fill in fdflags_append from File::append * short-circuit reentrance when allocating stack and `State` (#83) * short-circuit reentrance when allocating stack and `State` Per bytecodealliance/wasm-tools#919, `wit-component` needs to lazily allocate the adapter stack to avoid premature or infinite reentrance from the main module to the adapter. This means adding an additional `allocation_state` global variable and using it to determine when to return early from an exported function, e.g. because we're in the process of either allocating the stack or allocating `State`. This requires an updated `wit-component` dependency once bytecodealliance/wasm-tools#919 has been merged. Fixes #78 Signed-off-by: Joel Dice <joel.dice@fermyon.com> * remove redundant unsafe blocks Signed-off-by: Joel Dice <joel.dice@fermyon.com> * update dependencies This brings us up-to-date with wasi-tools, wit-bindgen, and the latest component ABI. Signed-off-by: Joel Dice <joel.dice@fermyon.com> --------- Signed-off-by: Joel Dice <joel.dice@fermyon.com> * Move to crates.io/releases of dependencies (#84) * Use precompiled `wit-bindgen` binaries from CI * Use a release branch of Wasmtime for 6.0.0 * Use `wit-bindgen`-the-crate from crates.io * replace `fn unwrap` and `fn unwrap_result` with postfix `.trapping_unwrap()` (#86) this is just aesthetic, but prefixed unwraps are a lot harder on the eyes than postfixed * Update wasi-filesystem, wasi-clocks, wasi-io, and wasi-poll. This updates to the latest versions of the wit files in the proposal repos, and updates the polyfill and host implementations accordingly. The changes here are just some renames and reorganizations. * Refactor adapter's allocators (#88) * create a reusable BumpArena, rename command_data to long_lived * use ImportAlloc::with_buffer to register a bump arena for each import func call * import alloc just hands out buffer, nothing more only improvement over prior implementation is this one makes sure alignment is followed * make merging easier... * bumparena is now a singleton so it can own the memory area * get environment and preopens through import functions * add arena allocator for get_environment and get_preopens and fix other bugs in those implementations * comments * Make Preview2's directory iterator omit `.` and `..`. (#89) Preview1's `fd_readdir` includes `.` and `..`, but Preview2 is changing to avoid this. Update the Preview2 host implementation to omit these entries, and add code the polyfill to re-add them. * Fix timestamp-to-nanosecond conversion in the adapter (#93) The order of operations for handling the seconds/nanoseconds part of the structure were a bit off which meant that a nonzero nanosecond field would have an unexpected effect on the result. Closes #92 * short-circuit `fd_write` in the adapter @guybedford was getting assertion errors in `State::new` due to the debugging print statements he had added to his `cabi_realloc` function, which called back into `fd_write`, leading to a circular dependency where `fd_write` needed to allocate, which caused `fd_write` to be called, etc. The fix is easy enough: short circuit this circularity the same way we're handling it in `environ_sizes_get`, `clock_time_get`, etc. If we wanted to be really paranoid, we could add this short circuit to all the functions, but I'd be really surprised if `cabi_realloc` needs to read from the filesystem, create directories, or other such things. If we see that in the wild, we can revisit. Signed-off-by: Joel Dice <joel.dice@fermyon.com> * Rebase on the new wasi-sockets. (#91) * Rebase on the new wasi-sockets. This switches to using the wasi-sockets wit files from WebAssembly/wasi-sockets#16. Many things are still stubbed out with `todo!()` for now. * Fix compilation on Windows. * Rebase on the wasi-cli world. (#100) * Rebase on the wasi-cli world. Rebase on the wasi-cli wit files. This incorporates some major renamings and reorganizations, and updates to use wasi-sockets from the repo rather than our old prototype files. At present this depends on several bugfixes in wasmtime and wit-bindgen so this is using git dependencies for now. This also temporarily preserves the dedicated stderr interface, which is useful for debugging the file descriptor table code, which needs to work before the regular stderr works. * Update the verify program for the new module names. * Disable debug-assertions in the adapter build. This is roughly the same as, the previous `unchecked` option in the bindings, with the changes in the latest wit-bindgen. * Update CI to use the new file names. * code motion: cli related feature names (#104) * rename host runtime tests to command tests * add a test for using wasi in a reactor * commit cargo.lock * reactor tests: fix wit deps * test-programs: reactor-tests can build on stable note that this fails because it exposes wasi-libc ctors calling import functions from inside cabi_realloc * test-programs: show that ctors fix in wit-bindgen fixes bug * ci: install wasm32 targets for stable as well as nightly * wit-bindgen: use 0.4.0 * ci: use wit-bindgen 0.4.0 * Co-habitate with wasi-common from wasmtime * adapter: code motion in cargo feature & artifact names to cli-command, cli-reactor there will shortly be a third type of reactor (non-cli, idk what to call it) --------- Co-authored-by: Trevor Elliott <telliott@fastly.com> * Update to the latest wit files. This updates to the latest wasi-filesystem, wasi-sockets, wasi-io, and wasi-cli wit changes, except for two things: - `filesystem.types` is temporarily still named `filesystem.filesystem`, to work around #5961 - wasi-stderr is temporarily still present, for debugging * Support command-line argument preopens. * Minor code simplification. * drop cli- prefix from reactor, command features & artifact names * intrinsics: just abstract the archive syms a little bit * intrinsics: add stderr stream to globals, plus getter/setter funcs * wire up adapter macros to stderr stream print * adapter: make it a little clearer where unreachable is coming from * host and adapter: provide stdio via preopens, dont pass stdio or arg preopens to main stdio never gets initialized for the reactor, tha * get rid of arg_preopens from state, env_preopens are just preopens * factor descriptors out into a separate module i didnt run verify step-by-step during this so i accidentally got panick machinery in here. thats what i get * fix macros * fix * renumber: allow overwriting. note it may invalidate preopen table * poll_oneoff: use import allocator solely on the import func call this is required because calling `state.descriptors()` could end up initializing the descriptor table, which also uses the import allocator. this is a huge whitespace change, but effectively its just a much smaller body in the closure. * Update build.rs Co-authored-by: Joel Dice <joel.dice@fermyon.com> * Descriptors is initialized on creation, use refcell<option<descriptors>> to represent uninit * adapter: arguments are lazily initialized in State instead of passed to main * rename command's `main` to `run` (unfortunately) this is an unfortunate problem with building the adapter: for some reason, wasm-ld will modify an export named `main` with no parameters by renaming it __original_main, and exporting a `main` with two i32 parameters. we should see if this behavior is fixable, this is hopefully a temporary change because I really liked that commands could just be invoked by calling `main`... * Remove resource abstraction from clocks (#129) * wit: remove resource index from monotonic-clocks and wall-clocks, delete instance-*-clock interfaces At this time, we don't have use cases for providing multiple wall clocks or multiple monotonic clocks to the same component. So, according to the principle of following preview 1's design as close as possible, we are changing these interfaces to being provided by ambient functions, rather than methods on a resource. * adapter: unresourcify clocks * host: de-resourcify clocks * Rename the command entrypoint from `run` back to `main`. (#131) LLVM has special handling for functions named `main`, which we need to avoid because we're creating the component-level `main` rather than the C-level `main`. To do this, write the `main` function in assembly, which is fortunately very simple now. * fix wasi renumber test: return an error in adapter if closing a closed fd * NFC: pull eventtype const definitions out to be reused further down * size assertion: refactor to de-duplicate MAX_DESCRIPTORS definition * File: we need to have the DescriptorType around to handle certain corner cases * fd_seek: return the expected errno when trying to seek on a directory. * fix fd_filestat_get: adapter special-case StreamType::Unknown (#139) giving an empty Filestat, instead of returning an error. This implements the behavior expected by the fd_filestat_get test. We might end up being able to rename StreamType's Unknown variant to Stdio, since it looks like that is the only place we construct it. * return ERRNO_BADF on directory descriptors in appropriate fd operations * dead code * adapter: fd_allocate gives notsup on all files * adapter: remove incorrect assertions from path_readlink * readlink: fix adapter to return right len, and test to expect truncation truncation behavior is backported to upstream in #6225 * adapter: track blocking state in File, remove uses of set-flags * adapter: change logic around interpeting ATIM and ATIM_NOW (and MTIM equiv) to be consistient with upstream in wasi-common preview1, ATIM and ATIM_NOW (and MTIM and MTIM now) were mutually exclusive and would result in an INVAL error, whereas in the adapter previously, ATIM_NOW implied ATIM, but would silently do nothing if ATIM was not set. I decided to be consistient with the upstream behavior here because it is pretty arbitrary and I don't think there's a good reason to break compatibility. This fixes the `path_filestat` test. * fd_seek: error with invalid offsets in the adapter * NFC, looks more consistient to my eye though * fix bug in adapter fd_fdstat_set_flags to frob the append flag. this fixes the fd_flags_set test. * delete spurrious extra line in adapter (this same var is created below) * adapter poll_oneoff: when descriptors.get_*_stream(fd) fails, die with that error (#154) * adapter poll_oneoff: when descriptors.get_*_stream(fd) fails, die with that error There was a special case in poll_oneoff that put in a fake clock stream when a read/write stream for a descriptor wasn't available. The poll_oneoff_files test (in `test_fd_readwrite_invalid_fd()`) checks that poll_oneoff returns a BADF when an invalid fd is subscribed to. I'm not sure what the special case was patching over, but this passes all of the other tests right now. * poll_oneoff_files fails on windows with god knows what error diff --git a/host/tests/command.rs b/host/tests/command.rs index 7af7bd0..67c8c0b 100644 --- a/host/tests/command.rs +++ b/host/tests/command.rs @@ -466,10 +466,11 @@ async fn run_path_symlink_trailing_slashes(store: Store<WasiCtx>, wasi: Command) } async fn run_poll_oneoff_files(store: Store<WasiCtx>, wasi: Command) -> Result<()> { - // trapping upwrap in poll_oneoff in adapter. - // maybe this is related to the "if fd isnt a stream, request a pollable which completes - // immediately so itll immediately fail" behavior, which i think breaks internal invariant... - run_with_temp_dir(store, wasi).await + if cfg!(windows) { + expect_fail(run_with_temp_dir(store, wasi).await) + } else { + run_with_temp_dir(store, wasi).await + } } async fn run_poll_oneoff_stdio(store: Store<WasiCtx>, wasi: Command) -> Result<()> { * adapter: path_ apis return NOTDIR instead of BADF when passed file; fixes path_open_dirfd test. (#152) * adapter: special case for NOTDIR instead of BADF fixes path_open_dirfd test. * get_dir now fails with NOTDIR for non-dir files * get rid of special case * complete test coverage for all path_ functions giving NOTDIR errors * adapter: StreamType::Unknown is actually Stdio (#161) We have only ever used Unknown for the stdio streams, and I don't expect us to use it for anything else in the future, so rename it. Set the returned filetype to character device: Closes #146. Also, fix some warnings. * Revert #131, renaming `main` back to `run`. (#165) Changing LLVM and/or Rust to avoid special handling of `main` is a fair amount of work, and there could be other toolchains with similar special rules for functions named `main`, so rename the command entrypoint back to `run`. We could potentially re-evaluate this in the future, such as in a preview3 timeframe, but for now, let's go with the simplest thing that works. * prepare adapter directory layout for upstreaming (#172) * delete adapter src/main.o: this was accidentally left out of #165 * move adapter, byte-array, and verify to a new workspace * rename byte-array crate to a name available on crates.io * add a readme for verify, also give it a slightly better name * CI: wit dep check in its own step, verify before publish, trim down publication * reactor-tests: delete deps symlinks * reactor-tests: manage wit with wit-deps * test: dont set default toolchain to nightly * wit-deps lock adapter * wit-deps lock reactor-tests wit-deps doesnt manage these for some reason * ci: add build-preview1-component-adapter step. prtest:full * temporary: always run the adapter build CI step * add ci/build-wasi-preview1-component-adapter script * put both adapter binaries in one artifact, name it bins-*, publish it * restore adapter to only building if run-full * change adapter to be part of root workspace * rm deps.lock, toml: wit-deps integration is not implemented * use latest wit-bindgen, add wildcard audits for wit-bindgen and wasm-tools crates * cargo-vet all remaining dependency bumps * cargo deny: allow the Unicode-DFS-2016 license this has similar requirements to OpenSSL: permissive, but the copyright statement must be advertised * run-tests: exclude the adapter, which doesnt build for native * adapter: compile_error when not wasm32-unknown-unknown, and note this where the static assertion fails --------- Signed-off-by: Joel Dice <joel.dice@fermyon.com> Co-authored-by: Dan Gohman <dev@sunfishcode.online> Co-authored-by: Alex Crichton <alex@alexcrichton.com> Co-authored-by: Joel Dice <joel.dice@fermyon.com> Co-authored-by: Trevor Elliott <telliott@fastly.com>
pchickey
pushed a commit
to pchickey/wasmtime
that referenced
this pull request
May 16, 2023
* Update wasmtime and wit-bindgen This commit updates the `wasmtime` dependency and the `wit-bindgen-guest-rust` dependencies to namely update the `wit-parser` and component underlying implementations. This pulls in bytecodealliance/wasm-tools#867 which is the implementation of `use` for WIT. This does not currently break apart the one large `wasi.wit` file, instead just gets tests working. * Split apart `wasi.wit` into multiple `*.wit` files This commit refactors the monolithic `wasi.wit` file into multiple files. I've taken some liberties in naming here so suggestions are definitely welcome! I've resolved some TODO annotations as well about `use`-ing types between interfaces. There are two issues remaining, however: * The `wasi-command` world still hardcodes `u32` because `world` items don't support `use` just yet. That isn't a hard limitation of WIT, however, it's just a temporary limitation of the implementation. * The `wasi-clocks` interface defines the `wasi-future` type. This is due to a cyclic dependency where `wasi-future` is defined within `wasi-poll` which depends on `wasi-clocks` for types. I've left this to a future refactoring to probably have a `types` interface of some form somewhere. * Update patch for wasm-tools * Switch to wasmtime upstream * Update CI for build * Remove patch overrides * Update names of uploaded files
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I'm opening this up now to share some early work on this. This is unfortunately going to again be a very large PR due to the sheer amount of infrastructure required to pull in all the WIT updates, namely to implement WebAssembly/component-model#141. By-and-large the headline features of this PR will be:
usekeyword will be fully implemented. This means that WIT now has the concept of packages, dependencies between packages, documents, etc. Packages can be split between multiple files and packages can depend on other packages as well. All of theusestatements are preserved in the internal representation to enable code generators to see connections between types and generate appropriate code, e.g. onestructin Rust.wit-parser-specific convention is invented here for a set of packages residing on the filesystem, intended to probably be in awit/folder at the top of a project. The "root package" is located atwit/*.witand dependencies are located atwit/deps/$dep/*.wit. Dependencies can use amongst themselves and this should provide a means by which to depend on packages (like WASI) before registry tooling is in place. Submodules may also help here too perhaps.wit-componentfor all of these features. This means that a component can be created from a world which is itself assembled from many packages/files/interfaces/etc.wasm-tools component new --types-onlyinterface is removed in favor ofwasm-tools component wit --binary. More-or-less the binary format for WIT is now supported underwasm-tools component witwhich can be used for printing the WIT of a preexisting component, converting a WIT binary to WIT text, or converting WIT text to WIT binary.I'm opening this up as a work-in-progress at this time. I'm almost done with
wit-parserand the text syntax bits. A good way to explore this is through the tests updated and added in this PR. Procedually I'll be taking next week and the week after off so this is not going to merge and get published until at least January 2023.I'll preemptively apologize again for the size and scale of this PR. As with #858 there's not really a great way to review this. That being said I think there's a case to be made for the WIT format being so "early days" that a meticulous review probably isn't worth it at this point anyway. We've got a process in place at the component-model repository to discuss syntax, semantics, etc, and the goal of this PR is for this repository to become a faithful implementation of that specification. Any surprising behavior should either be clarified upstream or fixed in this repository to match what's specified upstream. In that sense most of the "review" I think is best done as discussion about the concepts on PRs like WebAssembly/component-model#141. The technical implementation here certainly has quite a lot to review as well, but personally I feel it's best to largely get everything working first to have a basis on which to iterate, as currently that doesn't altogether exist.