Add pseudo-streams. #29

sunfishcode · 2022-12-16T00:16:03Z

This add a pseudo-stream type to the wasi-poll interface, and adds ways to obtain streams from command invocation and from files. In the future, it can support sockets too. With this, command takes streams for stdin/stdout, rather than filesystem descriptors.

Streams support reading and writing, as well as skipping, repeated-element writing, and splicing from one stream to another. And there are subscribe-* functions to produce pseudo-futures from pseudo-streams, allowing them to be polled.

This makes the polyfill somewhat more complex, but this is largely due to the polyfill being tied to the preview1 API.

This replaces the seek and tell functions, and implemented fd_seek and fd_tell in terms of the polyfill's own position.

Also, add a dedicated stderr API for writing to stderr in a way that tolerates strings that aren't necessarily expected to be newlines. And add a way to test whether stderr is a terminal.

sunfishcode · 2022-12-21T21:51:23Z

This now also implements polling, on both streams and clocks.

This is based on @npmccullum's [wasi-snapshot-preview2 draft] which is in turn based on the [wasi-sockets proposal], though for simplify for now it omits UDP sockets and some other features. It's also based on the [pseudo-streams PR]; so that should proceed first before this. [draft wasi-snapshot-preview2]: https://github.com/npmccallum/wasi-snapshot-preview2 [wasi-sockets proposal]: https://github.com/WebAssembly/wasi-sockets [pseudo-streams PR]: #29

This add a pseudo-stream type to the wasi-poll interface, and adds ways to obtain streams from command invocation and from files. In the future, it can support sockets too. With this, `command` takes streams for stdin/stdout, rather than filesystem descriptors. Streams support reading and writing, as well as skipping, repeated-element writing, and splicing from one stream to another. And there are `subscribe-*` functions to produce pseudo-futures from pseudo-streams, allowing them to be polled. This makes the polyfill somewhat more complex, but this is largely due to the polyfill being tied to the preview1 API. This replaces the `seek` and `tell` functions, and implemented `fd_seek` and `fd_tell` in terms of the polyfill's own position. Also, add a dedicated stderr API for writing to stderr in a way that tolerates strings that aren't necessarily expected to be newlines. And add a way to test whether stderr is a terminal.

This implements pseudo-futures and subscription functions, and adds polling for streams.

wasi.wit: - Remove the "timers" API from wasi-clocks, as it's now redundant with pseudo-future clock subscriptions. - Remove `subscribe-wall-clock`. Wall-clock timeouts were implemented by converting them to monotonic-clock timeouts anyway, so just make that explicit in the WASI API, and teach the polyfill how to convert wall-clock timeouts into monotonic-clock timeouts. - Move `subscribe-monotonic-clock` out of wasi-clocks and into wasi-poll, as it's closely tied to the pseudo-futures mechanism and the `poll-oneoff` implementation. - While here, fix `stream-read` and related functions to return an end-of-stream/file indicator. Code changes: - `default_wall_clock()` and `default_monotonic_clock()` now always create a new table entry, rather than holding a table index in the `WasiCtx` which could potentially dangle. - Add support for monotonic-clock poll subscriptions. - Say "wall clock" instead of "system clock" when we have a choice.

dicej

Looks good overall. Please see comments inline.

Also, (and this may be out of scope for this PR) I'm wondering how much of the existing cap-std-sync::sched implementation we want to keep given the comments in windows.rs indicate that it is buggy and unmaintained. Not that I'm volunteering to replace it :). Just wondering if we want to try to reuse something like Tokio instead.

host/src/clocks.rs

host/src/poll.rs

src/lib.rs

This ports the wording from WebAssembly/wasi-clocks#22.

sunfishcode · 2022-12-23T23:27:14Z

Also, (and this may be out of scope for this PR) I'm wondering how much of the existing cap-std-sync::sched implementation we want to keep given the comments in windows.rs indicate that it is buggy and unmaintained. Not that I'm volunteering to replace it :). Just wondering if we want to try to reuse something like Tokio instead.

I think we do want to do this, and I wouldn't object if someone volunteered to do it, but if not I think we're ok there for the time being. The higher priorities are building up the preview2 API and getting to feature parity with preview1.

This is based on @npmccullum's [wasi-snapshot-preview2 draft] which is in turn based on the [wasi-sockets proposal], though for simplify for now it omits UDP sockets and some other features. It's also based on the [pseudo-streams PR]; so that should proceed first before this. [draft wasi-snapshot-preview2]: https://github.com/npmccallum/wasi-snapshot-preview2 [wasi-sockets proposal]: https://github.com/WebAssembly/wasi-sockets [pseudo-streams PR]: #29

* Add basic TCP socket APIs. This is based on @npmccullum's [wasi-snapshot-preview2 draft] which is in turn based on the [wasi-sockets proposal], though for simplify for now it omits UDP sockets and some other features. It's also based on the [pseudo-streams PR]; so that should proceed first before this. [draft wasi-snapshot-preview2]: https://github.com/npmccallum/wasi-snapshot-preview2 [wasi-sockets proposal]: https://github.com/WebAssembly/wasi-sockets [pseudo-streams PR]: #29

* Add basic TCP socket APIs. This is based on @npmccullum's [wasi-snapshot-preview2 draft] which is in turn based on the [wasi-sockets proposal], though for simplify for now it omits UDP sockets and some other features. It's also based on the [pseudo-streams PR]; so that should proceed first before this. [draft wasi-snapshot-preview2]: https://github.com/npmccallum/wasi-snapshot-preview2 [wasi-sockets proposal]: https://github.com/WebAssembly/wasi-sockets [pseudo-streams PR]: bytecodealliance/preview2-prototyping#29

@npmccallum

…ime (#6374) * An initial preview1 polyfill stub. * Add test on CI and infrastructure for wasm globals (#1) * Updates * Add a verification test to CI * Produce a global-referencing object on stable * Restrict CI again * Add a comment about a disabled lint. * Add an `extern "C"` to the Rust code comment. * Use `*mut u8` instead of `usize` to hold pointers. This will reduce casting, and better preserve pointer provenance. * Mark the WASI functions `unsafe`. * Fix the inline asm comment to match the code. * Implement a polyfill for `path_readlink`. Implement a polyfill for `path_readlink` in terms of the preview2 `readlink_at` function. This involves adding a second wasm global, to hold the buffer size. * Use raw_strings, disable integer overflow checks, and suppress jump tables. This also updates the verifier to accept the new wasi imports. * Use a compiler_fence. * Add a slow-path to handle smaller buffers passed to `path_readlink`. * These `forget`s aren't needed because they just have references. * Properly forget returned buffers. * Remove the code that disabled the global allocator. * Mark `path_readlink_slow` as inline(never). * Minor cleanups. * Remove `unregister_buffer`. When `cabi_realloc` uses the buffer pointer, it sets the stored buffer pointer to null, so we don't need a separate `unregister_buffer` call. * Implement file descriptors. * Implement more stuff. * Implement more functions. * Fixes to avoid static inits. * Add a file descriptor bounds check. * Simplify a few casts. * Multiply memory.grow's result by the page size, and handle failure. * Remove an unneeded `register_buffer`. * Convert to use `u32` indices instead of handles. This is a temporary experiment. * fd_info fixup * switch to latest wit_bindgen_guest_rust macro * notes for what to do next * adapter: take args, have two cabi_{import,export}_realloc funcs, and call adaptee start * skip codegen for command, and fill in the entrypoint manually * cabi_export_realloc is another bad allocator for our bad allocator museum * Implement polyfills for several wasi-filesystem functions Implement fd_advise, fd_filestat_set_times, path_filestat_set_times, and path_remove_directory. * Implement fd_filestat_set_size, fd_pread, and fd_pwrite. * Implement path_create_directory, path_filestat_get, path_link. * Implement path_rename, path_symlink, and path_unlink_file. * Swap `info` and `fd-info` See the last commit in WebAssembly/wasi-filesystem#66 for details. * Implement `fd_fdstat_get` and `fd_fdstat_set_flags`. This incoporates several wasi-filesystem repo changes too. * Implement `clock_{time,res}_get` (#12) Currently traps for the `*_CPUTIME_*` clocks and doesn't check for overflow on nanoseconds. * Update Wasmtime/tooling dependencies (#14) This forces all the `*.wit.md` file to go into one large `*.wit` file for now which will get split up later once `use` is re-introduced. * Simplify global state management in adapter (#13) * Simplify global state management in adapter I was looking recently to implement args-related syscalls but that would require yet-more globals and yet-more state to be managed. Instead of adding a slew of new globals for this which are all manually kept in sync I opted to instead redesign how global state is managed in the adapter. The previous multiple `global`s are all removed in favor of just one, as sort of a "tls slot" of sorts. This one remaining slot points to a one-time-allocated `State` object which internally stores information like buffer metadata, fd information, etc. Along the way I've also simplified syscalls with new methods and `?`-using closures. * Turn off incremental for dev builds Helps with CGU splitting and ensuring that appropriate code is produced even without `--release`. * Review comments * Add accessors with specific errors * Update handling of `*_global_ptr` * Update internal mutability around path buffer Use an `UnsafeCell` layering to indicate that mutation may happen through `&T`. * Implement `args_{get,sizes_get}` functions (#16) This commit updates the implementation of `cabi_export_realloc` to allocate from a bump-allocated-region in `State` rather than allocating a separate page for each argument as previously done. Additionally the argument data is now stored within `State` as well enabling a full implementation of the `args_get` and `args_sizes_get` syscalls. * sketch of `poll_oneoff` polyfill (#11) This adds a `poll_oneoff` implementation to src/lib.rs. It's completely untested. I was planning on adding a host implementation and using that to test end-to-end, but that raised some tough questions about how much of the existing `wasi-common` scheduler(s) should be reused. I've decided to focus on other, more widely-used parts of WASI first, but I wanted to share the work I've already done here. Note that I've moved the clock- and socket-specific functions out of `wasi-poll` and into `wasi-clocks` and `wasi-tcp`, respectively. The latter is a new interface which will eventually contain functions and types resembling @npmccallum's https://github.com/npmccallum/wasi-snapshot-preview2#wasi-tcp proposal. Per discussion with @sunfishcode: - `wasi-tcp` includes an `error` enum type, intended to represent only socket-related errors. - It also includes a `socket` pseudo-handle type, distinct from `wasi-filesystem`'s `descriptor` type. These fine-grained types help move us away from the "everything is a file descriptor" and "all errors are errnos" approaches of Preview 1. If we decide `poll-oneoff` should be usable with files as well as sockets, we can add `subscribe-read` and `subscribe-write` functions to `wasi-filesystem` which accept file `descriptor`s. Likewise for any other pseudo-handle type from which we'd like to create futures. Signed-off-by: Joel Dice <joel.dice@fermyon.com> Signed-off-by: Joel Dice <joel.dice@fermyon.com> * Update a few aspects of the adapter build (#17) * Update a few aspects of the adapter build * Use the `wasm32-unknown-unknown` target for the adapter and specify flags in `.cargo/config.toml` to avoid having to pass the same flags everywhere. This allows using `wasm32-wasi` for tests to ensure the flags only apply to the adapter. * Use `opt-level=s` since speed is not of the utmost concern for this wasm but since it's likely to be included in many places size is likely more important. * Use `strip = 'debuginfo'` for the release build to remove the standard library's debugging information which isn't necessary. * Remove `debug = 0` from the `dev` profile to have debugging information for development. * Add a small `README.md` describing what's here for now. * Move `command` support behind a `command` feature This commit adds a `command` feature to main crate to avoid importing the `_start` function when the `command` feature is disabled, making this adapter useful for non-command WASI programs as well. For now this still emits the `command` export in the final component but with `use` in `*.wit` files it should be easier to avoid that export. * Implement `path_open`. (#18) * Implement `path_open`. * Implement `fd_close`, and finish `path_open`. * Stub out `close` in the host impl. * Add a `badf` error code. This pulls in WebAssembly/wasi-filesystem#62. * Fix prints in non-command builds (#19) * Fix prints in non-command builds When the `command` entrypoint isn't invoked, such as for non-command builds, then prints were not working as they would return `ERRNO_BADF` which is swallowed by at least Rust right now. This instead sets the initialization of `State` to reflect how fd 0 is an empty stdin stream, fd 1 goes to a `log` call, and fd 2 also goes to `log`. During `command` initialization fds 0 and 1 are overwritten with the provided input and for non-command builds this is the state permanently for the program. It's possible we can add further configuration hooks for this in the future, but this should get at least the initial state of non-command builds more workable. * Use BADF for reading log fds * Implement the `fd_readdir` function (#23) * Implement the `fd_readdir` function This roughly matches the implementation in `wasi-common` today where it uses the directory entry stream as a form of iterator and the `cookie` represents the `n`th iteration. Not exactly efficient if the `buf` provided to the hostcall is too small to hold many of the entries but there's not a whole lot else that can be done at this time. This also updates the WIT for `read-dir-entry` to return `option<dir-entry>` instead of `dir-entry` to indicate EOF. * Fix host compile * Add a specific method to close a dir-entry-stream * Add a cache to avoid quadratic dirent behavior When a readdir call is truncated due to the output buffer not being large enough save the final state of the readdir iterator into `State` to possibly get reused on the next call to `readdir`. Some limits are put in place such as: * The next call to readdir must be for the same fd and the required cookie. * The dirent that didn't fit must have its full name fit within the path cache. * If `fd_close` is called after a readdir it clears the cache so a future `fd_readdir` doesn't actually resume now-stale state. There's a fair bit of trickiness here so I've attempted to structure things as "simply" as possible to hopefully reduce the chance there's an issue, but this is all untested so there's still likely an off-by-one or similar bug. * Implement `fd_renumber`. (#21) * Implement `fd_renumber`. * Implement `Drop` for `Descriptor` to free backend resources. Instead of trying to remember to call `close` on file descriptors when they're replaced or removed, just have the `Drop` impl for `Descriptor`. * Use the local `unwrap_result` instead of `.unwrap()` This avoids static memory inits. * Don't print extra metadata in stdout/stderr, and implement proc_exit (#27) Add an API to implement proc_exit with, and implement it. * add tests for `wasi-random`, `wasi-clocks`, and stdin (#30) This required fleshing out the `wasi-clocks` host implementation a bit and adding a `read_vectored_at` implementation for `ReadPipe`. Signed-off-by: Joel Dice <joel.dice@fermyon.com> Signed-off-by: Joel Dice <joel.dice@fermyon.com> * Add a way to get a random `u64`. (#36) This anticipates WebAssembly/wasi-random#18. * Fix `fd_renumber` to handle output fds that aren't previously allocated. (#38) This fixes the crates/test-programs/wasi-tests/src/bin/stdio.rs test in the Wasmtime testsuite. * add support for environment variables and preopens Per #31, we pass env vars and preopens via `command`, just like CLI arguments. Later, we plan to add another interface (e.g. `cli-reactor`) for reactors which need env vars and/or preopens, which will have a single function `initialize` that takes the same two parameters (but not stdio or CLI arg parameters). This also removes unused parts of `wasi-common` (e.g. args, and env vars, and preopen paths) which have been superceded by the `command` interface. Our eventual goal is to provide a more explicit interface for environment variables using WIT star imports such that the guest will import a zero-argument function (or value, when that is supported) for each variable it needs, allowing the host to statically verify that it can provide all those variables. However, star imports are not yet supported in `wit-bindgen`, and once they are, we'll likely still need a dynamic interface to support existing software. Note that I've added a `file_read` test which does not actually do anything yet, since not all the required host functions have been implemented. I plan to address that soon. Signed-off-by: Joel Dice <joel.dice@fermyon.com> * implement more host filesystem functions The `file_read` test now passes. Signed-off-by: Joel Dice <joel.dice@fermyon.com> * Add pseudo-streams. (#29) * Add pseudo-streams. This add a pseudo-stream type to the wasi-poll interface, and adds ways to obtain streams from command invocation and from files. In the future, it can support sockets too. With this, `command` takes streams for stdin/stdout, rather than filesystem descriptors. Streams support reading and writing, as well as skipping, repeated-element writing, and splicing from one stream to another. And there are `subscribe-*` functions to produce pseudo-futures from pseudo-streams, allowing them to be polled. This makes the polyfill somewhat more complex, but this is largely due to the polyfill being tied to the preview1 API. This replaces the `seek` and `tell` functions, and implemented `fd_seek` and `fd_tell` in terms of the polyfill's own position. Also, add a dedicated stderr API for writing to stderr in a way that tolerates strings that aren't necessarily expected to be newlines. And add a way to test whether stderr is a terminal. * Implement the host side of `poll_oneoff`. This implements pseudo-futures and subscription functions, and adds polling for streams. * Implement clock subscriptions. wasi.wit: - Remove the "timers" API from wasi-clocks, as it's now redundant with pseudo-future clock subscriptions. - Remove `subscribe-wall-clock`. Wall-clock timeouts were implemented by converting them to monotonic-clock timeouts anyway, so just make that explicit in the WASI API, and teach the polyfill how to convert wall-clock timeouts into monotonic-clock timeouts. - Move `subscribe-monotonic-clock` out of wasi-clocks and into wasi-poll, as it's closely tied to the pseudo-futures mechanism and the `poll-oneoff` implementation. - While here, fix `stream-read` and related functions to return an end-of-stream/file indicator. Code changes: - `default_wall_clock()` and `default_monotonic_clock()` now always create a new table entry, rather than holding a table index in the `WasiCtx` which could potentially dangle. - Add support for monotonic-clock poll subscriptions. - Say "wall clock" instead of "system clock" when we have a choice. * Remove the `OFlags::APPEND` flag, which is no longer used. * Implement `command` exit statuses. (#40) Add a `result` return type to `command` so that it can indicate success or failure. The idea here is that this isn't a full `i32` return value because the meaning of return values isn't portable across platforms. Also, Typed Main is a better long-term answer for users that want rich error return values from commands. * implement `wasi-filesystem::readdir` and related functions (#45) * implement `wasi-filesystem::readdir` and related functions This adds a `directory_list` test and provides the required host implementation. I've also added a file length check to the `file_read` test, just to cover a bit more of the API. Signed-off-by: Joel Dice <joel.dice@fermyon.com> * fix memory corruption in `fd_readdir` polyfill We were copying `name.len() * 256` bytes instead of just `name.len()` bytes, which was overwriting other parts of `State` and causing untold havoc. Signed-off-by: Joel Dice <joel.dice@fermyon.com> * check type of entry in `Table::delete` Signed-off-by: Joel Dice <joel.dice@fermyon.com> Signed-off-by: Joel Dice <joel.dice@fermyon.com> * Minor cleanups (#49) * Use `use` imports consistently. * Rename `flags_from_descriptor_flags` to `descriptor_flags_from_flags`. * Rename `black_box` to `obscure`. * Make some comments be doc comments. * Use a hyphen for compound adjectives. * Delete an unused variable. * Update the name of `change-file-permissions-at`. * Use generated typedefs instead of hard-coding u64. * Use core::hint::black_box now that it's stabilized. * Try to detect memory corruption with "magic" canaries (#52) After two separate rounds of memory corruption now happening it's probably best to place some protections in place to try to detect when this happens earlier rather than much later after the fact. * Implement file append functionality. (#50) * Implement file append functionality. - In the preview1-to-preview2 polyfill, using `append_via_stream`. - In the host implementation, using a new system-interface `FileIoExt::append` function. * Add a basic testcase. * Fix some bugs turned up in the Wasmtime testsuite. * Allocate bool results at the end of the events buffer. * Don't fail poll for per-fd failures. When a fd is not pollable, report it as an immediately-ready future, rather than as a poll failure. * Implement `sync` and `datasync` on files and directories. On Windows, there doesn't appear to be a way to sync a directory, to ensure that the directory entry for a file is sync'd. So for now, just silently succeed. I've opened WebAssembly/wasi-filesystem#79 to track this at the spec level. * Add `eprintln`, `unreachable`, and other macros. (#62) Add a `byte-array` proc-macro crate for converting strings into byte arrays that don't use static initializers, and use it to implement `eprintln`, an `unreachable` that prints the line number, and other macros. * Use the `wasi-stderr` API for stderr (#59) Use the dedicated `wasi-stderr` API for stderr, instead of `wasi-logging`. Fixes #57. * Change the filesystem timestamp to seconds+nanoseconds. (#55) This corresponds to WebAssembly/wasi-filesystem#76. * Update wasmtime and wit-bindgen (#68) * Update wasmtime and wit-bindgen This commit updates the `wasmtime` dependency and the `wit-bindgen-guest-rust` dependencies to namely update the `wit-parser` and component underlying implementations. This pulls in bytecodealliance/wasm-tools#867 which is the implementation of `use` for WIT. This does not currently break apart the one large `wasi.wit` file, instead just gets tests working. * Split apart `wasi.wit` into multiple `*.wit` files This commit refactors the monolithic `wasi.wit` file into multiple files. I've taken some liberties in naming here so suggestions are definitely welcome! I've resolved some TODO annotations as well about `use`-ing types between interfaces. There are two issues remaining, however: * The `wasi-command` world still hardcodes `u32` because `world` items don't support `use` just yet. That isn't a hard limitation of WIT, however, it's just a temporary limitation of the implementation. * The `wasi-clocks` interface defines the `wasi-future` type. This is due to a cyclic dependency where `wasi-future` is defined within `wasi-poll` which depends on `wasi-clocks` for types. I've left this to a future refactoring to probably have a `types` interface of some form somewhere. * Update patch for wasm-tools * Switch to wasmtime upstream * Update CI for build * Remove patch overrides * Update names of uploaded files * Use a more targeted means of specifying link flags (#71) * Use a more targeted means of specifying link flags I had forgotten earlier that this could be done with build scripts so do that in the adapter's build script rather than as auxiliary rust flags. * Remove `.cargo/config.toml` file This now only serves the purpose to enable bulk-memory which is relatively minor. This removes the file for now and wasm features can always be reenabled at a later date if file size is truly an issue. * Fix the non-command build (#72) This commit fixes the non-command build of the WASI adapter by adding a new `wasi.wit` which only has the imports and nothing else. * Rename `wasi-future` to `waitable`. (#70) * Rename `wasi-future` to `waitable`. You can wait on `waitable`s multiple times, which differs from proper futures, which you can only wait on once. * Rename `waitable` to `pollable`. Because the function they're passed to is `poll-oneoff`. * use main module's `cabi_realloc` instead of `memory.grow` This depends on bytecodealliance/wasm-tools#900, which will polyfill an implementation if the main module does not export it. Signed-off-by: Joel Dice <joel.dice@fermyon.com> * Add basic TCP socket APIs. (#33) * Add basic TCP socket APIs. This is based on @npmccullum's [wasi-snapshot-preview2 draft] which is in turn based on the [wasi-sockets proposal], though for simplify for now it omits UDP sockets and some other features. It's also based on the [pseudo-streams PR]; so that should proceed first before this. [draft wasi-snapshot-preview2]: https://github.com/npmccallum/wasi-snapshot-preview2 [wasi-sockets proposal]: https://github.com/WebAssembly/wasi-sockets [pseudo-streams PR]: bytecodealliance/preview2-prototyping#29 * Add the preview1 `sock_accept` function declaration to the polyfill. (#77) * Split wasi-stream into separate input and output stream types. (#73) * Split wasi-stream into separate input and output stream types. This syncs the prototype's streams with the upstream wasi-io repo. Streams are unidirectional, so this allows us to statically describe whether something is an input stream or an output stream in an interface. This differs a little from the component model async streams, which don't have separate input and output streams, but it does partially reflect how the component model async design differentiates between input streams in type signatures, which are passed in as arguments, and output streams, which are returned as return values. * Fix compilation on Windows. * polyfill: fill in fdflags_append from File::append * short-circuit reentrance when allocating stack and `State` (#83) * short-circuit reentrance when allocating stack and `State` Per bytecodealliance/wasm-tools#919, `wit-component` needs to lazily allocate the adapter stack to avoid premature or infinite reentrance from the main module to the adapter. This means adding an additional `allocation_state` global variable and using it to determine when to return early from an exported function, e.g. because we're in the process of either allocating the stack or allocating `State`. This requires an updated `wit-component` dependency once bytecodealliance/wasm-tools#919 has been merged. Fixes #78 Signed-off-by: Joel Dice <joel.dice@fermyon.com> * remove redundant unsafe blocks Signed-off-by: Joel Dice <joel.dice@fermyon.com> * update dependencies This brings us up-to-date with wasi-tools, wit-bindgen, and the latest component ABI. Signed-off-by: Joel Dice <joel.dice@fermyon.com> --------- Signed-off-by: Joel Dice <joel.dice@fermyon.com> * Move to crates.io/releases of dependencies (#84) * Use precompiled `wit-bindgen` binaries from CI * Use a release branch of Wasmtime for 6.0.0 * Use `wit-bindgen`-the-crate from crates.io * replace `fn unwrap` and `fn unwrap_result` with postfix `.trapping_unwrap()` (#86) this is just aesthetic, but prefixed unwraps are a lot harder on the eyes than postfixed * Update wasi-filesystem, wasi-clocks, wasi-io, and wasi-poll. This updates to the latest versions of the wit files in the proposal repos, and updates the polyfill and host implementations accordingly. The changes here are just some renames and reorganizations. * Refactor adapter's allocators (#88) * create a reusable BumpArena, rename command_data to long_lived * use ImportAlloc::with_buffer to register a bump arena for each import func call * import alloc just hands out buffer, nothing more only improvement over prior implementation is this one makes sure alignment is followed * make merging easier... * bumparena is now a singleton so it can own the memory area * get environment and preopens through import functions * add arena allocator for get_environment and get_preopens and fix other bugs in those implementations * comments * Make Preview2's directory iterator omit `.` and `..`. (#89) Preview1's `fd_readdir` includes `.` and `..`, but Preview2 is changing to avoid this. Update the Preview2 host implementation to omit these entries, and add code the polyfill to re-add them. * Fix timestamp-to-nanosecond conversion in the adapter (#93) The order of operations for handling the seconds/nanoseconds part of the structure were a bit off which meant that a nonzero nanosecond field would have an unexpected effect on the result. Closes #92 * short-circuit `fd_write` in the adapter @guybedford was getting assertion errors in `State::new` due to the debugging print statements he had added to his `cabi_realloc` function, which called back into `fd_write`, leading to a circular dependency where `fd_write` needed to allocate, which caused `fd_write` to be called, etc. The fix is easy enough: short circuit this circularity the same way we're handling it in `environ_sizes_get`, `clock_time_get`, etc. If we wanted to be really paranoid, we could add this short circuit to all the functions, but I'd be really surprised if `cabi_realloc` needs to read from the filesystem, create directories, or other such things. If we see that in the wild, we can revisit. Signed-off-by: Joel Dice <joel.dice@fermyon.com> * Rebase on the new wasi-sockets. (#91) * Rebase on the new wasi-sockets. This switches to using the wasi-sockets wit files from WebAssembly/wasi-sockets#16. Many things are still stubbed out with `todo!()` for now. * Fix compilation on Windows. * Rebase on the wasi-cli world. (#100) * Rebase on the wasi-cli world. Rebase on the wasi-cli wit files. This incorporates some major renamings and reorganizations, and updates to use wasi-sockets from the repo rather than our old prototype files. At present this depends on several bugfixes in wasmtime and wit-bindgen so this is using git dependencies for now. This also temporarily preserves the dedicated stderr interface, which is useful for debugging the file descriptor table code, which needs to work before the regular stderr works. * Update the verify program for the new module names. * Disable debug-assertions in the adapter build. This is roughly the same as, the previous `unchecked` option in the bindings, with the changes in the latest wit-bindgen. * Update CI to use the new file names. * code motion: cli related feature names (#104) * rename host runtime tests to command tests * add a test for using wasi in a reactor * commit cargo.lock * reactor tests: fix wit deps * test-programs: reactor-tests can build on stable note that this fails because it exposes wasi-libc ctors calling import functions from inside cabi_realloc * test-programs: show that ctors fix in wit-bindgen fixes bug * ci: install wasm32 targets for stable as well as nightly * wit-bindgen: use 0.4.0 * ci: use wit-bindgen 0.4.0 * Co-habitate with wasi-common from wasmtime * adapter: code motion in cargo feature & artifact names to cli-command, cli-reactor there will shortly be a third type of reactor (non-cli, idk what to call it) --------- Co-authored-by: Trevor Elliott <telliott@fastly.com> * Update to the latest wit files. This updates to the latest wasi-filesystem, wasi-sockets, wasi-io, and wasi-cli wit changes, except for two things: - `filesystem.types` is temporarily still named `filesystem.filesystem`, to work around #5961 - wasi-stderr is temporarily still present, for debugging * Support command-line argument preopens. * Minor code simplification. * drop cli- prefix from reactor, command features & artifact names * intrinsics: just abstract the archive syms a little bit * intrinsics: add stderr stream to globals, plus getter/setter funcs * wire up adapter macros to stderr stream print * adapter: make it a little clearer where unreachable is coming from * host and adapter: provide stdio via preopens, dont pass stdio or arg preopens to main stdio never gets initialized for the reactor, tha * get rid of arg_preopens from state, env_preopens are just preopens * factor descriptors out into a separate module i didnt run verify step-by-step during this so i accidentally got panick machinery in here. thats what i get * fix macros * fix * renumber: allow overwriting. note it may invalidate preopen table * poll_oneoff: use import allocator solely on the import func call this is required because calling `state.descriptors()` could end up initializing the descriptor table, which also uses the import allocator. this is a huge whitespace change, but effectively its just a much smaller body in the closure. * Update build.rs Co-authored-by: Joel Dice <joel.dice@fermyon.com> * Descriptors is initialized on creation, use refcell<option<descriptors>> to represent uninit * adapter: arguments are lazily initialized in State instead of passed to main * rename command's `main` to `run` (unfortunately) this is an unfortunate problem with building the adapter: for some reason, wasm-ld will modify an export named `main` with no parameters by renaming it __original_main, and exporting a `main` with two i32 parameters. we should see if this behavior is fixable, this is hopefully a temporary change because I really liked that commands could just be invoked by calling `main`... * Remove resource abstraction from clocks (#129) * wit: remove resource index from monotonic-clocks and wall-clocks, delete instance-*-clock interfaces At this time, we don't have use cases for providing multiple wall clocks or multiple monotonic clocks to the same component. So, according to the principle of following preview 1's design as close as possible, we are changing these interfaces to being provided by ambient functions, rather than methods on a resource. * adapter: unresourcify clocks * host: de-resourcify clocks * Rename the command entrypoint from `run` back to `main`. (#131) LLVM has special handling for functions named `main`, which we need to avoid because we're creating the component-level `main` rather than the C-level `main`. To do this, write the `main` function in assembly, which is fortunately very simple now. * fix wasi renumber test: return an error in adapter if closing a closed fd * NFC: pull eventtype const definitions out to be reused further down * size assertion: refactor to de-duplicate MAX_DESCRIPTORS definition * File: we need to have the DescriptorType around to handle certain corner cases * fd_seek: return the expected errno when trying to seek on a directory. * fix fd_filestat_get: adapter special-case StreamType::Unknown (#139) giving an empty Filestat, instead of returning an error. This implements the behavior expected by the fd_filestat_get test. We might end up being able to rename StreamType's Unknown variant to Stdio, since it looks like that is the only place we construct it. * return ERRNO_BADF on directory descriptors in appropriate fd operations * dead code * adapter: fd_allocate gives notsup on all files * adapter: remove incorrect assertions from path_readlink * readlink: fix adapter to return right len, and test to expect truncation truncation behavior is backported to upstream in #6225 * adapter: track blocking state in File, remove uses of set-flags * adapter: change logic around interpeting ATIM and ATIM_NOW (and MTIM equiv) to be consistient with upstream in wasi-common preview1, ATIM and ATIM_NOW (and MTIM and MTIM now) were mutually exclusive and would result in an INVAL error, whereas in the adapter previously, ATIM_NOW implied ATIM, but would silently do nothing if ATIM was not set. I decided to be consistient with the upstream behavior here because it is pretty arbitrary and I don't think there's a good reason to break compatibility. This fixes the `path_filestat` test. * fd_seek: error with invalid offsets in the adapter * NFC, looks more consistient to my eye though * fix bug in adapter fd_fdstat_set_flags to frob the append flag. this fixes the fd_flags_set test. * delete spurrious extra line in adapter (this same var is created below) * adapter poll_oneoff: when descriptors.get_*_stream(fd) fails, die with that error (#154) * adapter poll_oneoff: when descriptors.get_*_stream(fd) fails, die with that error There was a special case in poll_oneoff that put in a fake clock stream when a read/write stream for a descriptor wasn't available. The poll_oneoff_files test (in `test_fd_readwrite_invalid_fd()`) checks that poll_oneoff returns a BADF when an invalid fd is subscribed to. I'm not sure what the special case was patching over, but this passes all of the other tests right now. * poll_oneoff_files fails on windows with god knows what error diff --git a/host/tests/command.rs b/host/tests/command.rs index 7af7bd0..67c8c0b 100644 --- a/host/tests/command.rs +++ b/host/tests/command.rs @@ -466,10 +466,11 @@ async fn run_path_symlink_trailing_slashes(store: Store<WasiCtx>, wasi: Command) } async fn run_poll_oneoff_files(store: Store<WasiCtx>, wasi: Command) -> Result<()> { - // trapping upwrap in poll_oneoff in adapter. - // maybe this is related to the "if fd isnt a stream, request a pollable which completes - // immediately so itll immediately fail" behavior, which i think breaks internal invariant... - run_with_temp_dir(store, wasi).await + if cfg!(windows) { + expect_fail(run_with_temp_dir(store, wasi).await) + } else { + run_with_temp_dir(store, wasi).await + } } async fn run_poll_oneoff_stdio(store: Store<WasiCtx>, wasi: Command) -> Result<()> { * adapter: path_ apis return NOTDIR instead of BADF when passed file; fixes path_open_dirfd test. (#152) * adapter: special case for NOTDIR instead of BADF fixes path_open_dirfd test. * get_dir now fails with NOTDIR for non-dir files * get rid of special case * complete test coverage for all path_ functions giving NOTDIR errors * adapter: StreamType::Unknown is actually Stdio (#161) We have only ever used Unknown for the stdio streams, and I don't expect us to use it for anything else in the future, so rename it. Set the returned filetype to character device: Closes #146. Also, fix some warnings. * Revert #131, renaming `main` back to `run`. (#165) Changing LLVM and/or Rust to avoid special handling of `main` is a fair amount of work, and there could be other toolchains with similar special rules for functions named `main`, so rename the command entrypoint back to `run`. We could potentially re-evaluate this in the future, such as in a preview3 timeframe, but for now, let's go with the simplest thing that works. * prepare adapter directory layout for upstreaming (#172) * delete adapter src/main.o: this was accidentally left out of #165 * move adapter, byte-array, and verify to a new workspace * rename byte-array crate to a name available on crates.io * add a readme for verify, also give it a slightly better name * CI: wit dep check in its own step, verify before publish, trim down publication * reactor-tests: delete deps symlinks * reactor-tests: manage wit with wit-deps * test: dont set default toolchain to nightly * wit-deps lock adapter * wit-deps lock reactor-tests wit-deps doesnt manage these for some reason * ci: add build-preview1-component-adapter step. prtest:full * temporary: always run the adapter build CI step * add ci/build-wasi-preview1-component-adapter script * put both adapter binaries in one artifact, name it bins-*, publish it * restore adapter to only building if run-full * change adapter to be part of root workspace * rm deps.lock, toml: wit-deps integration is not implemented * use latest wit-bindgen, add wildcard audits for wit-bindgen and wasm-tools crates * cargo-vet all remaining dependency bumps * cargo deny: allow the Unicode-DFS-2016 license this has similar requirements to OpenSSL: permissive, but the copyright statement must be advertised * run-tests: exclude the adapter, which doesnt build for native * adapter: compile_error when not wasm32-unknown-unknown, and note this where the static assertion fails --------- Signed-off-by: Joel Dice <joel.dice@fermyon.com> Co-authored-by: Dan Gohman <dev@sunfishcode.online> Co-authored-by: Alex Crichton <alex@alexcrichton.com> Co-authored-by: Joel Dice <joel.dice@fermyon.com> Co-authored-by: Trevor Elliott <telliott@fastly.com>

* Add basic TCP socket APIs. This is based on @npmccullum's [wasi-snapshot-preview2 draft] which is in turn based on the [wasi-sockets proposal], though for simplify for now it omits UDP sockets and some other features. It's also based on the [pseudo-streams PR]; so that should proceed first before this. [draft wasi-snapshot-preview2]: https://github.com/npmccallum/wasi-snapshot-preview2 [wasi-sockets proposal]: https://github.com/WebAssembly/wasi-sockets [pseudo-streams PR]: bytecodealliance/preview2-prototyping#29

sunfishcode mentioned this pull request Dec 16, 2022

add tests for wasi-random, wasi-clocks, and stdin #30

Merged

sunfishcode force-pushed the sunfishcode/stdio branch from 9a80044 to bd62536 Compare December 21, 2022 19:47

sunfishcode mentioned this pull request Dec 21, 2022

Add basic TCP socket APIs. #33

Merged

sunfishcode force-pushed the sunfishcode/stdio branch from bd62536 to fbf806f Compare December 22, 2022 17:33

sunfishcode added 18 commits December 23, 2022 12:05

Implement WasiStream for the pipe types.

5ff595e

Implement file streaming.

f29f2b7

Implement skip and write_repeated for more things.

06cb75a

Implement drop_stream.

79ea367

Minor simplification.

b85cc6f

Prevent FileStream from being misused.

abbe490

Implement the host side of poll_oneoff.

0b2355b

This implements pseudo-futures and subscription functions, and adds polling for streams.

Remove cfg(allow(...)) directives that are no longer needed.

b4460e5

Add more comments and minor code cleanups.

1f1d523

Add more comments and code cleanups.

ffcc3ab

Handle overflow gracefully in the polyfill clock APIs.

9d4c691

Fix an unused-import warning.

e7099d4

Delete redundant implementations.

b1d6bfb

poll_oneoff in the polyfill doesn't need mutable State.

6bd0dda

Simplify the host poll implementation.

65daab2

Add more documentation comments.

484077b

sunfishcode force-pushed the sunfishcode/stdio branch from d372516 to 484077b Compare December 23, 2022 20:07

dicej approved these changes Dec 23, 2022

View reviewed changes

host/src/clocks.rs Show resolved Hide resolved

host/src/poll.rs Show resolved Hide resolved

src/lib.rs Outdated Show resolved Hide resolved

src/lib.rs Outdated Show resolved Hide resolved

sunfishcode added 4 commits December 23, 2022 14:59

Remove the OFlags::APPEND flag, which is no longer used.

195847a

Add documentation for the default-clock functions.

ddd9632

This ports the wording from WebAssembly/wasi-clocks#22.

Use ...

a12af00

Clean up debugging code.

7dcf631

sunfishcode merged commit 106d19a into main Dec 23, 2022

sunfishcode deleted the sunfishcode/stdio branch December 23, 2022 23:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add pseudo-streams. #29

Add pseudo-streams. #29

sunfishcode commented Dec 16, 2022

sunfishcode commented Dec 21, 2022

dicej left a comment

sunfishcode commented Dec 23, 2022

Add pseudo-streams. #29

Add pseudo-streams. #29

Conversation

sunfishcode commented Dec 16, 2022

sunfishcode commented Dec 21, 2022

dicej left a comment

Choose a reason for hiding this comment

sunfishcode commented Dec 23, 2022