Size and duplication considerations in WASI API design?

[kripken]

The wasi path_open API incurs some size overhead, it appears. Measuring in wasi SDK output, around 1K for the preopen and related support. Experimenting in Emscripten, there is also overhead from flag conversion since the open flags+mode must be converted and split into wasi's dirflags+oflags+rights+fs_flags; looks like for us a naive implementation would add almost 1K (most on the JS side where we need to unconvert things; that might be optimized with a larger refactoring, but it's unclear).

That overhead is unfortunate in embedded systems, the Web, and other places where size is a high priority. Basically, path_open requires a bunch of permissions work to be done in "userspace" inside the wasm. (And to some extent this is not strictly needed work as the VM must monitor each path for access anyhow.)

Perhaps embedded/Web/etc. would benefit from a more minimalistic API, something closer to open(), saving that 1-2K? That does raise questions like

• Is Wasi ok with overlapping APIs for different scenarios?
• How much does Wasi care about code size considerations?

[sunfishcode]

Is Wasi ok with overlapping APIs for different scenarios?

I think it can be. For example, as we're discussing in #107 it may make sense to have an HTTP API even if there will eventually also be a sockets API.

How much does Wasi care about code size considerations?

So far the focus has been more on questions like "what is an API?", and now that this settling down a bit, we're thinking about "what is a file?" and "how should OCAP work?", and since these could lead to significant changes, we haven't put much focus on optimizing code size yet.

The libpreopen code comes from a different project with diffferent use cases, so it turns out to be more general than wasi-libc needs it to be, so it could be simplified. path_open's interface could probably also be simplified if someone wanted to take a close look at it. It would also be possible to allow some programs to avoid linking in libpreopen altogether.

It's not clear if you're suggesting adding a plain open which would support absolute paths. If so, that would be a significant shift in the sandboxing story for WASI. I'm open to discussing that, but we should be clear about what we're discussing.

[sbc100]

I think it might be worth taking another look at the benefits of the openat + libpreopen approach. It does seem to push extra complexity into libc/userspace. An i'm not sure the sandbox model WASI wants matches the cloudabi one in this way.

With WASI/wasm the embedder already has to effectively create a VFS and check/verify all FS accesses explicitly. In the current model it seems that both the embedder and libpreopen/libc have do duplicate this work of modelling the VFS and I'm not sure I see a great benefit.

The wasi VFS could work more like a docker container FS where userspace is exposed for a VFS constructed by the embedder. This VFS might or might not correspond to a part(s) of the embedding host FS, but that can be completely opaque to the WASI module.

I don't think a traditional open() would preclude an OCAP model. Files could also be opened via openat on directory nodes (capability objects) that a modules can send and receive. A module could , for example, share its roof FS by passing around the result of open('/').

[sunfishcode]

openat-style design is preparing for finer-grained sandboxing. The approach of creating a virtual filesystem where the application has the ability to open absolute paths, you get isolation between the application and the system, but you don't get any isolation between one part of an application and another.

Of course, with integer file descriptors, we still have forgeability. However, on one hand, it is easier to reason about file descriptors where the only valid values are those returned from WASI APIs than about paths which are strings which can legitimately come from anywhere. But also, more importantly, in the future languages which can use reference types will be able to take advantage of unforgeable references for very powerful and very fine-grained sandboxing.

And while it's true that current implementations of path_open do a fair amount of work to prevent .. and links from escaping, there are OS-specific ways to let OS's do much of this work for us, so I expect implementations won't do as much work in the future.

[sbc100]

openat-style design is preparing for finer-grained sandboxing. The approach of creating a virtual filesystem where the application has the ability to open absolute paths, you get isolation between the application and the system, but you don't get any isolation between one part of an application and another.

If you want to isolate one part of an application from another isn't the wasm module boundary the sensible and logical unit of isolation? Are you suggesting that one of our goals should be to able to create and enforce such boundaries within a single module?

Assume the boundary is the wasm module then can't we achieve fine grain sandboxing by giving each module its own VFS at the module level? I can image one module starting another module with a limited or empty VFS, and then passing fd/capabilities to it.

[kripken]

It's not clear if you're suggesting adding a plain open which would support absolute paths. If so, that would be a significant shift in the sandboxing story for WASI. I'm open to discussing that, but we should be clear about what we're discussing.

As an example of "duplication" here I do mean variations on the sandboxing story. Some environments may want the extra overhead of OCAP, while others may be happy enough with sandboxing at the VM or module boundary, as @sbc100 suggests, and benefit from something more lightweight.

But this isn't just about OCAP and sandboxing. Duplication also comes up in the graphics context - yes, WebGPU is an excellent starting point, but if you want the most optimal thing on say the XBOX, you presumably want DirectX bindings.

Do we want wasi to have "one" sandboxing, one graphics API, etc., and leave more lightweight/optimal alternatives to non-wasi APIs? (So XBOX might have wasm DirectX bindings, the Web might have a non-OCAP filesystem API). Or should those be options inside wasi?

[sunfishcode]

Are you suggesting that one of our goals should be to able to create and enforce such boundaries within a single module?

Yes :-). This is part of what capability-based API design, as we put in the high-level goals means. It's a big part of what a lot of people are excited about in WASI.

Duplication among WASI APIs isn't necessarily a blocker. It depends on whether people want to do the work of standardizing them, and whether the Subgroup decides the duplication is worth it.

It's hard to answer in general, as it will often depend on details of individual features. For APIs like DirectX, it would likely depend more on the vendor-specific nature of the API, rather than API duplication per se.

the extra overhead of OCAP

It's important to have a consistent sandboxing story across WASI. We see this sandboxing as a natural extension of the base WebAssembly sandbox, which is also not optional, even in areas where it adds overhead.

Also, wasi-libc still has a lot of low-hanging fruit at this point. It's new, and there's a lot of room for improvement.

[sbc100]

Are you suggesting that one of our goals should be to able to create and enforce such boundaries within a single module?

Yes :-). This is part of what capability-based API design, as we put in the high-level goals means. It's a big part of what a lot of people are excited about in WASI.

Oh, that contradicts the understanding got from the way Mark Miller (@erights) described OCAP and how WebAssembly modules could be used to implement it. Perhaps I misunderstood. @erights can you confirm, do you want to be able to have trust boundaries within a single WebAssembly module? (i.e. is the WebAssembly module a fine-grained enough security boundary?).

[sbc100]

(sorry I accidentally edited your comment.. hopefully its restored to its original form now).

[kripken]

It's important to have a consistent sandboxing story across WASI. We see this sandboxing as a natural extension of the base WebAssembly sandbox, which is also not optional

Yeah, I definitely get that there are upsides to that approach; it's a reasonable design!

At the same time, forcing OCAP has the downside of more non-WASI APIs - concretely, we may avoid path_open in Emscripten, or make it non-default (to save 2K as mentioned earlier). A world in which WASI included the more lightweight APIs too would be a more standards-based one.

(I'd prefer the more standards-based outcome myself, where WASI supports the lightweight stuff too. But I guess it's not the end of the world if we end up with de-facto standard non-WASI APIs.)

[tschneidereit]

The OCAP sandbox design based on explicit capabilities is fundamental to WASI. This design enables finer grained intra-module security for languages that support unforgeable references, but that's not the main reason for it—the ability to clearly define which capabilities a module has access to is, with "none" being the default.

This becomes even more important in multi-module setups, where you'd want to be able to pass on subsets of capabilities, potentially in attenuated form, from one module to another.

I think it's premature to talk about changes to the WASI subgroup's charter—which the proposals here would require. Certainly, such a change would have to be supported by very strong arguments and data (and, as @sunfishcode points out, investigation into e.g. optimizations to wasi-libc) demonstrating that use cases of vital importance can't be met without changing the charter.

[sbc100]

I agree we could investigate possible libc optimization. Perhaps there are ways to remove the current dependence on libpreopen with changing anything fundamental. For example, if we defined just a single filesystem OCAP object at startup time. Then we wouldn't need to maintain the libpreopen mapping as part of libc, and applications could still open subdirectories and pass around attenuated capability object.

The goal here is to reduce the amount of boilerplate code that every wasi program needs to include in order to build its initial model of the filesystem. The same goes for argv and envp handling. Its seems unfortunate we can't rely on the embedder to set those up ahead of time in linear memory. Or maybe we are focusing too much on code size at a few Kb isn't worth sweating over?

[kripken]

I think it's hard to avoid overhead due to OCAP in path_open when disallowing absolute paths, because it forces you to include code to (1) look for the file's parent and corresponding permission; and (2) you must preopen the parent so it will be found. That's where the current overhead comes from. Maybe I'm missing what could be optimized here?

A few K can be pretty important in some contexts (even in game engines; see Oryol) especially since extra overhead from multiple causes adds up. And of course in places like the Web and embedded systems size matters quite a lot. But yes, in most server use cases it's negligible. I guess the bottom line is it's hard to do the optimal thing for all use cases with a single solution :) (but again, I definitely get the benefits of a single and consistent solution as well!)

[AndrewScheidecker]

I think it's premature to talk about changes to the WASI subgroup's charter—which the proposals here would require. Certainly, such a change would have to be supported by very strong arguments and data (and, as @sunfishcode points out, investigation into e.g. optimizations to wasi-libc) demonstrating that use cases of vital importance can't be met without changing the charter.

Can you elaborate on this? The WASI charter doesn't seem to have any bearing on this issue.

I think path_open being equivalent to openat is actually not the problem here: I think it would be fine for Emscripten libc to require that a single directory, /, is preopened. That would allow use of absolute paths without all the libpreopen code that wasi-libc has.

[tschneidereit]

Can you elaborate on this? The WASI charter doesn't seem to have any bearing on this issue.

You're right, I misremembered how the various docs are structured. What I was referring to is the high-level goals document, which, same as for WebAssembly overall, contains the "how" bits of what the sub-group is working on, whereas the charter contains the "what" bits—in this case, a system interface :)

It might make sense to make this document more visible.

I think it would be fine for Emscripten libc to require that a single directory, /, is preopened. That would allow use of absolute paths without all the libpreopen code that wasi-libc has.

I agree that that'd work. Whether it'd be "fine" depends on Emscripten's goals: it'd entail giving up on most aspects of portability, and some of the most important aspects of WebAssembly's sandboxing. That might be the right trade-off for some applications, but it seems like a bad default for the vast majority of all use cases.

[AndrewScheidecker]

Whether it'd be "fine" depends on Emscripten's goals: it'd entail giving up on most aspects of portability, and some of the most important aspects of WebAssembly's sandboxing.

I'm not proposing that folks preopen the host filesystem /, but that they preopen some directory as the sandboxed root.

[sbc100]

Whether it'd be "fine" depends on Emscripten's goals: it'd entail giving up on most aspects of portability, and some of the most important aspects of WebAssembly's sandboxing.

I'm not proposing that folks preopen the host filesystem /, but that they preopen some directory as the sandboxed root.

Right, that some directory might not even correspond to a actual host directory. The embedder is free to construct it however it likes. It could be a pure memfs for example, seeded from a tar file.

However, IIUC, alon is claiming there is some overhead in userspace even if we only have single root fs preopened, so that doesn't solve the problem.

Also, if an application only opens the first preopen fd and ignores any other ones provided by the embedder I'm not sure it could be considered confirming to the spec, as it would only have a partial view of the filesystem compared to what the embedded intended.

[kripken]

Also, if an application only opens the first preopen fd and ignores any other ones provided by the embedder I'm not sure it could be considered confirming to the spec

That's exactly my concern, yes. Even if it somehow technically counts as conforming, it would be surprising for users in practice.

So the best options seem to be either using full preopening with the overhead, using a non-wasi API, or adding a lightweight API to wasi. One of the last two is what I'd prefer for emscripten (and if there is user demand we could eventually add a non-default option for the first perhaps).

[tschneidereit]

The wasi path_open API incurs some size overhead, it appears. Measuring in wasi SDK output, around 1K for the preopen and related support. Experimenting in Emscripten, there is also overhead from flag conversion since the open flags+mode must be converted and split into wasi's dirflags+oflags+rights+fs_flags; looks like for us a naive implementation would add almost 1K (most on the JS side where we need to unconvert things; that might be optimized with a larger refactoring, but it's unclear).

From this description, it sounds like there is a large amount of optimization potential still. The preopen code in wasi-libc isn't optimized for size at all so far, so that'd be an obvious first step. Another one would be a non-naive implementation of the JS parts. If those two steps still leave a significant amount of overhead, we could look into far less significant API changes, such as changing the flag handling, but not the semantics.

Is there a reason to consider radical changes to WASI's security model before these steps have been taken and the results analyzed?

[devsnek]

you also don't have to use libpreopen. for example if you've got seven files all in rom on some embedded device, you can generate a tree of permissions ahead of time, and skip the entire libpreopen dance.

[sbc100]

I could be wrong but AFAICT, if you skip the pre-open dance I don't think you end up with a portable WASI binary.. i.e. you won't be able be run-able by wasmer, wasmtime, etc who's users who want to be able run any binary with --dir arguments and have them be honored.

[kripken]

@tschneidereit I agree a radical change to the security model would be a big deal. But after yesterday's in-person meeting though I am more and more optimistic we don't need to propose that here! :)

Specifically, in the meeting some API options for HTTP were mentioned. The open command took a string there. I asked why that's different than the file fd_open command, and that discussion led to some interesting points, by @sunfishcode and others,

• For files there is sort of a hierarchy, so openat might make sense in a way that doesn't for HTTP URIs.
• However, even without a hierarchy, it seems like a permission per domain or such could make sense. So maybe there would be preopened domains?
• But this met with widespread opposition from the HTTP proposals, as it would not be consistent with any existing APIs in that space.
• And even for files, the hierarchy argument is just one perspective (I think @sunfishcode made this point), as openat assumes the parent is open, but do we want to assume all parents are open?
• There was wide agreement that code size is important in this space, and having preopened files, URIs, graphics devices, audio devices, etc. may be bad for that.

So there are multiple ways to think about OCAP and how it applies to different APIs. To clarify the discussion in this issue, I think we all agree OCAP on fd operations makes total sense - fd_write etc. should receive an FD which operates as an unforgeable reference. The question is what to do with actually getting the initial reference to anything.

I think for both files and URIs (and graphics devices etc.!), a natural approach is to need to ask for those permissions. So open of a network connection takes a string, open of a file takes a string, etc. This lets the initial permission work be done in the runtime, saving size in wasm files. It's a simpler model than requiring being told of your permissions, which is kind of what preopening is (where you are told your initial permissions, then do work in the wasm to ask for later permissions based on those).

[devsnek]

@sbc100 yeah you're right, that's why I specified where the wasi was being deployed to. sometimes you're just looking for platform portability, not runtime portability.

[lukewagner]

I'm not an ocap security export but, iiuc, APIs that create capabilities (file descriptors et al) out of arbitrarily-synthesizable (i.e., foregeable) bytes (including strings), relying on the system to accept or deny at that point, is almost exactly the antithesis of what a capability-based security model is; that's an ACL approach.

Instead, I think what's worth digging into is whether the design of WASI (and associated initialization ABIs) can be improved so that a .wasm module can portably say "I only care about receiving exactly one directory capability, the root directory" so that it can incur no overhead in the generated wasm; it can simply implement open(str) by calling path_open(rootDir, str) with no libpreopen. I.e., maybe producing a WASI-conformant module doesn't force you to accept (and map via libpreopen) an arbitrary sequence of preopened files at arbitrary paths.

[kripken]

@lukewagner Thanks, that sounds interesting to look into.

Before that, though, I wonder how it would fit with the current thinking on network APIs. It suggests we'd need a wasm module to say "I only care about establishing network connections with domain X", or something like that? The current proposals appear to violate your understanding of OCAP.

cc @pchickey

[tlively]

I had another compromise idea. WASI engines could be expected to ship a libpreopen polyfill that runs in userspace and is linked to user programs that would like to use more traditional APIs such as open (and potentially other functions that create capabilities from forgeable data). OCAP is not violated because the polyfills are in userspace and code size is saved by shipping the polyfills with engines instead of binaries. This is similar to the idea of having both low-level APIs and high-level, possibly polyfilled APIs, except that the high level API must be polyfilled to maintain OCAP.

[pchickey]

@kripken re the network APIs, where the strawman I presented yesterday uses a string to represent a URI: I didn't really think through that aspect in terms of following OCAP design principles the whole way through. I do strongly agree that there should be a way to allow the user's code to take a string representation of a URI and somehow validate that the system will allow them to make a request to that URI, but I'm not sure if the design I have right now expresses that at the right level of granularity.

I could imagine following the principle @lukewagner uses above - uri_open(str) -> Result<UriHandle, Error> could be implemented in terms of some other function that passes a capability corresponding to URI resolution, and rather than having any collection of URI resolvers preopened, we could statically indicate that only some default resolver is desired. I welcome more feedback and input on this, but maybe it should be moved to a different thread to keep this one on topic?

[kripken]

@tlively Very interesting idea! Sounds good to me in terms of fixing the size issue. @lukewagner does that conform to your understanding of OCAP?

@pchickey Thanks! Yeah, I agree any specific details probably belong in another issue. I just wanted to avoid this discussion being about a single API, so the network perspective is helpful I think.

[sunfishcode]

I'd like to understand the impact on code size in more detail. Alon, would you be able to post the code you add which show the code size impact?

[kripken]

@sunfishcode Sorry, I don't understand what code you're looking for?

As mentioned in the first comment, I simply built something (a tiny program using open) using the latest wasi SDK (with optimizations of course) and then inspected the binary for code size (using wasm-opt --func-metrics), and I added up the sizes for the relevant preopen things, and I saw around 1K.

(The emscripten code size measurement was more involved, but I assume you didn't mean that?)

[lukewagner]

That does seem like a potential alternative, but I increasingly wonder if the --dir argument shouldn't be removed altogether, replacing it by allowing a wasm module to declare what it wants from the host on startup by specifying an exported start function that is allowed to have a non-empty parameter list by which individual files, lists of files, or directories can be passed. (To wit, I've also been imagining the same thing for a while as a replacement for __wasi_args_get(), which seems like a pretty awkward way to pass arguments.) Lots of interesting questions to answer here, though.

[MarkMcCaskey]

A bit late, but at Wasmer we've been using the virtual root described above as fd 3 and put all preopened directories into it. It currently has some properties that on second thought are probably not what we want, like open(preopen1_fd, "../preopen2/some_file") being a valid way to access other preopened directories (the permissions are still checked, but it gives forgeable string access to preopened directories), but overall it's made things a lot simpler for us.

Being able to do arbitrary logic with the preopened files and directories has been very useful for trying out extensions to WASI. For example, having an embedder expose virtual files to the WASI module with arbitrary logic backing them. So that's something that I want to make sure we keep.

[sunfishcode]

@tlively's suggestion above is something I've thought about a fair amount, and my sense is now that it's not something we need to focus on right now.

For many users, the code sizes we're talking about here are quite small and not significant.

For those for whom it is, if they also want to run their code on the Web, they can always compile separately with Emscripten to target the Web. This is somewhat less convenient than compiling once, or compiling twice with the same toolchain and different flags. But let's be clear: what we're talking about here is saving a small amount of code size for people unwilling to do a modest amount of extra work.

And, anyone who wants to save even more space can modify their application to avoid using filesystem APIs altogether. If you're running on the Web, you ultimately don't have a real filesystem, so any amount of POSIX semantics is overhead. So, let's be clear: what we're talking about here is saving a small amount of code size for people unwilling to do a modest amount of extra work, and who are unwilling to modify their application in a way that would not just avoid this whole issue, but also decrease their code size even further.

[tlively]

I was going to say that portability is another good reason to go with the approach, but thinking about it more, you’re right that non-web users wouldn’t mind shipping their own portability layers. The only exception might be embedded use cases, but those won’t be dependent on POSIX file system APIs anyhow.

[kripken]

@sunfishcode That's fair.

I somewhat disagree about the term "modest", since it may take significant work for people to support another toolchain and/or refactor their code for this.

I also somewhat disagree with the implication of "users should do some work to optimize" when the reality is that many users will just do the easy thing even if it's suboptimal (since they understandably have lots of other more urgent things), so I believe it's up to us toolchain people to make the easy thing more optimal.

But yes, I agree it's reasonable for WASI to say that this level of code size focus is not what it wants to focus on. I do recommend stating this in the design doc.

[sunfishcode]

There exists a small fixed-size code size optimization, which doesn't apply to some users, is too small for many users to care about at this time, and which in practice can be obviated in multiple ways ways for users for whom it's really important, and which we could do in the future when we have more advanced tools, but which is awkward to do automatically today without distracting from our other goals.

I recognize that not everyone works for an organization with practically unlimited engineering resources. But some issues don't benefit from being studied in a microscope where we ignore the context.

[sunfishcode]

We care about code size a lot. As an example, over the last several months I have made a series of optimizations for code size, and by my measurement, the overhead of the libpreopen overhead discussed here has shrunk to 911 bytes. To put that in perspective, in terms of code size alone, this makes it a much lower priority than providing an alternative malloc, since wasi-libc's current malloc is several times larger than that. And it's a lower priority than finishing the LTO feature, since for some users, LTO can deliver much much greater code size wins than that. And there are a lot of other things we could do, such as moving some timezone and other logic out of libc. There are even ways we could move some parts of the printf code out of libc and into WASI calls (which in JS polyfills could use JS formatting to implement, as Emscripten did at one point, but I think we could make it easier to use by adding an LLVM optimization to help).

Concerning the overhead of flag conversion code, it sounds from your comment like that's using a naive implementation. I'd be very interested if someone wanted to dig into this and understand how we could change our flags to provide better code size.

And if anyone has any WASI use cases where WASI's current code size is a concern, please file an issue or otherwise reach out! In the context of specific use cases, there are often more options available.

[sunfishcode]

The discussion here seems to have subsided; please reopen or file new issues if there are further things to discuss.

WASI cares about code size, but it cares about other things too.