WASI-based filesystem access

[zkat]

All the use-cases I might want to use extism for (a shell, a package manager) involve fairly heavy filesystem access, and I think it would make a lot of sense to provide filesystem access to plugins (with configurable scope limitations). I saw on the site how you were looking for feedback on this front so you can consider this an official feature request for the functionality. Extism looks awesome and I'd love to make use of it!

[nilslice]

@zkat - thank you for the feedback, and I appreciate you taking the time to read the blog post especially at the level to notice the call for input!

These are super interesting use cases, and I absolutely see the need for filesystem access. There is a lot of consideration on this subject, and we want to make sure that however we add support for this, that its done in a way that leads users to safe defaults and no surprises.

Some questions I have for you:

1. Would you expect to be able to apply filesystem access on the Context level, or on an individual Plugin instance? In either case, regarding the configurable scope you mention - do you have a rough idea of what the API could look like? Its not uncommon to use some kind of host-path:guest-path mapping, similar to Docker volumes, and that's the initial sketch we've considered. Interested in what you might be looking for.

2. As far as the general desire to work with files more natively, is the goal:

• Ergonomics from the plug-in author perspective
• Use of existing posix-like implementations in available libraries
• Performance / memory usage (vs. copying specific files in/out of the guest/host barrier)
• Something else? All of the above?

Any additional detail about those points above will help us a lot!

[zkat]

1. In my mind, the way this would work is for users to be able to provide a list of plugins they want to load, which provide custom behavior for them. For the configurable scope, let's take the case of a package manager: I'm ok with plugins accessing the current project path, and its subdirectories (maybe with some specific subdirs excluded in general?), but nothing higher than that directory, to prevent malicious plugins from doing any damage outside of the current project, executing programs, etc.
2. I think looking at what Yarn uses plugins for is a good source of inspiration for what sort of abilities one might want a plugin to have. Answering questions more specifically:
   • Ergonomics: I think plug-in authors should be able to read and write files with language-native APIs (as made available by WASI support).
   • Using existing libraries feels like an important use-case. For example, a plugin might be a wrapper around an existing library that just glues it into the larger tool.
   • Perf/memory usage: when it comes to returning large amounts of data, it might be better for the host to read files directly, with the plugin fns simply returning instructions on what file to open (though they could verify that, e.g. the files already exist). Still, it might be convenient to have the ability to copy/stream that data across the barrier. I do think this is less important than the ability to read/write files at the plugin level and just return structured final results.

[nilslice]

This is very helpful, thank you!

Agree, the Yarn examples are great inspiration. I think we could support a much wider set of use-cases with file access.

@zshipko put together this API, which by default still disallows file access, but enables a user to incrementally build up host/guest filepath access. Since we think of this as a feature that requires more control over the runtime, we've added the API to the Manifest type.

What do you think? PR #137

[zkat]

If I understand it ok, then this is pretty much exactly what I'd want: for extism to disallow all filesystem access by default, even if wasi happens to be enabled, and for explicit paths to be passed in through the config/manifest at the host level that plugins will be given access to. That seems like the most secure but also reasonable way to do it!

[nilslice]

That's great! Ok, I think we will move forward with this and get it merged. Thanks for your feedback 🙏

[zshipko]

This has now been merged, including the fix for Windows 🎉Thanks for your help!