Using WASI for off-chain workers

[olanod]

Hi! maybe it has been considered already but I'll create this for reference ... what if instead of using magical macros and exposing custom off-chain-worker only interfaces to the WASM binaries users could plug their WASI binaries that do some random computation and things just work.
Perhaps is to early to use WASI for the immediate future since the initial MVP is quite limited with the number of APIs that are available but in the long run it should be the way to go! :) waiting for standards to be defined or people to agree with proposals often can take long but to me looks like the right path to take, it's an ecosystem that will evolve a lot and substrate can benefit from the work of others just like with the choice of Rust and WASM.
Even in its early stages it already starts to become easier to develop a WASI app(e.g. cargo-wasi), imagine I develop a standalone binary that takes some input through stdin and spits out to stdout, I can test it in isolation outside of the blockchain world, it can be an existing cli that does something useful and when plugged to substrate it works because substrate is smart enough to recognize its output, or the binary is passed a flag/env var to tell it to output things in a specific format, or there's some content type negotiation, a metadata file, or a set of non-mangled public functions are exposed as well using the proposed interface types to tell substrate what to do with that binary ... anyway there are ways to make it work 😅
Emm sorry, WASI is really gets me exited 🤩:stuck_out_tongue_closed_eyes: imagine my wishes come true(WebAssembly/WASI#53 & WebAssembly/WASI#59) and from an off-chain worker some day we end up being able to use gpu APIs to do heavy parallel work or running a hardware accelerated neural net, or take it even further, I wanted to prototype a wasi app that uses a directed graph to glue wasi binaries(wasi within wasi) to create a gstreamer-like pipeline where data flows from binary to binary probably just using stdin/stdout like how openfaas turns cli tools into web services, then the off-chain work becomes more of a plug and play/drag and drop thing that can be done graphically in the same way substrate hopes to enable blockchain creation by easily combining modules. 🤤

[gnunicorn]

This and other interfaces are definitely things we consider, but running an live-externally-upgradable-code-base comes with a range of security considerations to take into account. The same reason we are not actively allowing std to be run in Wasm, exposing system-interfaces to non-audited decentralised code comes with a huge attack vector. In particular if a malicious validator ever produced an upgrade-block, it could use all these interfaces to expose information, including reading the keystore from disk and read and expose private keys of the validator it attacks. For the time being we will probably not allow any direct system-interface-access, but we are happy to see experiments play with it!

[olanod]

I understand, makes total sense to be very careful with the security implications of exposing any system interface but if anything I trust more the security model of a (in the making)standard that is in the eyes of the whole world and a bunch of industries, not only blockchain ones and is expected to be as secure as possible. The sand-boxing, high modularity and capability based security model of WASI makes it ideal for this kind of use cases, embedders are free to chose which modules they include and what capabilities they give to the wasm binaries, substrate can start small and conservative if it wants to allowing only a small set of interfaces even if fancier ones are already standardized and implemented(or leave it to the blockchain author to decide or to the chain governance ;)).

[bkchr]

Not planned.