-
Notifications
You must be signed in to change notification settings - Fork 85
WASM support
Bacalhau currently targets POSIX-based, Docker-based data science people – it is about doing generic compute jobs that are not about the machinery of running a blockchain (like FVM is). Docker-based jobs work well for this but are somewhat heavy-weight for some use cases. What we are aiming at with this work is something more like "serverless for IPFS".
Juan's ultimate vision is (paraphrased) 'ultimately we want different WASM modules that together implement an execution system [like Bacalhau]' – i.e. a resilient global compute substrate that is upgradable. So we want to be able run WASM code from CIDs... and for it to support dynamic linking e.g. running functions from other CIDs. A.k.a "I should have just put V8 in IPFS…". This is the vision that IPVM are pursuing, above. This work is the first step towards that, but probably more for the learning rather than the code.
This is gonna become more important when incentives are invovled, and we need it to be easy to prove that what one execution is producing is the same as another. x86 contains a lot of sources of entropy that we have to find and switch off, which is error prone. WASM should be easier to start with a blank substrate and only expose features that are deterministic. When people start paying for execution, we may only want to support deterministic jobs [through WASM].
FVM also runs WASM, but that framework is about running code on the chain itself (i.e. to move things between wallets) and isn't designed to scale to generic compute that can read any data from IPFS CIDs.
As an example, we may want to transform some data (contained in IPFS/Filecoin) by e.g. joining some other data onto it or computing some aggregate over it. With the correct interface, it should be easier to write a very simple WASM-based script to do that. So WASM could be something more like a data transformation device. Ultimately, we are not sure what users will want to do with it, but it should provide an approachable interface for building higher level things.
We currently support running Python code by compiling it to WASM and then running it in a container. We shouldn't immediately shut the door to providing that sort of functionality with this module.
But we are allowed to write lots of executors. So we don't need to make this the perfect WASM interpreter that can run anything for all time. Instead, lets start with something simple without constraining ourselves later.
This is being developed in #664.
Broadly: download some data with this CID, download some code with that CID, load up that data and code and run it to get some output, and now store that into IPFS.
We will need:
- A storage provider that streams the bytes of a file from a CID and can list the entries in a CID.
- Work out how to provide those syscalls to WASM – is the answer WASI, or something else? Use the same one as the FVM team…
- For now write stuff into STDOUT, but we will also need to write to arbitrary IPFS later.
- Make this simple to use, with some user-facing commands.