Adding multi-language support (multiple language runtimes)

[Angelmmiguel]

This issue tracks an ongoing effort. The implementation will be done in the related issues you can find at the end

Wasm Workers Servers aim to create serverless applications based on different languages. As a starting point, we added support for Wasm modules (compiled from Rust) and JavaScript (by embedding the QuickJS runtime). All these workers run inside the WebAssembly sandbox.

For v1.0.0, the goal is to extend the number of languages by allowing different language runtimes. Even though the size of other runtimes like Ruby and Python are not big (see our python and ruby releases), including them in the main wws is not an option.

In addition to that, we want to make wws extensible. Any developer may implement new language runtimes and use them. For this reason, wws will allow to import runtimes from different sources.

CLI experience

In this section, I will explore the desired CLI experience for this feature. All the examples assume a project has two or more workers. One is a "native" worker (.wasm) file that doesn't require any runtime. The other file requires a runtime to run (.rb or .py). 

$ tree ./examples/multi-lang 
./examples/ multi-lang 
├── hello.rb 
├── basic.toml 
└──  basic.wasm 

Repositories

A repository is an index file that points to different runtimes. It can be hosted in a git repository or in any service that can expose files. wws will retrieve the index file from the given repository. This index file contains pointers to the available runtimes. The repository is configurable via arguments, so anyone can develop their own repository. If no --repo parameter is provided, the WebAssembly Language Runtimes will be the default repo. 

The CLI experience would be: 

$ wws runtimes list 
NAME            VERSION         EXTENSION       BINARY          STATUS 
ruby            3.2.0           .rb             https://...     not installed 
ruby-std        3.2.0           .rb             https://...     not installed 
python          2.3             .py             https://...     not installed 

$ wws runtimes install ruby-std 3.2.0 
Pulling the runtime binary... 
Installing it in ./wws/runtimes/wlr/ruby-std/3.2.0/... 
Validating checksum... 
Done!

$ wws ./examples/multi-lang 

To use a different repository, you can use the --repo flag:

$ wws runtimes list --repo=https://my-repo... 
NAME            VERSION         EXTENSION       BINARY          STATUS 
ruby            3.2.0           .rb             https://...     not installed 
ruby-std        3.2.0           .rb             https://...     not installed 
python          2.3             .py             https://...     not installed 

$ wws runtimes install ruby-std 3.2.0 --repo=https://my-repo... 
Pulling the runtime binary... 
Installing it in ./wws/runtimes/my-repo/ruby-std/3.2.0/... 
Validating checksum... 
Done! 

$ wws ./examples/multi-lang 

You may point by default to a specific repository by setting up an environment variable. If the --repo argument is also present, it will take precedence to the environment variable. 

Runtime metadata files

wws will read runtime metadata files. These files include all the information required to install and use a language runtime in wws. For example, they will have a name, version, extension they can manage and a pointer to the Wasm binary. It may include other configuration parameters to ensure wws knows how to run it. You have the full list available in #65.

These files can be hosted anywhere. We will provide the default set of runtimes from the WebAssembly Languages Runtime project. However, any other file repository can be used instead in wws.

Index file

To simplify the installation process, wws will rely on an index file that will live in the repository the CLI is configured to point to. This file points to the different runtimes available on that repository, so wws can list them and install easily. By default, wws will point to the WebAssembly Languages Runtime project, although developers can change it via flag and environment variable.

Project metadata

For reproducibility, wws will create a local .wws.toml file with all the installed runtimes. This file contains all the details to retrieve the runtimes in the future. When it's committed to the application repo, wws will install the required runtimes automatically.

Issues

This list will be updated during development as well as referencing GitHub issues

• Implement the runtime metadata file (toml) #65
• Implement the index metadata file #71
• Create a struct to manage runtime metadata files #66
• Implement the .wws.toml file #67
• Add the new command to install a language runtime #68
• Check required lang runtimes when running wws (from .wws.toml) #69
• Load and run workers with installed runtimes #80
• Write the documentation and tutorials for multi-language support #70
• Create the initial repository index #85
• Add tags to the runtime metadata definition #86

[Angelmmiguel]

All the subtasks are done! We can proceed and close this task too 🎉