wasm-compose: implement a component composition tool.

[peterhuene]

This PR implements wasm-compose, a tool for composing WebAssembly
components from other components.

Right now the tool is pretty simple: you specify a list of component imports,
specify how they get instantiated, and which of the instances gets its exports
exported from the composed component.

wasm-compose operates directly on WebAssembly components and has no
dependencies on wit or wit-bindgen.

[peterhuene]

Windows just has to be different, doesn't it? Will fix the normalization of the path characters in the test baselines.

[alexcrichton]

This is shaping up quite well, thanks for this! I left some various comments below for mostly improving some of the README docs, and I skimmed over the code and it's all reasonable. My biggest comment, though, is about the interface into this tool.

Currently a configuration file is required for invoking this tool, and correct me if I'm wrong but it seems like everything about instantiations and such are required to be in the configuration file. Historically when I've thought about a hypothetical tool along the lines of wasm-compose I've tried to draw parallels with native linkers (e.g. ld) which notably do not require configuration files. I think ideally a configuration file would not be necesary for wasm-compose.

Now that being said I don't think that the approach here is a dead end or anything close. Linkers have the "power user mode" of feeding in a linker script which has all sorts of fancy things I never understood. That's roughly how I view this wasm-compose.toml configuration file which is to say it's power-user configuration. Otherwise though my hope would be that the tool could roughly internally be architected as:

• As-is today everything executes over a "configuration"
• As-is today the configuration can be optionally specified by the user
• A new feature, though, would be that the configuration could be inferred-of-sorts from the input file.

For example I'm imagining that you could do wasm-tools compose main.wasm -o linked.wasm -L path/to/dependencies. All of the instance imports of main.wasm are matched, via string names, to *.{wasm,wat} files in path/to/dependencies. This process automatically happens transitively and effectively does a name-based inference for the [imports] and [instantiations] table. By default the [exports] table could be just the original main.wasm as well.

My hope would be that we can have pretty strong conventions that the need for wasm-compose.toml is quite rare. I don't know how best to handle the math example in this repository where string-based names wouldn't work but I could imagine a CLI flag like --define math=multiply or --define math=add which means that whenever the string-based dependency math is looked-up it would look up the specified wasm instead.

I'm curious if this aligns with your thinking about how this tool will eventually be used. One of the main benefits I think to relying on inference heavily rather than explicit configuration is that it makes it easier to pull in a dependency that you don't know a ton about because otherwise you have to have a detailed configuration file for the structure of all of your dependencies.

[peterhuene]

Now that being said I don't think that the approach here is a dead end or anything close. Linkers have the "power user mode" of feeding in a linker script which has all sorts of fancy things I never understood. That's roughly how I view this wasm-compose.toml configuration file which is to say it's power-user configuration

I think a simplified CLI-only experience for doing a basic "linking" (where imports can be satisfied automatically) of a root component is definitely needed.

However, I don't think that's the sole purpose of this tool and the need for more complicated instantiation graphs will be much more common than users using linking scripts with a conventional linker.

I think of wasm-compose more akin to docker compose; I'll explain that in just a sec.

For example I'm imagining that you could do wasm-tools compose main.wasm -o linked.wasm -L path/to/dependencies. All of the instance imports of main.wasm are matched, via string names, to *.{wasm,wat} files in path/to/dependencies. This process automatically happens transitively and effectively does a name-based inference for the [imports] and [instantiations] table. By default the [exports] table could be just the original main.wasm as well.

I think your example makes perfect sense for the simplified case where we can assume the import names must map directly to a filename base. I think I can move forward with such a requirement for a configuration-less "use these components to satisfy imports for this root component" implementation.

The general problem with this approach is that import names may be more complex than just a file name to capture versioning and origination information.

For example, tooling that pulls interface definitions from a registry might want to capture versioning and verifiability information in the instance imports of the components it generates. So names could potentially be something like bytecodealliance/foo:1.2.3:registry.example.com:07123e1f482356c415f684407a3b8723e10b2cbbc0b8fcd6282c49d37c9c1abc:.

Now perhaps this information belongs in a custom section mapping between import name and origination (one which likely should never be stripped...), but either way we're deriving some sort of inherent semantic meaning of the import names with whatever scheme we use.

And perhaps wasm-compose can't support such components without a configuration file or native support for registries; I don't think that's unreasonable either.

I'm curious if this aligns with your thinking about how this tool will eventually be used. One of the main benefits I think to relying on inference heavily rather than explicit configuration is that it makes it easier to pull in a dependency that you don't know a ton about because otherwise you have to have a detailed configuration file for the structure of all of your dependencies.

The intention behind wasmlink was to provide exactly what you described above, albeit before the component model was a thing: given a root "component", link in these other "components" to satisfy its imports and export the root's exports.

My intention with wasm-compose is to function more like docker compose, but instead of stitching together container instances as a unit, it stitches together component instances (and indeed, WebAssembly component instances can very much model a microservice in the same way) as a unit (i.e. a composed component).

So naturally this led me down the path of using a configuration file to define your (potentially complex) instantiation graph much like one would for container instances with docker compose and with much of the same functionality (i.e. integration with a registry and not specifying paths to things).

Stepping back, however, perhaps wasm-compose should focus instead of the simplified experience first since it's going to be a bit of time before we get to a place where it can function more like docker compose in a general sense.

I think we can move forward with this PR but instead of what I was going to focus on next (extending the ability to satisfying imports via instance export aliases or exporting of aliased items from instances from the root), I'll focus on a simplified wasmlink-like CLI experience.

[peterhuene]

I've started a simplified version of the tool that you can point it at a root component and it will infer everything from the local file system (and also have an option for generating a corresponding configuration file you can tweak, if desired).

I'll have a follow-up PR for that soon.

[Mossaka]

Thanks for the PR! I am really loving this tool for composing multiple components together!

I have a question about the versioning. I've noticed that there is an explicit version specified in the import dependency of math module: import = { path = "../op.wit", version = "0.1.0" }. What are the effects of specifying the version "0.1.0" here?

I first thought this version is specifying the rust cargo version, but then it felt to me that it's unreasonable to constraint wasm component interface to a language-specific version.

Then I thought this version is pointing to the wit file version, but I found no signs of version being mentioned in WIT definitions, and there isn't a formal definition of how to version WIT files.

Then I used wasm-tools print to understand math.wat and add.wat. The version 0.1.0 only appears in the math.wat to specify the instance version of op and there isn't any version in add.wat. I got even more confused, because I thought since math is importing add, there must be a version contraint to tell what version of the add component to import, but now it seems to me the purpose of this version isn't doing this.

Lastly, I found no version in the wasm-compose.yaml file.

I would greatly appreciate it if you would clarify the effects of the version in the component!

[peterhuene]

Hi @Mossaka. Thanks for the feedback! The version in the component dependency in Cargo.toml is about to be removed with this cargo-component PR. It wasn't really used for much other than to demonstrate capturing "some kind of version information" in the import names via the tooling.

We're removing that for now since it's not needed. With that PR merged, I'll update this PR's example components to the new Cargo.toml format.

[alexcrichton]

This all looks great to me! I haven't done a super detailed review of the internals here but I think it's fine to iterate on this over time in-tree now at this point.

[peterhuene]

I'm going to merge this tomorrow after updating the example to something more substantive (provided I get the various PRs in to make it all work).

After that, I hope to follow up with an update to make it easier to use (i.e. no config file) and something that supports propagating unsatisfied imports to the composed component (i.e. have the composed component import whatever the "root" component needs that didn't get linked in; should allow for host imports).