Spin out host generators into their own repositories

[alexcrichton]

This was discussed at the wit-bindgen meeting last Friday, and after some futher discussion with @pchickey I've come up with a proposal which I personally feel is the direction that wit-bindgen should go in. I wanted to open an issue though to get feedback from others, however.

To recap, the core issue at hand here is that the Wasmtime repository would like to vendor an implementation of wasi-preview2. This would necessitate Wasmtime depending on wit-bindgen for host bindings generation, however, but wit-bindgen depends back on the Wasmtime repository. This circular dependency between git repositories cannot be resolved and isn't a maintainable way to implement this code.

After much delibration and possible ways to go about this, I would like to propose that all host generators be "split out" of the wit-bindgen repository into their own repositories, in time possibly doing the same thing for guests. Concretely what I'm envisioning is:

• Move out wit-component and wit-parser into the wasm-tools repository. This sort of low-level tooling is the "base" of the dependency tree and would live next to all our other low level tools. We could create commands such as wasm-tools wit-validate or similar to validate the structure of *.wit files or generate the markdown documentation that's there today. No host/guest generator would live in the wasm-tools repository. Tests here would solely be about name resolution and syntactical correctness of *.wit files plus the *.wat tests that wit-component has today.
• Move the host wasmtime rust generator to the Wasmtime repository. This would depend on the crates from wasm-tools and would be tested with raw *.wat inputs, unlike today where it's tested with compiled guest inputs.
• Move the host wasmtime python generator to the wasmtime-py repository (exact location and naming TBD).
• Move the host JS generator to its own repository with naming TBD. The idea here is that this might make more sense as a hub for component tooling compiled to wasm to use in JS, for example wit-bindgen as-is today could be compiled to a *.wasm and published to NPM from this location.
• Leave the guest generators inside of the wit-bindgen repository, for now. While these may eventually move to their own repository it seems like it may make the most sense to leave them in one location now for discoverability and in general there's no pressing need for splitting them out. Guest generators would be tested with a host generator.
• Finally use the published-to-crates-io version of the rust guest generator to test the wasi-preview2 implementation that will live inside of the Wasmtime repository.

With this organization it solves the original problem of implementing and testing wasi-preview2 in the Wasmtime repository by having the Wasmtime generator live there and being able to depend on the Rust generator from this repository. Otherwise the benefits of this organization I believe are:

1. Clear spaces for language experts to live. No one has to "care about everyone else" in the sense that I don't think anyone wants the current issues where everyone implementing anything has to care about all hosts/guests simultaneously. Ideally ecosystems should be able to grow at their own pace and take inspiration from each other but it's becoming quite a burden to try to keep everything in-sync.
2. Clear locations for issues/growth/etc, each generator can grow independently of the others. In practice there's really not all that much code shared other than wit-parser and wit-component so I don't think there will be too much meaningful duplication.
3. Clear ways to grow each repository in custom per-language directions, such as integrating the wit-bindgen macro for Wasmtime into Wasmtime itself or integrating NPM tooling into the JS host generator. I can also imagine that the JS guest generator would have a more natural place for hosting a custom build of spidermonkey.

Despite all this I don't think that this is universally a net win. There are still a number of downsides to this new organization, chiefly among them the loss of discoverability. There will no longer be a single place to go and interact with wit-bindgen and supported languages. This can be somewhat recovered with a page of documentation but it will likely not be quite as good as the story of today. In reality, though, wit-bindgen is not currently documented for its language support and having such documentation anyway would be a relatively large undertaking to keep maintained over time.

Another major issue though with splitting repositories is versioning. It's not possible to atomically update all wit-bindgen-related repositories meaning that there will be a number of wit-parser versions in the wild and possibly incompatible host/guest generators. The hopeful solution for this, though, is strictly defined interchanges between guest/host generators. By using the official component binary format as an interchange along with *.wit which is also defined in the component-model repository itself it's hoped that most of the churn will be of a non-breaking sort where the Rust APIs to wit-parser may change but the underlying text format will not change as much. Nevertheless it's expected that this will be more of an issue than it would otherwise be with a monorepo-style approach.

Finally we don't actually have dozens of folks clamoring to maintain everything in wit-bindgen at this time. This means that splitting out some repositories runs the risk of actually decreasing maintenance in the short-to-medium term due to the same people being spread more thinly across repositories. I don't really know of a great answer for this other than "hope it gets better over time".

---

Personally I feel that this is the "least bad" option at this time. I think it also has a number of benefits over the current organization, though, slightly outweighing the downsides I can think of. Despite all that though I'd like to confirm with others that this is a reasonable approach to take and see if others have ideas as well.

If this is the route decided on (splitting host generators out) then I don't think that anything will be split out before worlds are implemented in the text format, so it will still be some time before this is all implemented.

[alexcrichton]

bytecodealliance/wasm-tools#819 contains the first step of this, splitting out the wit-{parser,component} crates.

[alexcrichton]

bytecodealliance/wasmtime-py#104 and #436 are the extraction of the Python host generator.

[alexcrichton]

#439 is the removal of Wasmtime, moved upstream as part of bytecodealliance/wasmtime#5317