WASI libc prototype implementation.

[sunfishcode]

Announcing WASI, a new system interface proposal for WebAssembly.

This PR contains WASI libc, a libc implementation built on top of WASI.

It incoporates pieces from musl-libc, cloudlibc, cloudabi, libpreopen,
and dlmalloc, as well as a significant amount of new code. See the README and LICENSE files for details.

For documentation on WASI, see https://wasi.dev.

[sbc100]

Does this bring this repo into line with https://github.com/CraneStation/wasi-sysroot? Is the intention to have this one supersede wasi-sysroot?

I'd be interested in somehow tracking changes fomr upstream musl. Does this PR contain the whole of musl? Would it make sense to sub-module it (i.e. does it contain purely new files, or also modifications to existing ones). Perhaps we could fork + sub-module? I don't feel super strongly about it but I'm generally not a fan of duplicating the musl tree.

[sunfishcode]

Yes, this PR contains what is currently in https://github.com/CraneStation/wasi-sysroot. I'm thinking it does make sense for that repo to be superseded by this one, and to possibly also rename this one to https://github.com/WebAssembly/wasi-sysroot. But I'm open to suggestions.

This PR does contain the whole of musl, and includes extensive changes to musl source files. It is distinctly a fork, though I do expect to track upstream changes. I agree this has downsides, and I do believe in upstreaming code in general (WASI support in LLVM is upstream, for example), however upstream musl is explicitly specialized for Linux, and WASI is distinctly not Linux.

In an attempt to partially ease the pain of maintaining a fork, I used an #ifdef technique that I've found useful in other projects I've worked on. It looks like this:

#ifdef __wasilibc_unmodified_upstream
       // Unmodified upstream musl code here.
# else
       // WASI-specific code here.
#endif

This allows us to preserve all the lines of musl's code, and have it coexist with the WASI code side-by-side. This can make it easier when reading the code and trying to understand the intent of the musl code, and to better understand how the WASI-specific changes modify it. That said, this technique also does add clutter, and I'm not strongly attached to it, and am open to opinions here.

One thing that I haven't done yet, but which might make sense, is to remove files from the musl tree that we're not using. For example, we're using the openat from cloudlibc rather than the openat from musl, but both are in the tree.

[sbc100]

Sounds good. I would support renaming this repo.

And also agree that given its a fork with extensive changes, having it live in tree probably makes sense. Did you consider a literal fork which would contain all of the musl history along with your change, that you could then sub-module into this repo?

[sunfishcode]

I did, though in my experience the awkwardness of git submodules outweigh the advantages of having the musl history in the tree. I expect we'll occasionally need to make changes to both the musl and non-musl parts of the codebase together, which is much nicer to do with a single repo.

[sunfishcode]

Ok, unless there are any objections, I plan to merge this PR soon, and request the repo be renamed to wasi-sysroot.

[binji]

This seems OK to me (landing and renaming the repo). Only concerns I have are these:

• The PR here is very big, it's hard to see what's new and what's from upstream. Not just for musl, but other libraries too. I'm a little concerned that it will be difficult to track why the changes were made from upstream (i.e. what was the rationale). Any ideas how we can make this clearer?

• Seems like the licenses should all be OK, though I don't know that we've seen any MIT before in the WebAssembly repo, so I'm curious if there are constraints there.

[sunfishcode]

My best idea for making it easier to understand the code here is to add more comments. As I mentioned above, all changes to upstream musl code are annotated with #ifdef __wasilibc_unmodified_upstream markers, and I've now added a commit adding comments to every such marker in musl mentioning what they're each for.

Concerning the license; if it helps, the WebAssembly org hosted lld, which is NCSA-licensed, which is similar to MIT. And waterfall depends on musl though doesn't directly include it. Otherwise, I'll follow up with folks offline to see what our options here are.

[binji]

Thanks, I didn't realize you added the #ifdefs to the other upstream projects too, that helps.

[sbc100]

I started to integrate this into the wasm waterfall: WebAssembly/waterfall#492

[sbc100]

So far it looks like a great starting place. Are you planning on squash this PR when you land it BTW?

[sunfishcode]

Cool! I was thinking I wouldn't squash the PR at this point, as the commits here carry meaningful history.

[binji]

As mentioned in the May 14 CG meeting, the choice of software license is up to the project, as stated in the CLA document.

Any source code created by the Project is not subject to this CLA, but rather subject to separate licensing terms for that source code.

[sbc100]

Do we want to ahead with the move here? It still seems reasonable to me. Especially since the main waterfall is now a consumer.

[binji]

@sbc100 yes, I think we should land this. @sunfishcode any concerns?

[sbc100]

We've been discussing this in the wasi sub-group. I don't think we reached an agreement yet. Also the repo was renamed to wasi-libc in the meantime, so we'll want to rename this one if we do the move, or delete it.

[sunfishcode]

As discussed at the 2019-11-12 CG meeting, I'm now moving WASI libc into the WebAssembly organization. Instead of this PR, which is too big to be meaningful anyway, I'm just going to remove this mostly empty repository and transfer the WASI libc repository in, which will preserve the existing issues and PRs.