Compile Zig (compiler) to Wasm/WASI

[syrusakbary]

It would be great to be able to compile Zig (the compiler) to the wasm32 target.

From what I understood this are the required things to make it happen:

• One of:
  • Flock support in WASI, OR
  • Print warning instead of calling flock
• CWD support in WASI Zig (it's already supported in wasi-libc: Add basic emulation of getcwd/chdir WebAssembly/wasi-libc#214 )
• One of:
  • Ship Zig just with the Zig syntax (no support for Clang at the beginning)
  • arocc merged upstream https://github.com/Vexu/arocc, OR
  • Compile LLVM to WASI https://github.com/binji/llvm-project (it's version 8.0, not sure if that would work or not, it should be easy to upgrade!)

[andrewrk]

I just pushed cd5b25c. Here are some of the compile errors that we hit right now:

$ ./zig build -Dtarget=wasm32-wasi
Semantic Analysis [10766/15600] | *"Unsupported operating system", std.target.Tag.wasi
/home/andy/Downloads/zig/lib/std/fs.zig:2222:9: error: WASI doesn't have a concept of cwd(); use std.fs.wasi.PreopenList to get available Dir handles instead
        @compileError("WASI doesn't have a concept of cwd(); use std.fs.wasi.PreopenList to get available Dir handles instead");
        ^
/home/andy/Downloads/zig/lib/std/os.zig:1764:9: error: WASI doesn't have a concept of cwd(); use std.fs.wasi.PreopenList to get available Dir handles instead
        @compileError("WASI doesn't have a concept of cwd(); use std.fs.wasi.PreopenList to get available Dir handles instead");
        ^
/home/andy/Downloads/zig/lib/std/fs.zig:2576:17: error: std.fs.selfExePath not supported for this target
        else => @compileError("std.fs.selfExePath not supported for this target"),
                ^
/home/andy/Downloads/zig/lib/std/fs/get_app_data_dir.zig:70:17: error: Unsupported OS
        else => @compileError("Unsupported OS"),
                ^
/home/andy/Downloads/zig/lib/std/os.zig:4635:9: error: Use std.fs.wasi.PreopenList to obtain valid Dir handles instead of using absolute paths
        @compileError("Use std.fs.wasi.PreopenList to obtain valid Dir handles instead of using absolute paths");
        ^
/home/andy/Downloads/zig/lib/std/os.zig:179:24: error: container 'std.os.wasi' has no member called 'R_OK'
pub const R_OK = system.R_OK;
                       ^
/home/andy/Downloads/zig/src/libc_installation.zig:251:23: error: expected type 'libc_installation.FindError', found '@typeInfo(@typeInfo(@TypeOf(std.process.getEnvMap)).Fn.return_type.?).ErrorUnion.error_set'
        var env_map = try std.process.getEnvMap(allocator);
                      ^
/home/andy/Downloads/zig/lib/std/os.zig:4983:5: note: 'error.Unexpected' not a member of destination error set
    Unexpected,
    ^
/home/andy/Downloads/zig/lib/std/os.zig:4582:26: error: container 'std.os.wasi' has no member called 'flock'
        const rc = system.flock(fd, operation);
                         ^
/home/andy/Downloads/zig/lib/std/os.zig:149:25: error: container 'std.os.wasi' has no member called 'pid_t'
pub const pid_t = system.pid_t;
                        ^
/home/andy/Downloads/zig/lib/std/os.zig:1548:25: error: container 'std.os.wasi' has no member called 'execve'
    switch (errno(system.execve(path, child_argv, envp))) {
                        ^
/home/andy/Downloads/zig/lib/std/child_process.zig:526:39: error: container 'std.os.wasi.O' has no member called 'RDWR'
            os.openZ("/dev/null", os.O.RDWR, 0) catch |err| switch (err) {
                                      ^
/home/andy/Downloads/zig/lib/std/os.zig:4205:18: error: container 'std.os.wasi.O' has no member called 'CLOEXEC'
    if (flags & O.CLOEXEC != 0) {
                 ^
/home/andy/Downloads/zig/lib/std/os.zig:1314:15: error: container 'std.os.wasi' has no member called 'open'
        system.open;
              ^
/home/andy/Downloads/zig/lib/std/Thread.zig:425:9: error: found compile log statement
        @compileLog("Unsupported operating system", target.os.tag);
        ^
/home/andy/Downloads/zig/lib/std/os.zig:4171:25: error: container 'std.os.wasi' has no member called 'pipe'
    switch (errno(system.pipe(&fds))) {
                        ^

It looks like we're actually pretty close already to me. Just need to tackle these one at a time.

[kubkon]

What about process forking? We spawn child processes at the very least for linking, right? That's a no-go in WASI as far as I remember (unless something has changed, but then that would go against the entire philosophy of capabilities-based security).

[kubkon]

Pulling in @Luukdegram and @jedisct1 as they might be very interested in this plus their expert opinion will be invaluable.

[andrewrk]

What about process forking?

• zig test can just print "unable to spawn test binary on the WASI platform" or "test binary created at [path]" and exit success.
• zig run can just print "WASI does not support spawning child processes" and exit failure.
• for clang we can invoke clang_main directly instead of as a child process (for WASI only)
• for lld we can invoke the lld library functions instead of as child process. (for WASI only)
• same thing for zig ar, zig dlltool, etc.
• for detecting native libc, just return error.LibcNotDetected or whatever the appropriate error is.

I think that's everything.

First things first, let's get it working without -Denable-llvm and then go from there.

[zigazeljko]

How will the compiler locate the lib/std directory?

[andrewrk]

That is a great question. The normal way that Zig finds its lib/ directory is by searching relative to its own executable path. It does not seem that WASI supports such an operation.

A similar question is how will the compiler locate a suitable global cache directory? The normal way it does this is by looking for an environment variable such as ZIG_GLOBAL_CACHE_DIR, XDG_CACHE_HOME, or HOME. It looks like WASI does in fact support environment variables, so that will be fine, provided that such paths are made accessible to the WASI runtime.

I'm guessing how this will go down is that the WASI package has the ability to specify parameters to the webassembly runtime, and so it would specify the cache directory in the package, as well as the installation directory, and make both of those paths accessible to the WASI runtime environment, perhaps as preopens. On the Zig side of things, if we need to add an environment var to control installation directory, we can. If we need to add support to specify that a specified cache directory or specified installation directory is a preopen (in other words a file descriptor that is already open) we can.

The first step here will be to get a build without compile errors, that errors out at runtime unless --zig-lib-dir is provided.