druntime: Treat WASI as Posix

[QuantumSegfault]

Adds WASI (or rather wasi-libc) support to core.sys.posix, and makes all WASI triples define Posix.

Also removes or adjust other WASI specific code accordingly.

[QuantumSegfault]

@thewilsonator
@kinke

[QuantumSegfault]

This is currently based on #5229. I'll rebase after it gets merged.

For now, see d699021 for the actual changes

[QuantumSegfault]

If/when this gets approved, I'll upstream it to DMD. It's easier to iterate on the functional codebase.

[kinke]

Oh wow, that was very quick - and it's green! 👍

How's the situation with CRuntime_WASI? Did you have to duplicate most CRuntime_Musl blocks, or were there enough differences justifying that version?

[kinke]

The hello-world size has increased a bit; was that one of the reasons you wanted to use the native API? I'm wondering if compiling/linking wasi-libc as an LTO lib would get rid of that overhead again.

[QuantumSegfault]

Oh wow, that was very quick - and it's green! 👍

Relatively. Worked on it like...6 hours straight (and then some). Felt plenty long at the time :p

But also, "all in a day's work".

How's the situation with CRuntime_WASI? Did you have to duplicate most CRuntime_Musl blocks, or were there enough differences justifying that version?

There were enough differences that I feel the separate version is justified. CRuntime_Musl (and occasionally linux) was the starting point for most of it, and some files have no differences, but there are collectively plenty of them, including some that are ABI breaking/changing.

Some headers are simply not supported at all, some are only available if you link in an extra library (see e.g. https://github.com/ldc-developers/ldc/pull/5230/changes#diff-48f529363632c9ca331332afb7e1b096e7396119f70ca4575abde282818a6d53), some individual functions are only available with emulation https://github.com/ldc-developers/ldc/pull/5230/changes#diff-c3629923a6cac8402dffd7abc4d4041a361be668ca72c64f1ac75a97051a9afeR96), and in unistd.d in particular, there are bunch of standard functions that are missing (so now there's a patch work of extra CRuntime_WASI blocks).

[kinke]

Impressive work. 👍 - Alright, all good wrt. NOT predefining CRuntime_Musl too then.

After a quick superficial glance, it seems like you used CRuntime_WASI also in places where there is normally an OS distinction (instead of version (WASI)). No strong opinion from my side, but if your intent was to emphasize that the Posix layer lives in wasi-libc, then I think that's just an implementation detail, and the CRuntime_* stuff is for standard C APIs, while WASI/linux/FreeBSD etc. is for the Posix APIs.

Out of curiosity, have you tried cross-compiling the importc_compare.d test, to verify the bindings?

With core.sys.posix support, porting Phobos should be WAY simpler than before. (std.net.curl will probably have to be disabled, no dynamic loading of libcurl...)

[QuantumSegfault]

The hello-world size has increased a bit; was that one of the reasons you wanted to use the native API? I'm wondering if compiling/linking wasi-libc as an LTO lib would get rid of that overhead again.

That was also part of the motivation yes (though it only decreases if you don't have any other C code using the same).

LTO MIGHT help. Might have more of a noticable impact if applied to libc as well and not just DRuntime...I can try that (default wsi-sdk builds seem to come with LTO variants, but getting -Oz requires a custom wasi-libc build/sysroot). -Oz might fight inlining too much.

[QuantumSegfault]

Impressive work. 👍 - Alright, all good wrt. NOT predefining CRuntime_Musl too then.

After a quick superficial glance, it seems like you used CRuntime_WASI also in places where there is normally an OS distinction (instead of version (WASI)). No strong opinion from my side, but if your intent was to emphasize that the Posix layer lives in wasi-libc, then I think that's just an implementation detail, and the CRuntime_* stuff is for standard C APIs, while WASI/linux/FreeBSD etc. is for the Posix APIs.

Yup, that's exactly why. If we ever get no-libc DRuntime (or realistically, an alternate libc), then I suppose that means we'd drop Posix on such targets that don't provide suitable emulation. I wouldn't necessarily be against changing it, but I did it this way specifically.

Out of curiosity, have you tried cross-compiling the importc_compare.d test, to verify the bindings?

No, not yet. I ran out of time last night. I'll try that out shortly.

With core.sys.posix support, porting Phobos should be WAY simpler than before. (std.net.curl will probably have to be disabled, no dynamic loading of libcurl...)

So far, it seems so. sockets and datetime.timezone are giving trouble...

[kinke]

If we ever get no-libc DRuntime (or realistically, an alternate libc), then I suppose that means we'd drop Posix on such targets that don't provide suitable emulation.

Right.

I wouldn't necessarily be against changing it, but I did it this way specifically.

Fine by me. [I'd expect most of these to be in version (Posix) blocks anyway, so not affecting potential future more-bare-metal druntime support.]

[kinke]

datetime.timezone

Maybe because of the reliance on available timezone data? (apt package tzdata on Debian/Ubuntu)

[QuantumSegfault]

datetime.timezone

Maybe because of the reliance on available timezone data? (apt package tzdata on Debian/Ubuntu)

wasi-libc simply has no tzname or tzset (versioned as "WASI has no timezone tables" in the header). I have to figure out an appropriate way to stub uses of it out is all.

Also yeah, no timezone files on system means that unittest can't be run

[kinke]

[As LDC-specific change, I'm using a version = Supported in multiple modules, to disable them on specific targets. They are defined internally per-module only.]

[QuantumSegfault]

Looks like only for stdcpp.

Are you suggesting renaming these to Supported instead (keeping them the same otherwise; and still upstreaming)?

[kinke]

Yeah just wrt. renaming to some consistent internal version, making it clear at a glance what it's about (since the …OrNotWASI suffix is what's relevant in 99% of cases :) when looking at the module).

[QuantumSegfault]

Okay, there's a few things I have to tweak. importc_compare to the rescue I guess.

Would never have thought to check e.g.:
Error: Type core.stdc.stdint.int_fast16_t has size 2 in C (ImportC), but size 4 in D

[kinke]

Cool that you got it working! Yeah I think this test is a showcase for D, extremely powerful and yet very compact. Most of the code I think is actually for some nice formatting, e.g., when structs layout diverge - I was pretty blown away when I saw the first failures on CI and opened that file...

[kinke]

And if you got it to pass (maybe/probably having to add some excludes to the existing lists), we could enable it for CI too (after running the druntime unittests), to make sure it doesn't regress.

[QuantumSegfault]

If you want, I can squash this just a bit so we get three clean commits. (squash the first 4 together, rename the last two a little bit so they make more sense in isolation)

[kinke]

Yep please do. 👍

[kinke]

nit: /// alignment

[QuantumSegfault]

Is that the only thing (only want to reset the CI once more)?

[QuantumSegfault]

Okay, since its been this long, I'm going to let CI finish.

I can tweak that and squash it in, and (if CI, other than circleci Ubuntu) is green, we can just merge before the next run finishes.

[kinke]

Yep, the rest is fine, just noticed this tininess while checking the 2 last commits separately.

[I don't understand why Circle still fails after #5232; it should rebase to the latest PR-merge-head automatically.]

[QuantumSegfault]

Okay. All green (sans circleci).

[QuantumSegfault]

Okay...now to upstream it.

Hopefully core.sys.posix hasn't diverged too much...might be able to just overwrite the files and revert the erroneous diffs.