Tracking issue for supporting asm.js and WebAssembly without Fastcomp

[tlively]

Breaking Rust's dependency on Fastcomp will allow upgrades to Rust's LLVM to be much smoother because they won't depend on Fastcomp being updated. Smoother upgrades will allow LLVM to be kept up to date more easily (#42389), which will be beneficial across the board but especially for WebAssembly as its LLVM backend matures. It is necessary that the asmjs and wasm targets emit object files instead of LLVM bitcode so that bitcode version mismatches between Rust and Emscripten won't be a problem. Work that needs to be done to break the dependency on Fastcomp includes:

• finish wasm2asm and integrate it into Emscripten as a backend for asm.js (WebAssembly/binaryen#1141)
• make Emscripten take WebAssembly object files as input (blocked on LLD supporting WebAssembly https://groups.google.com/forum/#!topic/llvm-dev/BwFL_ulYX4E, https://reviews.llvm.org/D34851)
• make wasm32-unknown-emscripten emit WebAssembly object files
• update asmjs-unknown-emscripten to emit WebAssembly object files and use the future wasm2asm backend in Emscripten
• remove PNaCl/NaCl support, since it depends on Fastcomp (#42420)

[est31]

@tlively I guess #42420 would also be part of this? If so, can it be added to the list?

[tlively]

@est31 I'm not sure I understand why #42420 is part of this. Does NaCl/PNaCl support depend on Fastcomp as well?

[est31]

@tlively I'm no LLVM expert, but when I look into rust's fork of LLVM, then lib/Target/JSBackend/NaCl/ is the place I can find the NaCl backend while upstream LLVM seems to lack the JSBackend subdir entirely. I also can't find any meaningful references to NaCl in upstream LLVM so it wasn't moved or something. It looks like an addition by fastcomp.

[dylanmckay]

Can confirm - the entire JavaScript backend lives inlib/Target/JSBackend/.

Every JS-specific modification to LLVM outside of this directory is decorated like so

@LOCALMOD-BEGIN
< .. js specific code .. >
@LOCALMOD-END

It looks like an addition by fastcomp.

It is

[est31]

cc #45905

[pepyakin]

(continuing discussion from #45905)

@RReverser

@pepyakin At least for Node.js side spawning process, net etc. are pretty easily implemented. Character for path can be taken from host system and so on. I think it makes sense to experiment with reusing Emscripten libraries for this as @badboy suggested.

But what should do Web version then? Should it implement fs, net, env, etc emulation in JS? Or should it trap? If we chose to implement emulation then we will need a JS shim library, which will call "real" APIs or emulated ones depending on which "sub-environment" we running on. But then, what if user wants to run only one "sub-environment"? Or user doesn't want either fs or net, or nothing at all?
Emscripten solves this problem with preprocessing of JS files. Should we do our own preprocessing?
But should we? Given that there is emscripten which do the job already?

Applications of wasm should be truly beyond the web. Fast, safe and deterministic execution seems to be pretty desired properties!

• IoT,
• plugins, scripts, and other embeddings in larger programs,
• deterministic execution, especially for p2p (i.e. blockchain applications),
• universal drivers,
• mobile apps,
• desktop and server apps,

Some of environments can implement complete std. However, others can't.
These features can or cannot be supported:

• processes,
• filesystems,
• TCP/UDP,
• environment variables,
• stdin, stdout
• etc...

Even not all WASM features are always desired (due to performance reasons and/or need of deterministic execution):

• threads 🦄,
• gc 🦄,
• simd 🦄,
• floating points,
• growing memory,

I think, this makes wasm to be more like ISA than end user platform. (Not to mention that we even have problems of combining JS/Web and JS/Node).

To properly handle all this zoo, I think, we need something like portability lint and maybe a few more triples, like this strawman:

• wasm32-unknown-unknown - generic target that assumes nothing about it's environment. Suitable for the web.
• wasm32-unknown-node - target specially for node. I think it might support all of the std?

Well other environments might provide own implementations of these system bindings if they need to.

It will make portability lint useless, isn't it?

[badboy]

I still think having wasm32-unknown-unknown providing the bare minimum (with a libstd where it makes sense) will get us a long way.

But what should do Web version then? Should it implement fs, net, env, etc emulation in JS? Or should it trap?

It should implement shims for these things.

If we chose to implement emulation then we will need a JS shim library, which will call "real" APIs or emulated ones depending on which "sub-environment" we running on.

Yes, we will need this shim. That's the same as it works in Emscripten today.

But then, what if user wants to run only one "sub-environment"?

If it is possible to modularize these different things, it would be up to the user to load the necessary things before loading the wasm module (again: I did not look into Emscripten's shims yet and how easy it would be to extract those things).
This does not even have to be part of Rust itself btw.
But I also don't expect every module out there needing or wanting to have this environment provided (JavaScript<->Wasm interaction is still the slower thing in the whole execution).

(I'm gonna read that portability lint RFC now)

[pepyakin]

I still think having wasm32-unknown-unknown providing the bare minimum (with a libstd where it makes sense) will get us a long way.

I agree on this too. But it seems we disagree on what "bare minimum" means : ) For me, "bare minimum" doesn't include fs, net, processes, rng, etc.

That's the same as it works in Emscripten today.

This does not even have to be part of Rust itself btw.

OK, here is my direct question then: why don't use Emscripten if one needs full-blown std library with support of Web, Node? 😃

[steveklabnik]

IMO, wasm32-unknown-unknown should have no shims. wasm32-unknown-web would be something more like emscripten, with shims for a bigger chunk of functionality.

non-web applications of wasm won't want the shims, and code that doesn't use the shims won't want the shims. Keep the minimum small, build from there, IMO.

[SimonSapin]

Rather than wasm32-unknown-unknown having an std crate with most of the APIs returning an "unsupported" error, could it not have std at all? If there’s functionality in std that is supported in "pure" wasm, isn’t that a sign that this functionality belongs in another crate like libcore or liballoc?

[pepyakin]

@SimonSapin yeah, maybe.

One thing that comes to my mind is threads.

WASM threads propolsal (it's still WIP though) suggests adding atomic.wake & atomic.wait operations, which roughly corresponds to thread::park/Condvar.

[pepyakin]

There is one more point: things that only could be implemented in terms of libcore AND liballoc automatically goes into libstd. For example, io::Read and io::Write.
I can imagine usage of them in pure wasm context.

Also, if I squint enough, I can imagine implementation of HashMap outside of std.

[SimonSapin]

There is one more point: things that only could be implemented in terms of libcore AND liballoc automatically goes into libstd. For example, io::Read and io::Write.

liballoc depends on libcore, so I don’t think depending on both is a sufficient reason to have things in libstd. In the case of io::Read and io::Write it’s probably rather because they depend on io::Error, which depends on std::sys for OS errors.

As to HashMap, I believe the reason it’s not in liballoc is that its default hasher requires a pseudo-random number generator to be seeded from the OS.

[NikVolf]

HashMap being hasher agnostic and residing in liballoc would be helpful btw

[pepyakin]

liballoc depends on libcore, so I don’t think depending on both is a sufficient reason to have things in libstd. In the case of io::Read and io::Write it’s probably rather because they depend on io::Error, which depends on std::sys for OS errors.

Oh, I see. That's unfortunate.

As to HashMap, I believe the reason it’s not in liballoc is that its default hasher requires a pseudo-random number generator to be seeded from the OS.

Agree with @NikVolf on that

[SimonSapin]

I don’t know if a type item can specify a default type parameter (in the case of HashMap, S = RandomState) while still allowing it to be overridden.

[RReverser]

@SimonSapin I imagine something like pub type HashMap<K, V, S = RandomState> = core::HashMap<K, V, S> in std would work (where core variant doesn't have any default).

[SimonSapin]

It would be alloc rather than core, but yeah that might work. Who feels like submitting a PR? :)

[RReverser]

@pepyakin

OK, here is my direct question then: why don't use Emscripten if one needs full-blown std library with support of Web, Node? 😃

For me, the reason not to use Emscripten is to avoid duplication and complexity in the building toolchain and not because of the libraries. The opposite, having emulation of native APIs that works on Node.js/Web has always been the most exciting part of Emscripten to me as it allows to write fully native apps and libraries and run them on different environments without any hassle.

[pepyakin]

For me, the reason not to use Emacripten is to avoid duplication and complexity in the building toolchain and not because of the libraries.

Can you elaborate on the duplication and complexity part?

[RReverser]

Can you elaborate on the duplication and complexity part?

It's about presence of LLVM in both Rust and Emscripten when you could just use Rust directly (and LLVM versions of both currently need to be kept compatible when you use Rust's -emscripten- targets).

Also, Emscripten evolves much slower and harder to contribute to in my experience, so when you need to implement something Rust-specific for the WASM/JS output, it might take much longer to try and implement it on Emscripten side than if we had everything done only on Rust side.

[cretz]

I'll drop my two cents here as an author of a non-web WASM backend: Emscripten and the idea that JS emul needs to be Rust's concern is off. The WASM community needs to get together and create a libc-esque set of abstractions that backends can support (like emscripten or mine). In the meantime, I'd really like to not see low-level impls for a specific backend (JS/Web) be part of the Rust lang. That the shim exists already is enough. If you want JS support, do it elsewhere external to the lang.

[cretz]

@chpio - Meh, the abstraction doesn't really matter. It can just be nix syscalls like emscripten does for all I care. Just WASM import what you expect and let the backend fill em in, but of course a consistent abstraction would be nice. But the real nice thing would be if all WASM frontends could define something they share, but that seems unlikely (Emscripten basically uses libc and other libs to assume this).

I just tossed my opinion in here that it should not be part of the Rust-lang repo (or even the core devs' concern) to implement JS forms of the stdlib. I haven't yet played w/ the unknown-unknown compilation target yet, but one way would be to have a separate wasm file emitted of all the abstracted functions and just have unreachable as the one and only command for all functions. Then just import that into the real wasm file...it can be supplanted by backend authors as need be.

[RReverser]

@rpjohnst Separating web and node would be very unusual, as JS libraries are expected to work universally across different environment without any change (this is why most of Node.js libraries have their own browserified counterparts, even if they don't support all the features, just to avoid porting same code across environments but rather bundle and use out of the box).

Either way, Rust doesn't have to know about these JS implementation details, just refer to generic kernel functions as said above, so having -web or -node or any other prefix in Rust side doesn't make any sense since it won't actually provide these implementations, just expect certain C-like bindings provided by whatever is running the generated WebAssembly code.

[steveklabnik]

So, just to make sure everyone's on the same page here, WebAssembly's spec defines no platform details. Platform details are expected to be layered on top. This is specifically so that wasm can be used outside of a web context, and in fact, not only is that an important use-case overall, but an important use-case for production Rust users: Etherium plans on using wasm as their scripting language, for example.

As such, I'd very very much prefer that the unknown target implements https://webassembly.github.io/spec/ and nothing else. We should define a web target for dealing with web platform integration.

[CryZe]

The thing we are arguing is that there shouldn‘t be a -web. Maybe -std or something that has a proper std implementation, which I‘m totally okay with, if there is actually is a good reason for the target with the stubbed out std. However the fact that people are arguing for a target with a stubbed out std makes it seem that there‘s actually rather something wrong with core, which people really try to avoid.

[RReverser]

To back @CryZe's words: all the bindings we are discussing are not any specific to Web, Node.js or any other host. We don't expect nor want Rust to do any web platform integration - that can be done in custom crates and npm packages. These bindings that we need are very generic and are needed by Rust just for proper functioning of std crate on any platform, but actual implementation on the other side can be provided by absolutely any code, VM or platform that hosts WebAssembly.

This is completely orthogonal to portability lints, which might or might be not used in conjunction with such generic "kernel" bindings.

[chpio]

However the fact that people are arguing for a target with a stubbed out std makes it seem that there‘s actually rather something wrong with core, which people really try to avoid.

There is a problem with core, no one supports it.

But i like the idea of portability lints. We could even make libcore and liballoc obsolete by providing a cfg for alloc and make std and alloc switchable from the main crate.

[rpjohnst]

Separating web and node would be very unusual, as JS libraries are expected to work universally across different environment without any change (this is why most of Node.js libraries have their own browserified counterparts, even if they don't support all the features, just to avoid porting same code across environments but rather bundle and use out of the box).

@RReverser The fact of the matter is, Node supports a lot of stuff that the web does not and cannot without emulation. This means that there is a place for a -web target with a subsetted std that excludes that functionality (e.g. File::open, TcpStream, std::process) and thus does not generate imports for it.

people are arguing for a target with a stubbed out std

@CryZe To be clear, I am not arguing for a stubbed out std. (I'm not sure anyone else is either??) I'm arguing for a subsetted std that includes only what the web platform supports. That way there are three tiers to using Rust in a browser-hosted wasm:

• -unknown: You control all the imports and go without the parts of std that require any kind of support from the host (unless you explicitly add the imports and glue yourself).
• -web: The compiler generates imports and glue code for non-web-specific stuff that std needs (like @RReverser describes in #44006 (comment)), but only for things that can be provided by the web platform (e.g. Math.*).
• -emscripten/-stdweb: The compiler generates imports for everything std needs (the full "generic kernel bindings") and provides the necessary glue code and emulation on the Javascript side. The emulation need not be maintained by the Rust team, it would be considered part of the platform.

The -node target would follow the same principle- generate imports and glue code for everything std uses that can be provided by Node. I suspect this would be everything in std, so there would be no need for an -emscripten-like target for Node?

[RReverser]

This means that there is a place for a -web target with a subsetted std that excludes that functionality (e.g. File::open, TcpStream, std::process) and thus does not generate imports for it.

Far not necessary. There are cases when TcpStreams are implemented in browser with a WebSocket proxy (this actually works and is used in e.g. x86 emulators in browser), File::open can be implemented with File API or point to local storage and so on. It's not the job of Rust to decide what to expect on the other side from a dynamic language and how these APIs are implemented, this is up to the consumer of compiled WASM.

-web

but only for things that can be provided by the web platform (e.g. Math.*)

-emscripten/-stdweb

To reiterate points from previous comments - why would this be separated into -web or similar targets if these are not any specific to Web platform and same math bindings can be universally provided by other consumers like JVM or C?

[est31]

There are cases when TcpStreams are implemented in browser with a WebSocket proxy (this actually works and is used in e.g. x86 emulators in browser), File::open can be implemented with File API or point to local storage and so on.

APIs for getting the date or something do need javascript right now, and there is only one way of implementing them. If you want to be free in the choice of the file system, put it into your program! Why should rust play the abstraction layer here? Your library code should work on anything that impls the Read/Write trait, and it can be a cursor.

To reiterate points from previous comments - why would this be separated into -web or similar targets if these are not any specific to Web platform and same math bindings can be universally provided by other consumers like JVM or C?

You also have a separation between windows, mac and linux targets. There is wine, people can just use the windows API everywhere and yet they everyone wants to compile stuff natively. The reason is that then everything is tailored for your specific target.

[RReverser]

There is wine, people can just use the windows API everywhere and yet they everyone wants to compile stuff natively. The reason is that then everything is tailored for your specific target.

The reason for this is simply that you can't expect your consumer to have Wine. Better example would be Linux GNU targets for which Rust compiles just once with -linux-gnu and this works without complaints for pretty much everyone.

We just suggest to do with WebAssembly - list expected bindings for std (for what I care, most of them can be even regular POSIX function declarations) and don't put burden of knowing their implementation details neither on Rust compiler nor Rust developers.

JavaScript libraries have been moving strictly towards being universal (or also, so-called "isomorphic") over the last years, and it would be a huge step back for WebAssembly target to separate -web, -node or any other subtarget when it was intentionally designed to be universal in the first place.

[RReverser]

Why should rust play the abstraction layer here?

@est31 Because why not? Would you prefer to have duplicates of each crate that work on different targets, or a single crate that just uses std APIs and works in any environments (just like majority of Rust code nowadays)? What would you gain from knowing how e.g. Math.fmul is implemented on the other side of whatever is consuming your WebAssembly code?

[rpjohnst]

There are cases when TcpStreams are implemented in browser with a WebSocket proxy (this actually works and is used in e.g. x86 emulators in browser), File::open can be implemented with File API or point to local storage and so on. It's not the job of Rust to decide what to expect on the other side from a dynamic language and how these APIs are implemented, this is up to the consumer of compiled WASM.

This doesn't make any sense. Nobody is opposed to providing a target that includes or supports TcpStream emulation via WebSocket proxy. We already have one, and that's great! What I want, in addition, is a -web target that doesn't provide TcpStream because the web platform does not provide TCP.

Would you prefer to have duplicates of each crate that work on different targets, or a single crate that just uses std APIs and works in any environments?

You seem confused. We already cross-platform crates via -emscripten, and we will continue to have them for -web and -node for crates that happen not to use any parts of std that don't need emulation. The point of those targets is not to split the ecosystem, but to give people a way to generate wasm modules that only need a thin layer of glue without any emulation. See, for example, the lack of an ecosystem split between x86_64-pc-windows-msvc and x86_64-unknown-linux-gnu.

[est31]

Would you prefer to have duplicates of each crate that work on different targets, or a single crate that just uses std APIs and works in any environments (just like majority of Rust code nowadays)?

I would absolutely hate duplicates. But that won't happen when your library exposes a functionality to work on a Read/Seek/Write trait impl and doesnt just eat a path and serialize it on itself.

JavaScript libraries have been moving strictly towards being universal (or also, so-called "isomorphic") over the last years, and it would be a huge step back for WebAssembly target to separate -web, -node or any other subtarget when it was intentionally designed to be universal in the first place.

Again, C/JVM based targets might not get in touch with javascript at all, and some use cases certainly will not. Don't put the API at the Js layer! About unification, sure it's great, and I think most code will work both on a -node and a -web target, but then there are differences between browsers and for those you should be able to differentiate at compile time, just like you are able to differentiate today between i586-pc-windows-gnu and i686-pc-windows-gnu targets.

[RReverser]

We already have one, and that's great! What I want, in addition, is a -web target that doesn't provide TcpStream because the web platform does not provide TCP.

but then there are differences between browsers and for those you should be able to differentiate at compile time

Again, the main question: why would you want it? What does it give you? You'll get exactly same error for unsupported stuff right away when you try to import such WebAssembly with missing bindings on your platform, except you'll have a choice to stub it on your side if it's already a dependency of consumed library. This way it will be possible to distribute same WebAssembly library for all supported platforms, and consumers will just need to import it with their platform's bindings, without recompiling Rust code.

We already cross-platform crates via -emscripten,

-emscripten is very different and provides internal JS bindings for everything. I thought we're all talking here about a more universal target for WebAssembly than just JS?

Again, C/JVM based targets might not get in touch with javascript at all, and some use cases certainly will not.

Of course, that's the goal we want to enable. None of the suggested bindings are JS-specific, and as I said dozen of times above, none of them have to be - they're all just regular "kernel functions" universal to any platform.

[CryZe]

This all seems to come down to whether WebAssembly should be portable enough to just stick the same WebAssembly module into Java / Web / C / ... or if you should always recompile it. If it should be that portable, then we need one defined kernel interface that the module talks to. If not, then we should go for multiple separate targets, but completely miss out on the wasm module being portable at all then.

[RReverser]

This all seems to come down to whether WebAssembly should be portable enough to just stick the same WebAssembly module into Java / Web / C / ... or if you should always recompile it.

Yup, I think you summarised the two camps in this thread pretty well. I'm in the portable one - after all, it was one of primary goals of WebAssembly to abstract away from specific operating systems, virtual machines and so on, and provide a single binary format executable on any of them right away. Removing this abstraction and recompiling for each target seems unnecessary and makes WASM barely more useful than some C code or LLVM IR.

[est31]

@CryZe the portability that wasm is about is for me at least about being portable across operating systems and hardware, not across different embeddings. Embeddings change rarely and if they do, you can recompile stuff. I mean you can disagree with me on the importance of being portable across embeddings, but "completely miss out" is quite overblown in any case.

[rpjohnst]

You'll get exactly same error for unsupported stuff right away when you try to import such WebAssembly with missing bindings on your platform, except you'll have a choice to stub it on your side if it's already a dependency of consumed library.

@RReverser

You can get that functionality from the -unknown target just by selectively enabling stuff! The difference is that you have to opt into it in the leaf crate.

There should be a way to compile a wasm module+js glue that you can just throw into a browser and have it work. Thus, -web and -emscripten. That way you don't need to do any extra work to build a web application.

This all seems to come down to whether WebAssembly should be portable enough to just stick the same WebAssembly module into Java / Web / C / ... or if you should always recompile it. If it should be that portable, then we need one defined kernel interface that the module talks to. If not, then we should go for multiple separate targets, but completely miss out on the wasm module being portable at all then.

@CryZe

This is an inaccurate description of my argument. We can have both! We should define a standard interface that std requires. We should also allow people to target specific embeddings and generate the glue code for them.

[rpjohnst]

Perhaps an additional target or flag would be helpful- a way to get the -emscripten/-stdweb behavior in that it generates all the imports, but doesn't produce the glue code.

[CryZe]

Oh, then we must've been talking past each other, cause I totally agree with that. I think the standard interface is necessary for supporting all kinds of random embeds that rust has no specific target for, but specific targets could definitely have more specific bindings and APIs, like a js! macro and maybe some DOM APIs.

[est31]

So.... we all agree on a js macro then? I'm writing an RFC to add one to the language. I'll link it here.

[RReverser]

I'd definitely like to these to exist, but I don't really see why custom macro, DOM APIs etc. can't live in custom crates and npm packages just like any other userland code, especially integration-specific one.

[rpjohnst]

They can and should, along with targets to organize #[cfg] and compiler support.

[Diggsey]

So these extra targets would be nice, but for the current wasm32-unknown-unknown target we should just export function pointers for:

• writing to stdout
• writing to stderr
• setting an exit code
• reading from stdin?

Regardless of how the webassembly is being hosted, the host can choose to set these function pointers or leave them unset. If they are left unset then the behaviour can be as it is today. If they are set, then libstd calls them to handle each case.

This solves the current blocker for me, which is that I can run wasm tests in the browser, but I can't tell if they pass or fail... (libstd currently throws away stdout, stderr and the exit code).

[est31]

@RReverser We have multiple different implementations of such a macro. If you use one implementation, you are required to use that implementation's tool after the fact. If it is in the compiler, it would serve as minimal standard. Also, it would be usable by std. std can't depend on other crates, so having such a macro would be a requirement for targets with tailored std APIs.

About the fully dynamic target idea, I think I'm convinced now that having one would be useful. wasm32-unknown-unknown can serve as such. I'm not running out of RFC ideas...

[RReverser]

you are required to use that implementation's tool after the fact

You are probably right, for js! macro it makes sense as it essentially requires own linking step (at least for release builds, for debug builds it could just dynamically create new Function on the JS side with string's pointer as a cache index; then you could import a single FFI bindings to make macro work).

For DOM and such though, I think it makes sense to put them into own crates / packages, they don't require much from compiler or std internals and can build on top of js!.

[est31]

For DOM and such though, I think it makes sense to put them into own crates / packages, they don't require much from compiler or std internals and can build on top of js!.

Definitely, this belongs into crates. std shouldn't have any additional features than it provides on other platforms as well.

[est31]

@Diggsey made a PR implementing their idea. cc'ing everyone subscribed: #47102

[alexcrichton]

We've had a ton of progress since this was opened and I'm going to close this now as we're now quite decoupled from the Emscripten backend on master and wasm32-unknown-unknown has started picking up a lot of these elements.

In general we're going to follow Emscripten master, but this is mostly a bug for Emscripten rather than rust-lang/rust at this point, so I'm going to close.