RFC: Externref lang item for Wasm targets

[guybedford]

This RFC proposes a new externref lang type (core::arch::wasm32::externref, feature(wasm_externref)): an opaque, unforgeable reference to a WebAssembly host value, lowering to the Wasm externref reference type in function signatures.

Following the semantics of Clang's __externref_t (shipped since Clang 15), it is a bare-position-only type, legal solely as the top-level type of function parameters, return values and local bindings and enforced at type-check time. This allows host references (e.g. JavaScript values) to be marshalled from one foreign call to another directly enabling better performance and interoperability between Wasm ecosystem crates.

Rendered

[programmerjake]

why forbid externref in closure arguments/return values? forbidding it in closure captures is reasonable since you can't store it in structs but a capture-less closure should behave the same as a function -- where externref is allowed.

View changes since the review

[guybedford]

You're correct that this could work mechanically, but it is a consequence of the blanket generics ban - FnOnce<Args> has Args = (externref, ...) which is banned as a generic externref type.

Note that my original version of this proposal only allowed externref in extern block foreign function definitions and extern "C" fn exports. That could still be a possible reframing of this proposal to avoid the complexity of arbitrary function scenarios.

[clarfonthey]

My immediate gut feeling toward this is that simply having the type "opt out" of the type system is a hammer that feels like it will lead to loads of annoying hacks, but the amount of restrictions on externref feel like they wouldn't be possible to cover via other means, which is a compelling reason to not support it.

Essentially, listing all the requirements:

• Cannot be referenced (no &-refs, or even &raw-refs)
• No defined layout (similar to extern type)
• No Type ID (!Any)
• Cannot be moved onto the stack (!Move, except…)
• Can be arguments or return values of C or Rust ABIs

Note that stuff like "never instantiated as a generic" could be useful as lints, but as type-system guarantees they seem fragile.

Note that being ABI-only have interesting consequences, like you could define repr(transparent) structs with them, but it would effectively be stuck with whatever struct you used at the ABI level, based upon what you've defined your extern fns as, which could be helpful.

Basically, I think that it's a steep bar to potentially satisfy all these requirements with explicit type-system details but it feels a lot better than just "this sidesteps the type system entirely" which is gonna lead to so many awful hacks.

View changes since the review

[Nadrieril]

Also, without a lot more opsem work:

• Cannot be the type of a struct/enum field (except maybe transparent structs)

[clarfonthey]

So, I kind of felt that would be implicit due to "no defined layout" but we can add it explicitly too. (I guess that without layout it could still technically be bolted onto the end of a struct but I'm not sure extern types allow that, especially if alignment is undefined.)

[Jules-Bertholet]

For a proposal like this one, if you want the slightest chance of acceptance, you cannot afford to skimp on this section or the Rationale and Alternatives.

First, do not assume everyone in your audience has more than passing familiarity with WASM, or any idea what an externref is. You will need to explain this in the RFC itself. RFC 3884 is a good example of providing background for a niche target feature.

Second, to convince the lang team to accept an invasive and controversial change like this one, you will need to, at minimum, prove that nothing less can meet the needs of WASM users. You cite two major motivations; you should elaborate on both:

• Interoperability and fewer custom wrappers: What are the specific patterns that break because of lack of interoperability, that would become possible with this proposal? And in the Rationale, why can't interoperability be achieved another way, e.g. standardizing on a common table system (similar to the C ABI and other tool conventions)? Ideally, give examples from specific real-world projects.
• Performance and optimization: What are the specific usage patterns you anticipate for this feature (taking into account the many limits you impose on how externref can be used)? And in the Rationale, why is this proposal the only reasonable way to meet the performance requirements of those specific patterns? Ideally, you should cite benchmarks from a real-world project.

View changes since the review

[Jules-Bertholet]

Note that whether or not this RFC ends up accepted in something like its current form, none of this work will be wasted. By fully describing the problem space, you enable everyone working on Rust to consider how what they are working on or thinking about intersects with Wasm's specific needs, and the end result will be a language that better supports them.

[Jules-Bertholet]

Please elaborate here as well.

• Clang / LLVM __externref_t: What was their motivation for doing this? Are people using and enjoying this feature despite the restrictions on where __externref_t can appear? Did they face any unexpected challenges, and how did they overcome them? Link to their proposals and documentation.
• "wasm-bindgen externref mode: post-link signature rewriting via walrus": What is this? Why is it "motivating precendent"?
• Wasm-GC languages: So, is this feature necessary to do FFI with those languages on WASM? If so, that's great motivation! Make it part of your case, don't be shy about it. How is "storing [externref] in language-level aggregates" represented in FFI?

View changes since the review

[Darksonn]

Could we change wasm to make this type less special?

You've explained that the table representation is currently bespoke, which causes interoperability problems. But why can't wasm just pick one and standardize it on the wasm side? Then it'd no longer be a bespoke representation, but just the standard ABI and we can build that into the language and support it as a completely normal type with integer size.

View changes since the review

[guybedford]

This is covered in the future possibilities section briefly but I could expand on it further.

Basically, the shared table would support &externref as the storable variant as an additional feature (__externref_t* in llvm/llvm-project#201466) This layers onto the base externref support though which itself keeps all the strong guards defined by this RFC.

So the design of this externref remains a requirement even when &externref exists - none of the work in this proposal is wasted and the layering works out naturally.

If we made externref just an alias for __externref_t* ignoring support for __externref_t then that would force all FFI values to be table indices, which is exactly the situation we are trying to get away from here.

If we made externref a magical alias that picks one or the other representation depending on usage position, that would break a whole bunch more invariants in having a far more complex Heisenberg type.

[RalfJung]

The semantics of this RFC cannot be supported by the Abstract Machine that we have developed so far to talk about Rust. That's a big setback since that Abstract Machine is needed for us to confidently answer questions like "is this MIR transform correct" or "are we correctly translating MIR to LLVM IR". In other words, as-is this RFC robs us of the ability to precisely reason about MIR, which is inacceptable. This isn't even properly discussed as a drawback, and no attempt is made at sketching what the Abstract Machine for Rust could look like with externref. You can look at MiniRust as a basis if you want to get an idea for what an Abstract Machine for current Rust could look like. (The RFC talks about the fact that this breaks type system assumptions, but that's only a small fraction of the problem. We've had RFCs that break type system assumptions before. This RFC breaks Abstract Machine assumptions. We've never had an RFC anything like this.)

Usually the approach to represent "external resources" in a language like Rust is something similar to file descriptors: a number into a table that is managed outside the realm of the language and that we can only talk about indirectly. The RFC mentions tables but only as "third-party" options. But a table does not have to be third-party, the intrinsics required for that could be in libcore. Given the gigantic cost of the proposed solution, the RFC needs to spend a comparable gigantic amount of space and effort explaining why the table solution, even if it became a first-party concept with good language and library integration, is so bad that we have to undertake the biggest change of the language since Rust 1.0 in order to avoid it.

View changes since the review

[RalfJung]

There 's a major drawback that's not discussed enough IMO: none of the existing language and library ecosystem works for this, meaning it will be a major pain to do anything with these types. You cannot return them from Result functions, you cannot use them in async blocks/functions, you cannot use == on them, you cannot newtype them... this RFC is proposing to introduce a fundamentally new kind of thing to Rust only to then make that thing a complete nightmare to use for anything.

The result of this will not feel like "Rust supports externref now". It will feel like "externref has been bolted on to Rust in a poorly integrated unergonomic way".

View changes since the review

[guybedford]

It's an FFI feature primarily, yes.

[guybedford]

Would it help if we formally restricted externref to only extern "C" and extern block functions? That would still provide the major benefit IMO.

[Jules-Bertholet]

I don't think that makes a difference. As long as values of this type are passing through Rust code, we need a full semantics to understand what is happening.

It's an FFI feature primarily, yes.

As I've suggested elsewhere, please please provide a full fleshed-out example so that those of us who don't know Wasm can fully grok your use-case. (Take all the time you need, don't rush it)

[ds84182]

You could skip all of the language changes if externref naturally decayed to an index within a table when it needs to be stored to memory. This needs runtime support. Similar to panic handlers, the compiler would depend on externally provided table allocation and free functions.

Otherwise I don't think this RFC will make any reasonable progress.

[programmerjake]

You could skip all of the language changes if externref naturally decayed to an index within a table when it needs to be stored to memory.

I think this is the third time that something like that was proposed -- I proposed it on Zulip. I do think that having the Rust type be an index into a table except when passed by value in function args/return values (maybe limited to extern "C") is a good idea, and that LLVM can help with optimizing to remove redundant table allocations/reads/writes.

[RalfJung]

Would it help if we formally restricted externref to only extern "C" and extern block functions? That would still provide the major benefit IMO.

One can define and call extern "C" functions from Rust so on its own that doesn't really help.

I think this is the third time that something like that was proposed -- I proposed it on Zulip. I do think that having the Rust type be an index into a table except when passed by value in function args/return values (maybe limited to extern "C") is a good idea, and that LLVM can help with optimizing to remove redundant table allocations/reads/writes.

Yeah that makes sense: have this behave like a normal type with a special ABI. ABIs already involve special concepts we cannot represent in Rust, like registers. Having a type that's a table index normally but a bare externref on the ABI will stretch the limits of our already overstretched ABI handling in rustc, but at least that is "just" an implementation concern.

[RalfJung]

wasm-bindgen's JsValue does not interoperate with other custom table wrapping systems.

So let's define a standard table system for Rust that's used across the ecosystem, and let's design it in a way that it can be picked up by other languages as well (or can be adjusted to eventually achieve interop with other languages, once anyone else is interested).

View changes since the review

[juntyr]

While the proposed design would mirror externref exactly, I think a non-zero cost wrapper could work better. The wrapper would still be special in that any stores or loads to memory would be translated into stores and loads into a Rust-owned table. As an optimisation, a pass could then look for cases where an externref in fact only passed as a function parameter and remove the table calls. In that case, Rust users could use an almost-normal opaque type, and cases that are written to comply with externref's usage rules would be optimised to compile to the same code as using this restricted proposal

View changes since the review

[guybedford]

In this scenario, how would you distinguish between the opaque form, and actually getting the index of an &externref on an FFI boundary? Or would externref mean different things depending on usage site? How to categorize those cases and avoid &&externref being amgiuous?

[juntyr]

My idea would be that externref in Rust-only code is effectively a table index, but that at FFI boundaries (when defining or calling extern "C" functions), the actual WASM externref is passed, with table index conversions happening right at the boundary. A reasonably clever optimisation pass could then see that

unsafe extern "C" {
    fn create_ref() -> externref;
    fn use_ref(v: externref);
}

pub fn run() {
    unsafe {
        let r = create_ref();
        use_ref(r);
    }
}

which would lower to a externref -> table insert -> table extract -> externref sequence, could be replaced by just passing the externref directly.

[ds84182]

If anything, I don't think Clang should be a role model. __externref_t is a non-standard extension to C and C++ implemented in Clang to handle a bespoke target feature. Other compilers, like GCC and MSVC, do not have this (or any type like this).

More generally, I do not think languages should change at such a core level to support something extremely niche like this. WebAssembly has so many oddities that have to be papered over due to fundamental issues in its design. Those oddities should be dealt with by the compiler and/or runtime.

View changes since the review