ENV module aliasing

[guybedford]

I'm interested in compiling C to WASM in a way that allows me to alias the 'ENV' import for external functions ((import "env" "external_func"...) output in wast).

The natural thing I'd like to be able to do is to supply a function attribute in C that associates the external function with an import module name in WASM - something like:

void custom_logger(int val) __attribute__ ((import, name ("./custom-logger.wasm")));

This repo seems to define the "env" as being the default import, along with an importMap system, although I couldn't tell how that could be configured.

Does something similar to this already exist, or does work along these lines sound like a worthwhile effort to pursue through the toolchain?

[kripken]

Is the idea that you'd import one wasm module from another? You'd need to specify a function name then, not an entire wasm module, unless you're suggesting an approach where there is a whole module for each function?

In general in the current toolchain, every unsupplied symbol becomes an import from env. Generally people then implement those in JS (using --js-library etc.). For implementation in wasm, that could be done using dynamic linking, e.g. dlopen, which works (but it not optimized).

[guybedford]

Potentially for importing a pre-built wasm library from wasm, but a better example might be loading a js function (say a logger) where we know it already exists in a compiled form in js. The function name can come from the same name as the c function, although a custom symbol could be provided for that as well, but I'm not sure it's strictly necessary flexibility.

It's just about controlling the 'env' name when it is known to be another resolution during the linking process, to get some analog to c-style static library linking.

Let me know if that makes sense?

[kripken]

I don't think I understand yet. Where do you replace "env" in your example void custom_logger(int val) __attribute__ ((import, name ("./custom-logger.wasm")));?

[guybedford]

The example workflow would replace 'env' in the output with './custom-logger.wasm' (should really use '.js' in the example though), where 'void custom_logger(int val)' is a function definition only.

[guybedford]

To clarify the exact workflow, the idea is that the following:

void custom_logger(int val) __attribute__ ((import, name ("./custom-logger.js")));

where custom_logger is treated as an external symbol, provides the WAST output:

(import "./custom-logger.js" "custom_logger" (func $custom_logger (param i32)))

The process of setting this env path could be an option somewhere in the workflow as well, it just seemed like an attribute carried through the chain of tooling the easiest from what I could tell.

[kripken]

I see. Yeah, seems like an option to allow customizing the module name for an import makes sense. Curious what other people think.

[guybedford]

@sunfishcode I'd be interested to hear your feedback here if you can. It's just about ensuring we have the tools for managing this compilation boundary down to the runtime semantics, in whatever form.

[dschuff]

This sort of capability is pretty much the sort of thing we've been thinking about defining along with the linking work.

At a high level, we want a way to define:

• What functions are exported when a module is linked (this would probably include some C++ syntax and a linker capability; see also Valid imports and exports on the web #585)
• What the "module name" of a linked module is. This would at least include a JS capability because you explicitly provide this mapping from import module name to module when you instantiate a wasm module. But it might also include some sort of metadata on the module, so the JS loader code knows what the module thinks its name is
• How undefined functions are handled when linking, and what module name imported functions have after linking. This would also include both C++ syntax and linker capability, and is what you are talking about here.

Obviously how this is designed depends on the overall static and dynamic linking scheme is expected to work. We've been working on the linking bits in https://github.com/WebAssembly/tool-conventions/blob/master/Linking.md and in the LLVM/binaryen/wabt code and haven't gotten as far as the C++ syntax yet but

1. we are getting close enough now that it's probably worth bringing it up again and
2. it makes sense to have the syntax also work as an "escape hatch" to operate outside that system. Because it totally makes sense to say "I just want to provide some JS module and function and make it work myself", which is kind of what I think you are talking about here. Also emscripten's various wasm<->JS binding mechanisms would presumably build on this primitive too.

[dschuff]

oh and along with 2 above, I think the syntax you propose here is pretty much along the lines of what we've been thinking about too.

[sunfishcode]

As far as I'm aware, the "env" module is just something that works for simple cases right now, rather than being part of a more elaborate plan.

The way two-level namespacing works on Darwin is that, at static link time, the linker resolves symbols that will be provided by dynamic libraries, and records which library each symbol is resolved by. I expect we'll eventually have something similar to this, where the source code doesn't specify module names, and then linkers can rewrite the module name afterwards when it's determined which modules will be providing which symbols.

Do we also need some way for source code to specify module names up front? It would certainly be useful right now, while most of the tools aren't mature and there aren't many other options. Longer term, it might be a little redundant with the linker module detection, but it might continue to be more convenient for some use cases.

[dschuff]

Yeah I agree that linker module detection would be the expected way for things to work, but I also think it would be useful to have the "escape hatch" to allow the user (or some source-level framework) to specify how particular symbols are imported.

[guybedford]

To give some background, I'm the author of SystemJS and node-es-module-loader, both of which provide the ability to load WASM modules alongside ES modules in the browser and NodeJS respectively.

As I've been experimenting with build tooling for WASM, I've been defining this boundary by using visibility attributes on functions exactly as in the #585 approach which works perfectly to my needs, and then this exactly handles the module name component.

Certainly this is a process that should be handled by the linker and overall build tooling, where the resolution should be determined by the tool - so an attribute shouldn't necessarily be used directly by end-users... but if the attribute were to be supported through the toolchain, that would then enable a path for the linker tooling to use this attribute to communicate to LLVM potentially? Then it just happens to provide a path forward for tooling today.

So my question is - does a function attribute make sense as the unit of management for link tooling in future, and if not, what other mechanism might there be?

[sunfishcode]

The use case of typical C/C++ code that doesn't have module names, and that relies on the linker to assign module names, wouldn't use attributes. In this scenario, the linker doesn't need to feed any information back into LLVM etc.; it just needs to add module names to the imports of the output wasm module.

So, I think the answer to your specific question is that no, the attribute can't be the exclusive unit of management for link tooling. However, it may yet be worth having, along side other tools.

One question I have is about the meaning of module names. A filesystem path like "./custom-logger.js" is probably very handy for simple cases, because you can just provide that file yourself at the appropriate path in the filesystem. But it's another matter in the case of using libraries written by different people. If you're using a library, do you want it using attributes to specify paths to where it expects various modules will be at runtime? Will module names differ if you're using node versus running in a browser? I haven't thought about this deeply and don't know what the answers are.

[guybedford]

The linker in C/C++ is well-established such that resolving a symbol can be assumed a standard protocol. We don't have this same thing in the world of the JS/WASM runtime - rather we have NodeJS module resolution and browser module resolution, and then compilers and build tools all have their own configurations and options for customizing this module resolution.

I think it will be important for LLVM to be able to serve this need for WASM compilations to be able to fit into the JS ecosystem in this way. This would mean accepting that LLVM shouldn't be a source of truth for determining the output import names, but rather should allow these to be tuned from tools that drive the process around LLVM. It is possible in JS (and likely also in WASM) to build modules without even having external modules present at build time as we don't need binary information as the export interface is already clear.

A function attribute would lend itself well to being driven through tooling, alternatively some ability to provide a manifest describing the import boundary and what import names should be emitted.

I really think some path here is needed, and it would be great to see how progress can be made. I'd be interested to hear further what would be the best way to go about this, whether is it being taken care of, is it not yet necessary or ready, or if can I assist in any way?

[sunfishcode]

I'm not opposed to the attribute. I'd like to understand it a little better.

Would you mind spelling out a use case or two in a little more detail? Who would typically pick the module names, and at which phase in the build pipeline relative to C/C++ compilation? What information do they include (filesystem paths, versioning, prefixing/mangling, or anything else than a simple identifier)?

Do module names ever need to be different depending on the environment the code will be run in? For example node versus browser? Or, runtime deployments with differing filesystem layouts?