BREAKING CHANGE: ESM, JS bindings, triple equals and general cleanup

[dcodeIO]

ECMAScript modules

Fixes #1306. Migrating our dependencies to ESM made it possible to migrate the compiler and frontend as well, alongside internal utility and scripts. This is a breaking change for consumers of the compiler API (i.e. transforms), but not for command line usage. Viable strategies to account for the changes are:

1. Migrate any code utilizing compiler APIs to ESM as well:

   import assemblyscript from "assemblyscript";
   ...

2. Utilize a dynamic import while consuming code remains in its current module format:

   const assemblyscript = (await import("assemblyscript")).default;
   ...
   // or
   import("assemblyscript").then(({ default: assemblyscript }) => { ... });

Running the compiler on the Web with ESM

Prior, we provided a browser SDK that made use of the AMD module format to load the components necessary to run the compiler on the Web. With the switch to ESM, it is not necessary anymore to provide a separate SDK, but native browser functionality can now be used to utilize the compiler in browsers. To ease the transition, the build system outputs an example dist/web.html template with all the right versions in <script ...> tags. Alongside, it also sets up es-module-shims for import maps support as is currently necessary to support browsers other than those based on Chromium. General outline is:

<script async src="url/to/es-module-shims"></script>
<script type="importmap">
{
  "imports": {
    "binaryen": "url/to/binaryen",
    "long": "url/to/long",
    "assemblyscript": "url/to/assemblyscript"
  }
}
</script>
<script type="module">
import asc from "url/to/asc";
...
</script>

Asynchronous compiler APIs

Prior, programmatically executing for example asc.main was synchronous despite accepting a callback. The relevant APIs are now asynchronous, making use of the opportunity to perform asynchronous I/O under the hood, and as such return a promise with the compilation result instead of accepting a callback.

const { error, stdout, stderr } = await asc.main([ ... ], { ... });
...

It is no longer necessary to override stdout and stderr, since the default has been switched to return a memory stream. If binding to the Node.js console is desired, which is rare, the behavior can be overridden to the previous by specifying { stdout: process.stdout, stderr: process.stderr } in API options.

In addition, transform hooks gained the ability to evaluate asynchronously when being async / returning a promise.

Unification of === and ==

Fixes #856. Closes #1111. The semantics of === and !== are redundant in AssemblyScript since comparing two values of different types is not permitted anyhow. In the early days of the compiler we hence repurposed the operator to perform what seemed like a useful operation, that is test for identity equality (the exact same object). Doing so, however, introduced a subtle footgun into the language for those coming from TypeScript, and now has been removed.

Means: === is now the same as ==, !== is now the same as !=, largely matching intuition.

Deprecation of the loader

The loader was a stopgap solution intended to keep glue code minimal until WebAssembly integrates natively with the rest of the Web platform. Sadly, we made a bet there that didn't materialize, and the loader has now been deprecated with the compiler now supporting static generation of glue code with the --bindings option.

Available bindings are:

  --bindings, -b        Specifies the bindings to generate (.js + .d.ts).

                          esm  JavaScript bindings & typings with ESM integration.
                          raw  Like esm, but exports just the instantiate function.
                               Useful where modules are meant to be instantiated
                               multiple times or non-ESM imports must be provided.

Generated JavaScript bindings support the data types

Type	Strategy	Description
Number	By value	Basic types except 64-bit integers
BigInt	By value	64-bit integers via js-bigint-integration
String	Copy
ArrayBuffer	Copy
TypedArray	Copy	Any kind
Array	Copy	Any kind
StaticArray	Copy	Any kind
Object	Copy	Plain objects (no constructor or non-public fields). Can opt-out by providing an empty constructor.
Internref	By value	Reference counted pointer via FinalizationRegistry. Essentially anything that isn't a plain object.
Externref	By value	Via reference-types

For now, only top-level functions, globals and enums receive bindings. Classes do not (yet). Bindings automagically utilize exported runtime helpers so users don't have to.

Preserving side-effects in static type checks

Fixes #531. The static type checks isInteger, isFloat, isBoolean, isSigned, isReference, isString, isArray, isArrayLike, isFunction, isNullable, isConstant, isManaged, isVoid, lengthof, nameof and idof accept either a type argument, an argument or both. In the uncommon case of providing an argument, that is not solely operating on a type, these builtins now preserve side-effects of the argument which is safer but may prevent compile-time branch elimination in such cases.

Unified consumption of the assemblyscript package

Entrypoints for the various components are now:

• assemblyscript as before
• assemblyscript/asc to obtain the compiler frontend
• assemblyscript/transform to obtain the transform base class
• assemblyscript/binaryen to obtain the exact instance of Binaryen used
• assemblyscript/util/*.js to obtain various utility, i.e. for configuration parsing

Removal of ascMain in package.json

Fixes #1954. As it turned out, specifying an alternative entry point in package.json led to more issues than it solved, often not finding files and types because existing tooling does not recognize it. As such, this mechanism has been removed in favor of plain node-style resolution, i.e. import { x } from "myPackage/assembly", which is not susceptible to these problems.

Removal of experimental --extension CLI option

See #1003. While it is likely that discussion about using another file extension will continue in the future, the CLI option was meant for experimentation, had various issues and lately became stale. Hence it has been removed to simplify matters on common ground.

Replaced --explicitStart CLI option with --exportStart NAME

Fixes #2099. The new option is more general in that it also accepts the desired export name to use for the start function. Typical options are:

• --exportStart _start for a WASI command. This is equivalent to the former --explicitStart.
• --exportStart _initialize for a WASI reactor.
• --exportStart myCustomStartFunctionName for anything else.

Note that the start function, no matter how it is named, must always be called before any other exports to initialize the module.

Renamed untouched/optimized to debug/release

Default compilation targets (generated) for projects are now named debug and release to better fit their purpose and to have only one set of names to remember.

Enabled various WebAssembly features

Some meanwhile standardized features have been enabled by default:

• Nontrapping float to int conversions brings additional conversion operations.
• Bulk Memory replaces fallback implementations of memory.fill and memory.copy.

Note that we are still holding back on enabling SIMD by default as it is not yet supported in Safari, but one can already play with it using --enable simd.

Reworked development workflow

Prior, AssemblyScript utilized ts-node in development to lessen overhead from recompilation on changes. This approach turned out to be not viable anymore due to various issues with modern language features and has been dropped, alongside webpack for final builds, and replaced with esbuild. The new development workflow is to execute npm run watch which will automatically and quickly produce builds. As a side effect, workarounds and dependencies to support the prior development workflow could be dropped.

And the kitchen sink

• Various clean ups have been performed, with no longer needed files and long deprecated APIs being removed
• Support for older Node.js versions not supporting ECMAScript modules has been dropped
• The WebIDL and Asm.js targets have been dropped as they were of little use and largely unmaintained
• The --listFiles (not Wasm-target compatible) and --traceResolution (now tested programmatically) CLI options have been removed.
• The Binaryen dependency now uses Wasm builds and has been significantly sped up by enabling more optimizations. Optimizing the AssemblyScript compiler itself now only takes about a third of the time.

• I've read the contributing guidelines
• I've added my name and email to the NOTICE file

[MaxGraey]

What do to on the Web? We'd need import maps, but these aren't supported universally yet.

How about apply them in AOT style with esbuild-plugin-import-map plugin?
https://github.com/trygve-lie/esbuild-plugin-import-map

[dcodeIO]

When the dist files are transformed, these will no longer work under node. Would be cool if we could have universal builds.

[MaxGraey]

When the dist files are transformed, these will no longer work under node. Would be cool if we could have universal builds.

I don't think it's possible now. FF and Safari still desn't support import-maps. So either abandon import-maps completely or have two builds. One for node.js and Chrome with import-maps and one without it for everyone else

[dcodeIO]

There is es-module-shims but I'm not sure if it needs to parse the dist files. That would probably be slow for multi-MB artifacts.

[MaxGraey]

Hmm: guybedford/es-module-shims#1 (comment)
also this:
guybedford/es-module-shims#5

[trusktr]

What does pos stand for?

[trusktr]

Oh, piece of poo

[dcodeIO]

Stands for position here. Would you like it to be changed?

[trusktr]

why support the option if it is always enabled?

[trusktr]

Oh, I see, it's not AS-specific, we just always want Node to have it enabled.

[dcodeIO]

Right, previously was using an npm package for source map support, but recent Node has this option which works better. In particular, when crashing, this produces a proper trace instead of a lot of gibberish.

[trusktr]

yay, obliterate namespace

[dcodeIO]

It's still an open question to me to what extent we'd want to support more complex exports. In particular, I could imagine that when a class is exported, and it is used as, say, the return type of an exported function, that a wrapper would be nice. Right now, the two options are either to a) copy field by field to a JS object or b) pass an internal reference. The latter could be extended with a wrapper, but while it would function like a normal object, would have some hidden cost depending on the situation. Right now I'm leaning to leaving this TBD and tackle once use cases arise.

[trusktr]

Is there a more readable way to do this, perhaps as template strings instead of concatenating?

[dcodeIO]

I assume you refer to the whole file? If so, both the .js and .d.ts generation utilize a mix as I saw fit. In particular it uses .pushes where there are branches in between, and there are a lot. Looks a bit overloaded, I'd agree.

[trusktr]

so cool

[trusktr]

So if we call this in AS, we get back an externref. How will it be useful? We have to pass back to JS for every string manipulation?

[dcodeIO]

So far the bindings generate one half of the story, that is translating from AS -> JS. The other half, based on reference types, would be a follow-up PR. I could imagine that it would look like declared classes, to annotate the capabilities of external objects. Something like

declare class JSString {
  toString(): JSString;
  ...
}

then "backed" by externref, with the bindings then switching from externref to JSString and so on. But that's still TODO and tied to eventual table allocation etc.

[trusktr]

Is the assumption that a library like asdom would import these namespaces, then export new APIs with the actual types/wrappers? (f.e. a class with method getElementById(id: string): Element | null)

[dcodeIO]

This is merely what's possible today, since we cannot yet "describe" DOM capabilities purely in AS code, which is still TODO. See also the previous comment on how this could look in the future :)

[trusktr]

I want to add a webassembly editor on a web page build with webpack, can I use this branch ?

@lalop Yeah, I think so, and probably recommended since the other stuff is becoming obsolete after this huge PR.

[trusktr]

@dcodeIO How ready is this to merge? Is it usable enough, such that any further work can be done in new PRs?

[dcodeIO]

This is mostly ready, according to the initial post. There are still a few open questions, for example what to do with exported classes or namespaces, which are currently not supported, or how to bind the other way around from JS->AS, but none of them are blocking (can be tackled later) if we agree that the general approach here is what we want to continue with.

The accompanying documentation update could need a bit of review before this is merged, I think, just to make sure that the transition is smooth for everyone involved, say checking that the documentation covers everything one needs to know from here onward.