A threaded SDK

.eslintrc.yaml

@@ -0,0 +1,16 @@
+env:
+  browser: true
+  es2021: true
+  node: true
+extends: ['eslint:recommended', 'plugin:@typescript-eslint/recommended']
+parser: '@typescript-eslint/parser'
+root: true
+ignorePatterns: ["oldsrc/", "tests/mod.test.ts", "tests/index.test.ts"]
+rules:
+  no-constant-condition: 'off'
+  no-unused-vars: 'off'
+  '@typescript-eslint/no-unused-vars':
+    - 'error'
+    - argsIgnorePattern: '_.*'
+      varsIgnorePattern: '_.*'
+  '@typescript-eslint/no-explicit-any': 'off'

.github/workflows/ci.yml

@@ -19,20 +19,19 @@ jobs:
     steps:
     - uses: actions/checkout@v3
 
-    - name: Setup Node.js environment
-      uses: actions/setup-node@v3.8.1
-
-    - name: Setup Deno
-      uses: denoland/setup-deno@v1.1.2
-
-    - name: NPM install
-      run: npm ci
-
-    - name: Build
-      run: npm run build
-
-    - name: Node Test
+    - uses: extractions/setup-just@v1
+
+    - uses: actions/setup-node@v3.8.1
+      with:
+        node-version: lts/*
+        check-latest: true
+
+    - uses: denoland/setup-deno@v1.1.2
+
+    - uses: oven-sh/setup-bun@v1
+      if: ${{ matrix.os != 'windows-latest' }}
+      with:
+        bun-version: latest
+
+    - name: Test
       run: npm run test
-
-    - name: Deno Test
-      run: deno test -A ./tests/mod.test.ts

.gitignore

@@ -1,3 +1,9 @@
+# test artifacts
+tests/artifacts
+
+# generated docs
+docs/
+
 # Logs
 logs
 *.log

.vscode/settings.json

@@ -3,8 +3,8 @@
     "deno.lint": true,
     "deno.unstable": true,
     "deno.enablePaths": [
-        "./src/deno/mod.ts",
+        "./src/mod.ts",
         "./examples/deno.ts",
-        "./tests/mod.test.ts"
+        "./src/mod.test.ts"
     ]
-}
+}

DEVELOPING.md

@@ -0,0 +1,115 @@
+# Developing
+
+## The build process
+
+The Extism SDK targets several platforms:
+
+- Deno
+- Node ECMAScript Modules ("ESM")
+- Node CommonJS Modules ("CJS")
+- Browser ECMAScript Modules
+
+The source of this library is written as valid TypeScript, which may be
+consumed and run directly by Deno. The latter three platforms are treated as
+compile targets. There are two other compile targets:
+
+- The source of the [Worker](https://mdn.io/worker), compiled for the browser.
+- The source of the [Worker](https://mdn.io/worker), compiled for node.
+- Tests
+
+For compiled targets, the worker is compiled to a single artifact with an entry
+point starting at `src/worker.ts`, base64-encoded, and included in the
+resulting artifact.
+
+Builds are orchestrated by the `justfile` and `esbuild`: each build target recipe accepts
+incoming `esbuild` flags as an array of JSON data and prepends its own configuration.
+This allows dependent recipes to override earlier flags. An annotated example:
+
+```
+build_worker_browser out='worker/browser' args='[]': # <-- we accept args and an out dir
+    #!/bin/bash
+    config="$(<<<'{{ args }}' jq -cM '
+      [{
+        "format": "esm",
+        "alias": {
+          "js-sdk:capabilities": "./src/polyfills/browser-capabilities.ts",
+          "node:worker_threads": "./src/polyfills/worker-node-worker_threads.ts",
+          "js-sdk:fs": "./src/polyfills/browser-fs.ts",
+          "js-sdk:wasi": "./src/polyfills/browser-wasi.ts",
+        }
+      }] + . # <--- add this recipe's flags to the incoming flags.
+    ')"
+    just build_worker {{ out }} "$config"
+```
+
+There is a `_build` recipe that all other build targets depend on, at
+varying degrees of indirection. This `_build` fixes all Deno-style `.ts`
+import statements, invokes `esbuild`, and emits TypeScript declarations
+via `tsc`.
+
+### Polyfills
+
+We use `esbuild` to compile to these targets. This allows us to abstract
+differences at module boundaries and replace them as-needed. For example: each
+of Node, Deno, and the Browser have different WASI libraries with slightly different
+interfaces. We define a **virtual module**, `js-sdk:wasi`, and implement it by:
+
+1. Modifying `deno.json`; adding a mapping from `js-sdk:wasi` to `./src/polyfills/deno-wasi.ts`.
+2. Adding a `types/js-sdk:wasi/index.d.ts` file.
+3. Modifying the esbuild `alias` added by `build_worker`, `build_worker_node`,
+   `build_node_cjs`, `build_node_esm`, and `build_browser`.
+    - Node overrides are set to `./src/polyfills/node-wasi.ts`.
+    - Browser overrides are set to `./src/polyfills/browser-wasi.ts`.
+
+In this manner, differences between the platforms are hidden and the core of
+the library can be written in "mutually intelligble" TypeScript.
+
+One notable exception to this rule: Deno implements Node polyfills; for
+complicated imports, like `node:worker_threads`, we instead only polyfill the
+browser. The browser polyfill is split into `host:node:worker_threads.ts` and
+`worker-node-worker_threads.ts`: these polyfill just enough of the Node worker
+thread API over the top of builtin workers to make them adhere to the same
+interface.
+
+### Testing
+
+Tests are co-located with source code, using the `*.test.ts` pattern. Tests
+are run in three forms:
+
+- Interpreted, via `deno test -A`
+- Compiled, via `node --test`
+- And via playwright, which polyfills `node:test` using `tape` and runs tests
+  across firefox, webkit, and chromium.
+
+The `assert` API is polyfilled in browser using
+[`rollup-plugin-polyfill-node`](https://npm.im/rollup-plugin-polyfill-node).
+This polyfill doesn't track Node's APIs very closely, so it's best to stick to
+simple assertions (`assert.equal`.)
+
+## The Extism runtime, shared memory, and worker threads
+
+This SDK defaults to running on background threads in contexts where that is
+feasible. Host functions require this library to share memory between the main
+and worker threads, however. The rules on transferring buffers are as follows:
+
+- ArrayBuffers may be transferred asynchronously between main and worker threads. Once
+  transferred they may no longer be accessed on the sending thread.
+- SharedArrayBuffers may be sent _only_ from the main thread. (All browsers allow
+  the creation of SharedArrayBuffers off of the main thread, but Chromium disallows
+  _sending_ those SharedArrayBuffers to the main thread from the worker.)
+- Browser environments disallow using `TextDecoder` against typed arrays backed by
+  SharedArrayBuffers. The Extism library handles this transparently by copying out
+  of shared memory.
+
+These rules make navigating memory sharing fairly tricky compared to other SDK platforms.
+As a result, the JS SDK includes its own extism runtime, which:
+
+- Reserves 16 bits of address space for "page id" information, leaving 48 bits per "page"
+  of allocated memory.
+- Creates sharedarraybuffers on the worker thread and shares them with the worker thread on
+  `call()`.
+- Worker-originated pages are transferred up to the main thread and copied into sharedarraybuffers
+  whenever the worker transfers control to the main thread (whether returning from a `call()` or
+  calling a `hostfn`.)
+- When new pages are created during the execution of a `hostfn`, they will be
+  _copied down_ to the worker thread using a 64KiB scratch space.